Using wind farms for frequency regulation of the power grid requires the ability to track a changing power signal for an extended period of time. In this paper, we investigate the use of multi-agent reinforcement learning (MARL) to achieve power tracking by blade pitch control of turbines. The wind farm power tracking problem is modeled as a multi-agent cooperative task. Independent actor-critic reinforcement learning agents are trained to convert a farm power target to individual pitch commands for each turbine. To our knowledge, this is the first time a model-free decentralized control approach is investigated for secondary frequency regulation in wind farms. The effectiveness of the proposed algorithm is assessed by tracking historic regulation data under varying wind conditions, using the dynamic medium-fidelity simulator FAST.Farm. A study of different observation spaces is also provided to assess which information is necessary to accurately track a power signal.