Policy gradient

A policy gradient method improves a policy ${\pi }_{\theta }$ by estimating the gradient of expected return with respect to the parameters $\theta$ and taking a step uphill. Instead of learning values first and acting greedily, it adjusts the probabilities of actions directly: actions that led to high return become more likely.

The policy gradient theorem

Let $J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[G_0]$ be the expected return of trajectories $\tau$ generated by the policy. The policy gradient theorem states

where ${\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$ is the score function (the direction in parameter space that makes action $a_t$ more likely in state $s_t$) and $Q^{{\pi }_{\theta }}$ is the action-value function. The key fact is that the gradient does not involve the derivative of the environment's dynamics, which are usually unknown. You only need samples from the policy and the ability to differentiate its log-probabilities.

In practice $Q$ is replaced by an estimate: the sampled return gives REINFORCE; subtracting a baseline or using an advantage estimate from a critic gives actor-critic methods.

Worked example

Take a two-action softmax policy with logits $({\theta }_1,{\theta }_2)=(0,0)$, so each action has probability 0.5. Action 1 pays 1, action 2 pays 0, so $J={\pi }_1$. For a softmax, $\partial \log {\pi }_1/\partial {\theta }_1=1-{\pi }_1=0.5$ and $\partial \log {\pi }_2/\partial {\theta }_1=-{\pi }_1=-0.5$. The expected gradient is $0.5\times 0.5\times 1+0.5\times (-0.5)\times 0=0.25$, which matches the exact derivative ${\pi }_1(1-{\pi }_1)=0.25$. A step of size 1 raises ${\theta }_1$ to 0.25 and ${\pi }_1$ to about 0.56.

Properties

• Works with continuous and high-dimensional action spaces, where taking a max over actions is impractical.
• Learns stochastic policies naturally, which helps exploration and matters in partially observed problems.
• Gradient estimates are unbiased but high variance, which is why baselines, critics and trust regions exist.

Why it matters

Almost every method used to fine-tune language models with reinforcement learning, including PPO and GRPO, is a policy gradient method: the model is the policy, tokens are actions, and the score function is the gradient of the token log-probabilities. Understanding the theorem explains why these methods need sampled outputs, why their updates are noisy, and why variance reduction dominates their design.

Related terms

• REINFORCE
• Baseline (policy gradient)
• Advantage function
• Actor-critic
• PPO (Proximal Policy Optimization)
• Policy

Sources

1. Richard S. Sutton, David McAllester, Satinder Singh, Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation, 1999. proceedings.neurips.cc
2. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, 1992. doi.org
3. Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net