Baseline (policy gradient)

In policy gradient methods, a baseline is any quantity $b(s)$ that does not depend on the action and is subtracted from the return before it scales the gradient. It does not change the expected gradient, but a good baseline greatly reduces the variance of the estimate, so learning needs fewer samples.

Definition and why it is unbiased

The baseline-corrected estimator is $\hat{g}=(G_t-b(s_t)){\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$. It stays unbiased because the score function has zero mean under the policy:

Here $G_t$ is the sampled return and ${\pi }_{\theta }$ the policy. The baseline acts as a control variate: it removes a component of the estimate that is pure noise.

Worked example

A two-armed bandit has a softmax policy with both arms at probability 0.5. Arm 1 always pays 10 and arm 2 always pays 11. The score for logit ${\theta }_1$ is $+0.5$ after pulling arm 1 and $-0.5$ after arm 2. The true gradient is ${\pi }_1(1-{\pi }_1)(10-11)=-0.25$.

• No baseline: the estimate is $10\times 0.5=5$ or $11\times (-0.5)=-5.5$. Its mean is $-0.25$, but its variance is about 27.6. Every sample points strongly in a random direction.
• Baseline $b=10.5$: the estimate is $(10-10.5)\times 0.5=-0.25$ or $(11-10.5)\times (-0.5)=-0.25$. Same mean, variance zero.

A seeded simulation of 100,000 pulls gives sample variances of 27.56 and 0.0. Rewards that are large relative to the differences between actions are exactly the case where a baseline helps most.

Common choices

• A running average of recent returns, the simplest option.
• A learned state-value estimate $\hat{V}(s)$, which makes $G_t-\hat{V}(s_t)$ an estimate of the advantage and turns the method into actor-critic.
• The mean reward of other samples for the same input, as in leave-one-out REINFORCE and GRPO for language models.

A baseline must not depend on the current action; one that does generally biases the gradient.

Why it matters

Without a baseline, policy gradients are often too noisy to train anything larger than a toy. The choice of baseline is a large part of what separates REINFORCE, actor-critic, PPO and group-based methods used in RL for language models, and it is usually where the real sample-efficiency gains come from.

Related terms

• Advantage function
• REINFORCE
• Policy gradient
• Actor-critic
• GRPO (Group Relative Policy Optimization)

Sources

1. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, 1992. doi.org
2. Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net