Advantage function

The advantage function measures how much better taking action $a$ in state $s$ is than what the policy would do on average from $s$. Positive advantage means the action beats the policy's typical behaviour; negative means it falls short.

Definition

where $Q^{\pi }(s,a)$ is the expected return after taking $a$ in $s$ and then following policy $\pi$ (the Q-function), and $V^{\pi }(s)={\mathbb{E}}_{a\sim \pi }[Q^{\pi }(s,a)]$ is the state value. A direct consequence is that the advantage averages to zero under the policy: ${\mathbb{E}}_{a\sim \pi }[A^{\pi }(s,a)]=0$.

Worked example

In some state a policy picks three actions with probabilities 0.5, 0.3 and 0.2, and their Q-values are 3, 5 and 10. The state value is $0.5\times 3+0.3\times 5+0.2\times 10=5$. The advantages are $-2$, $0$ and $+5$. Their policy-weighted mean is $0.5(-2)+0.3(0)+0.2(5)=0$, as it must be. A policy gradient step using these advantages makes the third action more likely, the first less likely, and leaves the second roughly where it is, even though all three have positive Q-values.

How it is estimated

The true advantage is unknown, so algorithms estimate it:

• Monte Carlo: ${\hat{A}}_t=G_t-\hat{V}(s_t)$, unbiased if the value estimate is only used as a baseline, but high variance.
• One-step TD: ${\hat{A}}_t=r_{t+1}+\gamma \hat{V}(s_{t+1})-\hat{V}(s_t)$, the TD error, which is low variance but biased when $\hat{V}$ is wrong.
• GAE: an exponentially weighted mix of the two, controlled by $\lambda$.
• Group-relative: for language models, the reward of one sample minus the mean reward of other samples for the same prompt, often divided by their standard deviation.

Advantage versus baseline

A baseline is a tool, a number you subtract; the advantage is the quantity you are trying to estimate when you subtract the best state-dependent baseline, the value function. Using $V^{\pi }(s)$ as the baseline is close to, but not exactly, the variance-minimising choice.

Why it matters

Advantages are the signal that modern policy gradient methods, including actor-critic, PPO and GRPO, actually multiply into their updates. Most of the practical differences between these algorithms come down to how they estimate the advantage and how much bias they accept for lower variance.

Related terms

• Q-function
• Value function
• Baseline (policy gradient)
• Generalized advantage estimation (GAE)
• TD error
• Actor-critic

Sources

1. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation, 2015. arxiv.org
2. Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net