Generalized advantage estimation (GAE)

Generalized advantage estimation (GAE) is a way to estimate the advantage function for policy gradient updates. It averages many n-step advantage estimates with exponentially decaying weights, so a single parameter $\lambda \in [0,1]$ slides between a low-variance, biased estimate and a high-variance, unbiased one.

Definition

With a learned value function $\hat{V}$, first compute the TD error at each step, ${\delta }_t=r_{t+1}+\gamma \hat{V}(s_{t+1})-\hat{V}(s_t)$. GAE then sums them:

where $\gamma$ is the discount factor and $\lambda$ controls how quickly distant TD errors are down-weighted. In code this is computed backwards over a trajectory with the recursion ${\hat{A}}_t={\delta }_t+\gamma \lambda {\hat{A}}_{t+1}$, resetting at episode ends.

The two extremes

• $\lambda =0$: ${\hat{A}}_t={\delta }_t$, the one-step TD advantage. Low variance, but biased whenever $\hat{V}$ is wrong.
• $\lambda =1$: ${\hat{A}}_t=G_t-\hat{V}(s_t)$, the Monte Carlo return minus a baseline. Unbiased with respect to value errors, but high variance.

Values around 0.9 to 0.97 are common. This mirrors TD($\lambda$) for value learning, applied to advantages.

Worked example

A trajectory has TD errors ${\delta }_0=1$, ${\delta }_1=0.5$, ${\delta }_2=-0.2$ and then ends. With $\gamma =0.99$ and $\lambda =0.95$, $\gamma \lambda =0.9405$ and

With $\lambda =0$ the estimate would be just $1.0$, ignoring the later evidence that things kept going somewhat better than the critic predicted.

In practice

GAE is the default advantage estimator inside PPO and many actor-critic implementations. Advantages are usually normalised per batch to zero mean and unit variance, and the value function is trained on the $\lambda$-return targets ${\hat{A}}_t+\hat{V}(s_t)$. Note that $\gamma$ and $\lambda$ play different roles: $\gamma$ is part of the objective, while $\lambda$ only changes how the gradient is estimated.

Why it matters

The quality of advantage estimates often matters more than the choice of policy optimiser. GAE gives a principled dial for the bias-variance trade-off and is the reason PPO-style training works reliably on long-horizon tasks. In RL for language models with a single reward at the end, a value head plus GAE is the standard PPO recipe, and its cost is what group-based methods such as GRPO try to avoid.

Related terms

• Advantage function
• TD error
• Actor-critic
• PPO (Proximal Policy Optimization)
• Discount factor
• Temporal-difference learning

Sources

1. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation, 2015. arxiv.org