Actor-critic methods and generalized advantage estimation (GAE), with the λ trade-off measured

An actor-critic method trains two things at once: an actor, the policy that chooses actions, and a critic, a learned value function that judges them. The critic turns noisy Monte Carlo returns into lower-variance advantage estimates, most simply the temporal-difference error ${\delta }_t=r_t+\gamma V(s_{t+1})-V(s_t)$. Generalized advantage estimation (GAE) blends multi-step TD errors with a parameter $\lambda$: $\lambda =0$ trusts the critic completely, $\lambda =1$ ignores it except as a baseline, and values in between trade bias for variance.

This page builds actor-critic up from the policy gradient, derives GAE, and then measures the trade-off it controls. On a simulated chain we computed the squared bias and the variance of the policy gradient estimate for every $\lambda$ and several levels of critic error. With a perfect critic the best $\lambda$ was 0.1; as the critic got worse, the best $\lambda$ rose to 0.5, 0.7, 0.8 and 0.85. In a learning run with a critic that starts out ignorant, every seed with $\lambda \le 0.5$ locked onto a myopic policy. All numbers come from seeded numpy runs we did for this article.

What is an actor-critic method?#

The policy gradient theorem [2] says the gradient of expected return is $\mathbb{E}[{\sum }_t{\gamma }^t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)A^{\pi }(s_t,a_t)]$, where the advantage $A^{\pi }=Q^{\pi }-V^{\pi }$ measures how much better an action is than the policy's average. REINFORCE [3] estimates the advantage with the sampled return $G_t$, minus a baseline if it has one. That estimate is unbiased, but $G_t$ adds up every random reward until the end of the episode, so its variance grows with the horizon.

The idea of actor-critic is old: Barto, Sutton and Anderson's 1983 pole-balancing system already had an "adaptive critic element" that learned to predict reinforcement and an "associative search element" that learned to act on the critic's signal [5]. The critic was trained with what Sutton later formalized as temporal-difference learning [6], and Konda and Tsitsiklis gave convergence results for actor-critic algorithms with function approximation [7]. Modern deep RL algorithms such as A2C, its asynchronous predecessor A3C [8], and PPO with a value head are all actor-critic methods.

The loop has two updates per batch of experience:

1. Critic: move $V_w(s)$ toward a target built from observed rewards and its own predictions, for example $r_t+\gamma V_w(s_{t+1})$.
2. Actor: move $\theta$ along ${\sum }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t){\hat{A}}_t$, where ${\hat{A}}_t$ is computed from the critic.

The TD error as an advantage estimate#

The temporal-difference error is

where $r_t$ is the reward at step $t$, $\gamma$ the discount factor and $V$ the critic's value estimate (with $V=0$ at terminal states). If the critic is exact, $V=V^{\pi }$, then conditioning on the state and action gives

So the one-step TD error is an unbiased advantage estimate when the critic is right, and it involves only one random reward and one random transition, far less noise than a full return. When the critic is wrong, the error $\gamma V(s_{t+1})-V(s_t)$ leaks straight into the advantage, and because it depends on $s_{t+1}$, which depends on the action, it does bias the policy gradient. That is the trade: the TD error gets its low variance by trusting a model that may be wrong. The Bellman equation page explains why $V^{\pi }$ satisfies the consistency condition that makes $\delta$ average to zero under the policy.

n-step advantages#

Between one step and the whole episode there is a family of $k$-step estimates. Summing $k$ discounted TD errors telescopes the intermediate value estimates away:

The $k$-step estimate uses $k$ real rewards and then bootstraps from the critic. The critic's error enters only through ${\gamma }^{k}V(s_{t+k})$, shrunk by ${\gamma }^k$, plus the $V(s_t)$ term, which acts as a baseline and does not bias the gradient. As $k$ grows, bias falls and variance rises. At $k=\infty$ the estimate is $G_t-V(s_t)$: REINFORCE with a value baseline.

Generalized advantage estimation#

Rather than picking one $k$, GAE averages all of them with exponentially decaying weights $(1-\lambda ){\lambda }^{k-1}$, which simplifies to a discounted sum of TD errors [4]:

Here $\lambda \in [0,1]$ controls how far into the future the estimate looks before relying on the critic. The two ends are the cases we already know:

• $\lambda =0$: ${\hat{A}}_t={\delta }_t$, the one-step TD error. Lowest variance, and biased whenever $V$ is wrong.
• $\lambda =1$: ${\hat{A}}_t={\sum }_l{\gamma }^l{\delta }_{t+l}=G_t-V(s_t)$. Unbiased for any $V$, because the critic only appears as a baseline, with the high variance of Monte Carlo returns.

The construction is the same as the $\lambda$-return in TD($\lambda$), applied to advantages; Sutton and Barto treat the $\lambda$-return and eligibility traces at length [1]. In practice GAE is computed backwards through a trajectory in one pass, ${\hat{A}}_t={\delta }_t+\gamma \lambda {\hat{A}}_{t+1}$, and the critic is usually trained toward ${\hat{A}}_t+V(s_t)$, the $\lambda$-return.

Two parameters, two roles. Schulman and colleagues point out that $\gamma$ also acts as a bias-variance knob, since discounting down-weights distant, noisy rewards at the cost of changing the objective, while $\lambda$ introduces bias only through critic error. In their experiments the best results came from intermediate values, roughly $\gamma \in [0.96,0.99]$ and $\lambda \in [0.92,0.99]$ on their cart-pole task [4]; $\lambda =0.95$ has since become a common default in PPO code.

Measuring the λ trade-off on a chain#

Setup (simulation). A chain of 10 states. The agent starts at the left end; moving right costs 0.1, and stepping right from the last state reaches a goal worth +10 and ends the episode. Moving left pays +0.2 (a small temptation) and moves back one state. Every reward carries Gaussian noise with standard deviation 1, and $\gamma =0.95$. We fix the policy at $\pi (\text{right})=0.75$ in every state and parameterize it with one logit per state, so the exact policy gradient (squared length 0.306) and the true values (3.9 at the start, rising to 9.5 next to the goal) can be computed by solving linear systems. To control critic quality we used $\hat{V}(s)=V^{\pi }(s)+ϵz_s$, where $z$ is a fixed pattern of standard normal errors and $ϵ$ the error scale. From 6,000 episodes we computed, for each $\lambda$, the squared bias of the GAE policy gradient estimate (its mean over all episodes minus the exact gradient, squared and summed) and the variance of a 10-episode batch estimate.

The bias at $\lambda =1$ (about 0.002 to 0.004) is the Monte Carlo noise floor of the experiment, not real bias: with an exact or inexact critic, $\lambda =1$ is unbiased, as the algebra says. Everything else in the table is the trade-off made visible.

• With a perfect critic, bootstrap. At $ϵ=0$ the TD error is unbiased and its variance (0.218) is seven times lower than the Monte Carlo estimate's (1.543).
• Critic error turns into bias fast at low $\lambda$. An error of one unit in a value scale of 4 to 10 gives $\lambda =0$ a squared bias of 1.39, more than four times the squared length of the true gradient. That estimate does not just wobble around the right direction; it points somewhere else.
• A bad critic also raises variance. At $\lambda =1$ the critic is only a baseline, yet its variance grows from 1.54 to 2.70 as $ϵ$ goes from 0 to 3, because a baseline far from $V^{\pi }$ removes less variance, as the baselines article shows.
• The best $\lambda$ tracks critic quality. It moves from 0.1 to 0.85 as the critic degrades, and when the critic is poor the MSE curve is flat between about 0.8 and 0.9, so a value in that region is a reasonable default when you do not know how good your critic is.

Explore the trade-off#

The figure below reruns the same simulation in your browser (2,400 episodes, a different random critic-error pattern, so the numbers differ slightly from the table). Move the critic error to see the bias curve rise and the best $\lambda$ shift right; the dashed line marks the $\lambda$ you select.

An actor-critic that learns: when bias locks in#

A fixed critic error is a simplification. In a real actor-critic the critic is learned alongside the policy, starts out wrong, and its errors are systematic, not random. We trained a tabular actor-critic on the same chain, now with the policy free to learn.

Setup (simulation). Actor: one logit per state, starting at $\pi (\text{right})=0.5$. Critic: a table initialized to zero. Each iteration collects 4 episodes (capped at 300 steps), computes GAE advantages, takes an actor step of size 0.1 along ${\sum }_t(a_t-{\pi }_t){\hat{A}}_t$ per state, and moves each visited state's value 10% of the way toward its average $\lambda$-return. We ran 500 iterations for 20 seeds per $\lambda$. The optimal policy (always right) is worth 5.5 from the start; camping in the first state and collecting +0.2 forever is worth exactly 4.0; the initial policy is worth 1.65.

Every seed with $\lambda \le 0.5$ converged to the camping policy, worth exactly 4.0. The reason is the zero-initialized critic. Early on, the critic believes every state is worth nothing, so a bootstrapped advantage for "right" sees a cost of 0.1 and a next state worth zero, while "left" sees a reward of +0.2. The goal's value has to propagate back through ten states of TD updates before the critic knows about it, and by then the actor has made "left" near-deterministic, so the states near the goal are rarely visited and the critic never learns their value. Higher $\lambda$ lets the real +10 reach the advantage estimates directly. In this task, with a poor critic and a distant reward, $\lambda =1$ ended marginally ahead of 0.95; with a pretrained or quickly learned critic, the earlier experiment says a lower $\lambda$ would win. The general point: bias in actor-critic is not only an error in one gradient, it can steer which data you collect next, and that can make it permanent.

Actor-critic vs REINFORCE, and A2C vs A3C#

REINFORCE with a learned value baseline is GAE at $\lambda =1$; one-step actor-critic is GAE at $\lambda =0$. Sutton and Barto reserve the name actor-critic for methods that bootstrap, where the critic's estimate actually replaces part of the return [1]. In practice the boundary is a slider, and most modern implementations sit near the REINFORCE end with $\lambda \approx 0.95$.

A3C, introduced by Mnih and colleagues [8], runs many actors in parallel threads, each computing $n$-step advantage actor-critic gradients and applying them asynchronously to shared parameters; an entropy bonus keeps the policy from collapsing early. A2C is the synchronous variant: all workers step in lockstep and their gradients are averaged into one batched update, which uses accelerators better and is easier to reproduce. The algorithmic core, an advantage actor-critic with $n$-step or GAE advantages, is the same. PPO adds a clipped objective on top of that core so one batch can support several update epochs; see TRPO and PPO explained.

GAE in numpy#

This is the backward recursion used throughout this page, with the two details implementations most often get wrong: no bootstrapping across true episode ends, and bootstrapping from the critic when an episode is cut off by a time limit.

import numpy as np

def gae(rewards, values, next_values, terminated, gamma=0.99, lam=0.95):
    """rewards, values, next_values, terminated: arrays of length T for one rollout.
    values[t] = V(s_t); next_values[t] = V(s_{t+1}) (use V of the last observed state
    when the rollout was truncated by a time limit); terminated[t] = 1 if s_{t+1} is terminal.
    Returns advantages and lambda-return targets for the critic."""
    T = len(rewards)
    adv = np.zeros(T)
    running = 0.0
    for t in range(T - 1, -1, -1):
        nonterminal = 1.0 - terminated[t]
        delta = rewards[t] + gamma * next_values[t] * nonterminal - values[t]
        running = delta + gamma * lam * nonterminal * running
        adv[t] = running
    return adv, adv + values

Treating a time-limit cutoff as a true terminal state teaches the critic that the world ends at step 1,000, which biases both the critic and the advantages; Pardo and colleagues analyse this failure and the fix [9].

Practical guidance#

1. Start at $\lambda$ between 0.9 and 0.95, $\gamma =0.99$. These are safe when the critic is poor: at critic errors of 2 and 3, $\lambda =0.9$ was within 7% of the lowest MSE in our chain. With a good critic they are too conservative; at error 1 the best value was 0.7.
2. Lower $\lambda$ only when you trust the critic. A pretrained or well-converged value function, dense rewards, and long noisy episodes favour more bootstrapping.
3. Raise $\lambda$ when rewards are sparse or delayed. A critic cannot bootstrap information it has never seen, as the learning experiment showed.
4. Give the critic enough capacity and updates. More value-function steps per batch, a separate network or a larger learning rate for the critic are common fixes. Andrychowicz and colleagues' large study of on-policy design choices recommends GAE with $\lambda =0.9$ and found separate policy and value networks performed better [10].
5. Normalize advantages per batch. It stabilizes step sizes across tasks; remember it rescales, so it interacts with the learning rate.
6. Handle episode ends correctly. Do not bootstrap through terminal states; do bootstrap through time-limit truncations.
7. Compare settings across seeds. Differences between $\lambda =0.95$ and 1 in our runs were within run-to-run spread; see how to evaluate RL agents.

Failure modes#

• Myopic lock-in. A pessimistic or untrained critic plus low $\lambda$ can drive the actor to a short-sighted policy before the critic learns, as in our chain.
• Critic lag. The critic estimates the value of an older policy. Large actor steps make it stale; trust regions limit how far the actor moves per batch.
• Shared-network interference. A shared torso for actor and critic is cheaper but lets value-loss gradients overwhelm policy gradients; weight the losses or separate the networks.
• Reward scale. Value targets that span orders of magnitude make the critic hard to fit. Normalizing rewards or returns helps the critic more than the actor.

Actor-critic methods are the practical form of policy gradients for most problems, and the same structure appears in RL for language models, where the critic is a value head on the policy model and $\lambda$ controls how per-token rewards and a sequence-level score are blended into advantages. The broader map is in what is reinforcement learning.

Frequently asked questions#

Sources#

1. R. S. Sutton, A. G. Barto. Reinforcement Learning: An Introduction (2nd ed.), 2018. incompleteideas.net
2. R. S. Sutton, D. McAllester, S. Singh, Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation, 1999. proceedings.neurips.cc
3. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, 1992. doi.org
4. J. Schulman, P. Moritz, S. Levine, M. Jordan, P. Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation, 2015. arxiv.org
5. A. G. Barto, R. S. Sutton, C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems, 1983. doi.org
6. R. S. Sutton. Learning to predict by the methods of temporal differences, 1988. doi.org
7. V. R. Konda, J. N. Tsitsiklis. Actor-Critic Algorithms, 1999. proceedings.neurips.cc
8. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, K. Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning, 2016. arxiv.org
9. F. Pardo, A. Tavakoli, V. Levdik, P. Kormushev. Time Limits in Reinforcement Learning, 2017. arxiv.org
10. M. Andrychowicz, A. Raichuk, P. Stańczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, O. Bachem. What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study, 2020. arxiv.org