Actor-critic: Definition and Examples

An actor-critic method learns two things at once: an actor, which is the policy that chooses actions, and a critic, which estimates a value function and tells the actor how good its recent actions were. The critic replaces noisy Monte Carlo returns with a learned estimate, which lowers the variance of policy gradient updates.

One-step actor-critic

After each transition $(s, a, r, s^{'})$ , the critic computes the TD error and both networks update:

δ = r + γ V^_{w} (s^{'}) - V^_{w} (s), w \leftarrow w + α_{w} δ \nabla_{w} V^_{w} (s), θ \leftarrow θ + α_{θ} δ \nabla_{θ} lo g π_{θ} (a ∣ s) .

Here $V^_{w}$ is the critic with parameters $w$ , $π_{θ}$ is the actor, $γ$ is the discount factor and $α_{w}, α_{θ}$ are step sizes. The TD error is a one-sample estimate of the advantage: positive if things went better than the critic expected.

Worked example

The critic currently says $V^(s) = 2$ and $V^(s^{'}) = 3$ . The agent takes action $a$ , receives $r = 1$ , and $γ = 0.9$ . Then $δ = 1 + 0.9 \times 3 - 2 = 1.7$ . With a tabular critic and $α_{w} = 0.1$ , $V^(s)$ moves to $2.17$ . The actor raises the log-probability of $a$ in $s$ by a step proportional to 1.7. Had the reward been $- 1$ , $δ$ would be $- 0.3$ and $a$ would become slightly less likely.

Variants

A2C and A3C: synchronous and asynchronous versions that use n-step returns and many parallel environments, with an entropy bonus.
GAE-based: the critic feeds an exponentially weighted advantage estimate; this is the standard setup inside PPO.
Off-policy actor-critic: methods such as soft actor-critic learn a Q-function critic from a replay buffer, which makes them far more sample efficient for continuous control.

The bias-variance trade-off

A critic is never exact, so the actor's gradient becomes biased in exchange for much lower variance. If the critic is poor, the actor optimises the wrong signal; if the actor changes quickly, the critic chases a moving target. Separate learning rates, with the critic usually learning faster, help keep the two in step.

Why it matters

Most practical deep RL, from robotics to the PPO setups used to fine-tune language models with a value head, is some form of actor-critic. Knowing that the critic is a learned, fallible estimate explains many training failures: a value head that lags behind the policy gives misleading advantages, and the policy follows them.

Sources

Vijay R. Konda, John N. Tsitsiklis. Actor-Critic Algorithms, 1999. proceedings.neurips.cc
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning, 2016. arxiv.org
Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net