TD error

The temporal-difference (TD) error is the difference between what an agent expected from a state and what it got one step later, measured with its own value estimates. A positive TD error means the step went better than predicted; a negative one means worse.

Definition

$R_{t+1}$ is the observed reward, $V$ the current value estimate and $\gamma$ the discount factor. If the episode ends at $S_{t+1}$, its value is taken as 0. When $V$ equals the true $V^{\pi }$, the TD error has zero mean under $\pi$ in every state, which is just the Bellman equation rearranged.

Worked example

The agent estimates $V(s)=4$. It acts, receives reward 1 and lands in $s^{\prime }$ with $V(s^{\prime })=5$. With $\gamma =0.9$, $\delta =1+0.9\times 5-4=1.5$. The outcome beat expectations, so TD learning with step size $\alpha =0.1$ raises $V(s)$ to $4+0.1\times 1.5=4.15$.

Three roles of the TD error

• Learning signal for critics. Value networks are trained to minimise ${\delta }_t^2$ with the target held fixed.
• Advantage estimate for actors. If $V=V^{\pi }$, then $\mathbb{E}[{\delta }_t\mid S_t=s,A_t=a]=Q^{\pi }(s,a)-V^{\pi }(s)$, the advantage. One-step actor-critic uses ${\delta }_t$ directly as the advantage, and GAE uses a discounted sum of them: ${\hat{A}}_t={\sum }_{l\ge 0}(\gamma \lambda {)}^l{\delta }_{t+l}$.
• Bridge to Monte Carlo. For a fixed $V$, the Monte Carlo error decomposes into TD errors: $G_t-V(S_t)={\sum }_{k=0}^{T-t-1}{\gamma }^k{\delta }_{t+k}$.

A note from neuroscience

The firing of dopamine neurons in primates looks strikingly like a TD error: they respond to unexpected rewards, shift their response to cues that predict reward, and dip when a predicted reward is omitted. This finding made the TD error one of the best-known links between machine learning and biology.

Why it matters

The TD error is the most compact diagnostic in RL training. Large, persistent TD errors mean the critic is not tracking the policy; errors that grow without bound often signal divergence. Because advantage estimates are built from TD errors, a poor critic directly corrupts policy updates, including in language-model RL with a learned value head.

Related terms

• Temporal-difference learning
• Value function
• Advantage function
• Generalized advantage estimation (GAE)
• Actor-critic
• Bellman equation

Sources

1. Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net
2. Wolfram Schultz, Peter Dayan, P. Read Montague. A Neural Substrate of Prediction and Reward, 1997. doi.org