Reward overoptimization

Reward overoptimization is the measurable pattern in which optimising a policy against a learned reward model first improves real quality and then, past some point, makes it worse while the proxy score keeps climbing. It is Goodhart's law with a curve you can fit.

The Gao et al. functional forms

Gao, Schulman and Hilton (2022) trained proxy reward models on labels from a fixed "gold" reward model and tracked the gold score as the policy moved away from its initialisation. Measuring distance as $d=\sqrt{D_{\mathrm{KL}}(\pi \Vert {\pi }_{\text{init}})}$, they found

where $R$ is the gold reward gain over the initial policy, the first form is for best-of-n sampling and the second for RL. The $\alpha$ term is the useful signal; the $\beta$ term is the cost of exploiting the proxy's errors. Both coefficients changed smoothly with reward model size, and larger reward models delayed the peak.

Worked example

Take the best-of-n form with $\alpha =1$ and $\beta =0.25$. Gold reward peaks where $dR/dd=0$, at $d=\alpha /2\beta =2$, so $D_{\mathrm{KL}}=4$ nats. For best-of-n the KL from the base policy is $\log n-(n-1)/n$, which reaches 4 nats at about $n\approx 147$. The table shows the gold curve flattening:

• $n=4$: KL 0.64, $d$ 0.80, gold gain 0.64
• $n=16$: KL 1.84, $d$ 1.36, gold gain 0.90
• $n=64$: KL 3.18, $d$ 1.78, gold gain 0.99
• $n=256$: KL 4.55, $d$ 2.13, gold gain 1.00, already past the peak and about to fall

The proxy score, meanwhile, rises monotonically with $n$ by construction. These coefficients are illustrative, not fitted to any particular model.

How it differs from reward hacking

Reward hacking names a qualitative exploit, often a discrete trick. Overoptimization is the quantitative, gradual version: no single sample looks like cheating, but the policy drifts into regions where the reward model's errors are systematically positive, such as longer or more flattering answers.

Why it matters

It tells you that "more optimisation" has an optimum, and that the KL budget is the main dial. In practice teams hold out a stronger or human-labelled grader, plot gold against proxy over training, and stop or raise the KL penalty once the curves diverge. Reward model ensembles and fresh preference data move the peak further out but do not remove it.

Related terms

• Reward model
• Goodhart's law
• Reward hacking
• KL penalty
• Best-of-n sampling
• RLHF (reinforcement learning from human feedback)

Sources

1. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org