Reward model overoptimization: why gold reward peaks while the proxy keeps climbing

Every reward model is a proxy. Push a policy hard enough against it and the proxy score keeps climbing while the quality you actually care about rises, peaks and then falls. This note shows where that hump comes from, how its shape differs between best-of-n sampling and RL, why measuring "how far you optimised" in KL is both useful and treacherous, and what ensembles and KL penalties really buy you. Every number below comes from a seeded simulation of a small toy model that we wrote to make the mechanics visible; the code is described inline and the interactive figure runs the same model in your browser.

A concrete example: one prompt, two features#

Imagine a prompt whose possible answers differ in two ways. One is genuine quality $q$: accuracy, relevance, actual helpfulness. The other is a stylistic feature $s$, think length or confident hedging, that a careful human rater is indifferent to in normal amounts but dislikes at the extremes. Under the base policy both are standard normal and independent.

The reward model was trained on preferences where longer, more elaborate answers tended to win, so it learned to reward $s$ a little, and like any learned model it also has idiosyncratic errors. We give it a weight of 0.8 on $s$ and an error term with standard deviation 0.5. The "gold" reward, standing in for careful human judgment, is $q-s^2/2$.

Now run best-of-n: draw $n$ answers, keep the one the reward model likes most. In our simulation, best-of-45 raises gold reward by 0.91 (in units of the base policy's quality standard deviation) and proxy reward by 3.04. That is the best it gets. At best-of-1,024 the proxy gain has grown to 4.47 but the gold gain has fallen to 0.72, and at best-of-16,384 it is down to 0.37 while the proxy says 5.46. More selection pressure was spent; the answers got worse. Nothing about this toy is exotic: one modest spurious association and a little noise are enough. It is a mild, measurable case of what Skalse et al. formalise as reward hacking [2].

The phenomenon is old. Stiennon et al. saw summaries get worse by human judgment once a policy was optimised far enough against a learned reward model [3], and Gao, Schulman and Hilton measured it systematically with a large "gold" reward model standing in for humans [1]. The toy adds a setting where every quantity is computable exactly.

Gold, proxy and the KL budget#

Write ${\pi }_0$ for the base policy (in practice the supervised fine-tuned model), $R_{\text{gold}}$ for the reward you care about and $R_{\text{proxy}}$ for the learned reward model. Any optimisation procedure produces a new policy $\pi$, and the natural measure of how far it moved is the KL divergence from the base policy. Following Gao et al., we use its square root:

Here $y$ is a response, $\pi (y)$ and ${\pi }_0(y)$ are its probabilities under the optimised and base policies, and $d$ is measured in square-root nats. The square root matters because KL behaves like a squared distance: a policy whose mean shifts by $\delta$ standard deviations has KL of about ${\delta }^2/2$, so $d$ grows roughly linearly with "how many standard deviations of the base distribution you moved".

Best-of-n and the KL-regularised optimum#

For best-of-n with continuous, untied rewards, the KL from the base policy has a closed form:

Here $n$ is the number of samples drawn. It comes from averaging the log of the reweighting $nF(r{)}^{n-1}$, where $F$ is the proxy reward's cumulative distribution under ${\pi }_0$. KL grows only logarithmically in $n$, which is why best-of-n is so cheap in KL: $n=45$ costs 2.83 nats and $n=1,024$ costs 5.93.

RL with a KL penalty maximises ${\mathbb{E}}_{\pi }[R_{\text{proxy}}]-\beta \mathrm{KL}(\pi \Vert {\pi }_0)$. If the optimiser were perfect, the solution is known in closed form [10]:

Here $\beta$ is the KL coefficient and $Z_{\beta }$ is the normalising constant that makes the probabilities sum to one. The base policy is tilted towards high proxy reward, more strongly as $\beta$ shrinks. Sweeping $\beta$ traces a path of policies; we call that path the KL-optimal frontier, because no policy gets more expected proxy reward for the same KL.

The two functional forms#

Gao et al. found that gold reward as a function of $d$ is well described by different forms for the two methods [1]:

$R(d)$ is the gold reward gain over the base policy, $\alpha$ sets the initial slope (how much real improvement each unit of distance buys at first) and $\beta$ sets how quickly the proxy's errors take over. Both forms start at zero, rise and turn over. In their experiments the coefficients varied smoothly, roughly logarithmically, with reward model size, and RL needed much more KL than best-of-n to reach the same amount of optimisation and overoptimisation. They also noted that the RL form has infinite slope at the origin and is not meant to hold near $d=0$.

Where the hump comes from: a closed form#

The toy makes the mechanism exact. Let the proxy be

$q$ is quality, $s$ the stylistic feature, $b$ the reward model's spurious weight on it, $m$ a unit-variance error feature that the reward model mistakes for quality (a property of the response, not fresh noise on every call), $\sigma$ the size of that error, and $\lambda$ how strongly gold reward penalises extreme style. We use $b=0.8$, $\lambda =1$ and $\sigma =0.5$ unless stated. With Gaussian features, the tilted policy ${\pi }_{\beta }$ simply shifts each feature's mean by its proxy weight divided by $\beta$: quality moves by $1/\beta$, style by $b/\beta$, error by $\sigma /\beta$. Writing $\Vert w\Vert =\sqrt{1+b^2+{\sigma }^2}$ for the length of the proxy's weight vector, the KL is $\Vert w{\Vert }^2/(2{\beta }^2)$, and substituting $d$ gives

The first term is the real gain: of each unit of distance travelled, only the fraction aligned with $q$ helps. The second term is the cost of being dragged along the style direction, which the gold reward penalises quadratically. For our parameters $\alpha =1.029$ and $\beta =0.339$, the peak is at

Here $d^{\star }$ is where gold reward peaks, $R^{\star }$ is the peak gain and ${\beta }^{\star }$ is the KL coefficient whose optimum sits exactly at the peak. Three things fall out that are worth carrying into real work. First, in this light-tailed setting the height of the peak does not depend on $\sigma$ at all; it is set entirely by the systematic error $b$. Second, $\sigma$ moves the peak to larger KL and widens the gap between proxy and gold, because the optimiser spends part of its KL budget pushing the error feature around. Random error behaves like a KL tax. Third, the best KL coefficient depends on quantities you never observe directly, which is why it has to be tuned against gold labels rather than derived.

Simulation: best-of-n versus RL in the toy#

Setup. Because both best-of-n and the tilted policy depend on a response only through its proxy score, every quantity reduces to the one-dimensional distribution of $R_{\text{proxy}}$ and the conditional mean of gold reward given it. We compute these by numerical integration on a fine grid, sweep $n$ from 1 to 16,384 and $\beta$ from 20 to 0.12, and record KL, proxy gain and gold gain. We checked the grid against seeded Monte Carlo (20,000 best-of-n draws per point, seed 20241217): at $n=16$ the grid gives 0.876 and Monte Carlo 0.890 $\pm$ 0.011; at $n=256$, 0.847 versus 0.865 $\pm$ 0.025. The interactive figure runs the same computation in your browser.

Findings. Proxy gain is close to linear in $d$ for both methods, as Gao et al. also observed. Gold gain rises, peaks and falls.

RL reaches more proxy reward per nat than best-of-n, as it must, since the tilted policy is KL-optimal for the proxy by construction. Yet best-of-n reaches the higher gold peak (0.91 against 0.78). The reason is that gold reward is concave in the proxy score here: an answer with a very high proxy score is likely to be stylistically extreme. Best-of-n concentrates the proxy score in a narrow band around the order statistic it selects, while the tilted policy keeps the base distribution's full spread and so pays the concavity penalty on its upper tail. "Who uses KL more efficiently" depends on which reward you measure it with.

In Gao et al.'s PPO experiments RL instead consumed far more KL than best-of-n, and they warned that KL is an inadequate measure of optimisation when comparing methods [1]. The toy agrees from the other direction: the KL axis depends on the optimiser's path.

Fitting the scaling-law forms#

Both forms are linear in their coefficients ($R=\alpha d-\beta d^2$ or $R=\alpha d-\beta d\log d$), so ordinary least squares fits them. We fitted each curve for $0<d\le 3$:

The quadratic form fits all three curves an order of magnitude better than the logarithmic form. For the KL-optimal policy the fitted coefficients recover the closed form above (1.029 and 0.339) almost exactly. The REINFORCE row is a real policy-gradient run, described next.

A real policy-gradient run#

To check that an actual optimiser behaves like the frontier, we trained a Gaussian policy over the three response features with REINFORCE: batches of 64 sampled responses, a batch-mean baseline, learning rate 0.01, 1,500 steps, 32 seeds per setting. With no KL penalty, mean gold gain peaks at 0.79 around step 160 ($d\approx 1.53$), with a seed-to-seed standard deviation of 0.06, and has fallen to 0.19 by step 300 ($d\approx 2.87$). Seeds wander because sampling noise randomly widens or narrows the policy along the style feature, which changes gold but not proxy reward.

Mitigations, measured#

A KL penalty is early stopping with a dial#

Adding $-\eta \log (\pi (y)/{\pi }_0(y))$ to each sampled reward, as RLHF implementations do [4], gives a run that converges to ${\pi }_{\beta }$ with $\beta =\eta$. We ran it at four penalties:

Every penalised run converged onto the frontier: $\eta =2$ stopped at $d=0.49$ with gold 0.42, $\eta =1$ at 0.97 with 0.68, $\eta =0.64$ at 1.52 with 0.78, and $\eta =0.4$ overshot to 2.42 with 0.50. That is exactly the effect Gao et al. reported: in their setup the KL penalty did not improve the gold-versus-KL frontier and acted like early stopping, though they cautioned that this could be sensitive to hyperparameters [1]. The penalty is still useful: the run sits at its chosen point stably instead of drifting past the peak. But the right coefficient, 0.64 here, has to be found with gold labels.

Reward model ensembles#

Coste et al. trained ensembles of proxy reward models and optimised either the worst case across members or the mean penalised by their disagreement; for best-of-n this practically eliminated overoptimization in their setup, and for PPO it always reduced it [5]. Eisenstein et al. found that ensembles mitigate but do not eliminate reward hacking: members pretrained from different seeds generalise better than members that differ only in fine-tuning seed, and some hacks survive because every member shares the same error [6]. The toy reproduces both halves of that story once we separate shared error ($b$, learned from the same preference data by every member) from independent error ($m$, different per member).

With Gaussian error, averaging five members moves the peak only modestly (from 0.912 to 0.925 at $\sigma =0.5$; from 0.853 to 0.902 at $\sigma =1.5$). The ensemble shrinks the independent error, so the gap between proxy and gold narrows and the peak arrives earlier, but the shared spurious weight $b$ is untouched, and that is what sets the peak height. For the KL-optimal RL policy the peak is 0.78 in every Gaussian row, as the closed form predicts.

Heavy tails change the picture#

Now replace the Gaussian error with a Student t distribution with 3 degrees of freedom, scaled to the same variance. A few responses now get enormous, undeserved proxy scores, and best-of-n finds them.

Here ensembles earn their keep. At $\sigma =1.5$ the single heavy-tailed model peaks at 0.667, the five-member mean at 0.870 and the worst-of-five at 0.908. Worst-case aggregation helps most at the peak because an outlier score from one member is overruled by the others. It is not uniformly better: at $\sigma =1$ and $n=16,384$ the worst-of-five gold is 0.34 against 0.48 for the mean.

Heavy tails also break the KL-regularised optimum itself. With power-law tails, ${\mathbb{E}}_{{\pi }_0}[\exp (R_{\text{proxy}}/\beta )]$ is infinite for every $\beta$, so $Z_{\beta }$ does not exist; the interactive figure hides the RL curve in that case for this reason. Kwa, Thomas and Garriga-Alonso proved the general version: with heavy-tailed reward error, policies can reach arbitrarily high proxy reward with no more true utility than the base model even under a KL penalty, while light-tailed error is benign in the limit; they also reported that the reward models they probed looked light-tailed [7]. Ensembles of t-distributed errors keep power-law tails, so they reduce the damage at practical $n$ without restoring a finite optimum.

The best-of-n KL formula is an upper bound#

The formula $\log n-(n-1)/n$ assumes no ties. Beirami et al. showed that for discrete outputs it is only an upper bound on the true KL and proposed a better estimator [8]. Language models often put a lot of probability on a single short answer, which is where the gap opens. We computed the exact best-of-n KL for three discrete base distributions (the dominant answer, if any, placed at the median reward rank):

With one answer holding 90 percent of the mass, best-of-4 is reported as 0.64 nats but is really 0.16. Plotted against the formula's KL, peaked prompts sit too far to the right.

In practice#

Overoptimization is invisible in the proxy, whose line keeps climbing in every curve above. The only defence is to measure gold reward, or a trusted stand-in such as expert human labels or a much stronger judge validated against them, at several points along the optimisation path. The estimator below gives a whole best-of-n curve from one batch of scored samples, with no resampling.

import numpy as np
from math import comb, log

def bon_curve(proxy, gold, ns):
    """proxy, gold: arrays of shape (prompts, m), m scored samples per prompt.
    The i-th lowest proxy sample (1-based) is the best of a random n-subset
    with probability C(i-1, n-1) / C(m, n), so no resampling is needed."""
    prompts, m = proxy.shape
    order = np.argsort(proxy, axis=1)
    g_sorted = np.take_along_axis(gold, order, axis=1)
    out = []
    for n in ns:
        w = np.array([comb(i - 1, n - 1) for i in range(1, m + 1)], float) / comb(m, n)
        out.append((n, log(n) - (n - 1) / n, float((g_sorted * w).sum(1).mean())))
    return out  # (n, KL upper bound, mean gold of best-of-n)

We checked it against brute-force subsampling on the toy (400 prompts, 64 samples each): at $n=16$ it gives 0.331 against 0.333 $\pm$ 0.011 from 4,000 random subsets. Gold labels are scarce, so put error bars on these curves; with labels clustered by prompt, the interval needs the clustered treatment in our note on benchmark error bars. If the gold labels come from an LLM judge, remember that the judge has its own spurious preferences, length among them, as discussed in our note on position and verbosity bias; a judge that shares your reward model's taste for length will hide exactly the overoptimization you are looking for.

Failure modes#

Evaluating with the reward you optimised. A metric used as a training signal stops measuring quality: the regressional and extremal Goodhart effects in Manheim and Garrabrant's taxonomy [11]. Hold out a reward signal the policy never saw.

Assuming direct alignment avoids the problem. Methods such as DPO have no separate reward model, but Rafailov et al. found the same degradation pattern at higher KL budgets, often before a single epoch of training was complete [12].

Treating ensembles as independent. Members trained on the same preferences share their biases. In our toy, a five-member ensemble with Gaussian error raised the best-of-n peak by between 1 and 6 percent across the error levels we tried, because the shared term set the height.

Tuning the penalty on the proxy. A lower penalty always looks better on the proxy. The coefficient that maximises gold has to be found with gold.

Mistaking a gap for a peak. A widening proxy-gold gap does not on its own mean gold is falling. With pure random error and no shared bias ($b=0$), the toy's gold gain kept rising across our whole range, reaching 3.55 at $n=16,384$, even as the gap grew. You need the gold curve's slope, not the gap.

What the toy does not show#

The toy has three features, Gaussian (or t) marginals, a linear proxy and a fixed gold reward; real reward models are nonlinear functions of text, and real gold signals are noisy humans. The RL results use the exact KL-regularised optimum and a small REINFORCE policy rather than PPO on a language model, so they say nothing about the logarithmic RL form or the KL efficiency of real PPO runs. Reward model size and data scaling, which drive the coefficients in Gao et al., are absent. Everything here is simulation; no language model or human rater was involved.

Frequently asked questions#

Sources#

1. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
2. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David Krueger. Defining and Characterizing Reward Hacking, 2022. arxiv.org
3. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. Learning to summarize from human feedback, 2020. arxiv.org
4. Long Ouyang, Jeff Wu, Xu Jiang, et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org
5. Thomas Coste, Usman Anwar, Robert Kirk, David Krueger. Reward Model Ensembles Help Mitigate Overoptimization, 2023. arxiv.org
6. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, et al.. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking, 2023. arxiv.org
7. Thomas Kwa, Drake Thomas, Adrià Garriga-Alonso. Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification, 2024. arxiv.org
8. Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy, 2024. arxiv.org
9. Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D. Dragan, Stephen McAleer. Confronting Reward Model Overoptimization with Constrained RLHF, 2023. arxiv.org
10. Tomasz Korbak, Ethan Perez, Christopher L Buckley. RL with KL penalties is better viewed as Bayesian inference, 2022. arxiv.org
11. David Manheim, Scott Garrabrant. Categorizing Variants of Goodhart's Law, 2018. arxiv.org
12. Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, Scott Niekum. Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms, 2024. arxiv.org