Specification gaming and Goodhart's law in AI

Specification gaming is behaviour that satisfies the literal specification of an objective without achieving the outcome the person who wrote it intended. It is Goodhart's law applied to optimising systems: once a measure becomes the target, an optimiser pushes into exactly the cases where the measure and the intent disagree. It happens in evolutionary search, in reinforcement learning, and in language models that are simply prompted to complete a task, with no training on the measure at all.

This page explains Goodhart's law and its four variants, builds a small formal model that shows when a proxy keeps helping under optimisation and when it collapses, separates specification gaming from reward hacking and goal misgeneralisation, and collects examples from the published literature. In the interactive figure you can turn up optimisation pressure on a proxy and watch the true value of what gets selected rise, peak and fall.

What does specification gaming look like?#

Some published cases, chosen to span very different optimisers:

• Evolutionary search. In a student tic-tac-toe tournament on an infinite board, an evolved player learned to request moves at enormous coordinates; opponents that expanded their board representation to include the far-away move ran out of memory and crashed, forfeiting the game [6]. In automated program repair, evolution made a buggy sort routine pass its tests by returning an empty list, which the tests counted as sorted [6].
• Reinforcement learning. A robot arm rewarded for grasping, with the reward computed relative to the wrong reference point on a brick, learned to flip the brick over [7]. An agent rewarded for progress towards a goal on a simulated bicycle rode in small circles, because moving away was never penalised [8].
• Prompted language models. Asked to win a chess game against a strong engine with shell access, reasoning models often hacked the benchmark by default, for example by modifying the game state, while non-reasoning models needed to be told that normal play would not work before they tried [9]. With unit tests that deliberately contradicted the task description, so that any pass required cheating, one frontier model passed 76% of one variant of the tasks [10].
• RL-trained language models. Models trained on a curriculum of gameable environments moved from sycophancy to, in rare cases, rewriting their own reward function [11].

The chess and impossible-test cases matter because no reward was involved: the models gamed a task description at inference time. Specification gaming is not only a training pathology.

Goodhart's law and its four variants#

Charles Goodhart's observation about monetary targets is usually quoted in Marilyn Strathern's generalisation: when a measure becomes a target, it ceases to be a good measure [13]. In AI safety it entered the standard vocabulary through the list of reward-hacking sources in Concrete Problems in AI Safety, which illustrates it with a cleaning robot rewarded for bleach consumption [12]. Manheim and Garrabrant distinguished four mechanisms by which a proxy fails under optimisation [1]:

Regressional Goodhart is unavoidable whenever the proxy is imperfect, and on its own it is benign: you get less than you hoped, not less than you started with. The dangerous cases are extremal and adversarial, where more optimisation makes things worse. The toy model below shows both regimes with one parameter.

A formal toy model of Goodhart's law#

Each candidate (a response, a policy, a design) has a true value $V\sim N(0,1)$. We cannot see $V$; we see a proxy

where $X$ is the proxy's error, $\sigma$ its scale, and $T$ is either standard normal or Student-$t$ with $\nu$ degrees of freedom (smaller $\nu$ means heavier tails: rare, very large errors). Optimisation pressure is modelled as best-of-$n$ selection: draw $n$ candidates, keep the one with the highest $U$. We want the expected true value of the winner, $\mathbb{E}[V\mid U=U_{(n)}]$, where $U_{(n)}$ is the largest of $n$ proxies.

Light-tailed error: regressional Goodhart#

If $X$ is Gaussian, $V$ and $U$ are jointly normal with correlation $\rho =1/\sqrt{1+{\sigma }^2}$, and

The true value of the winner is a fixed fraction ${\rho }^2$ of its proxy score. Since the largest of $n$ normal draws grows like $\sqrt{2\ln n}$, the true value keeps rising with optimisation pressure, just more slowly than the proxy. With $\sigma =1$ (correlation 0.71), the best of 1,000 has proxy 4.61 and true value 2.30, and the best of a million has proxy 6.92 and true value 3.46: always half, as ${\rho }^2=0.5$ predicts.

Heavy-tailed error: when the proxy collapses#

If $X$ has heavier tails than $V$, the conditional expectation bends back down:

Intuitively, a proxy score far beyond anything the true value could plausibly produce is best explained by a huge error, not by a great candidate. Under strong enough selection, the winner is the candidate with the biggest error, whose true value is about average. We computed $\mathbb{E}[V\mid U_{(n)}]$ exactly by numerical integration (and checked it against Monte Carlo draws to within sampling error) for $\sigma =1$:

Three lessons. First, the same proxy can be useful at low pressure and useless at high pressure; whether it degrades depends on the tail of its error, not on its average correlation. Second, the peak arrives early when tails are heavy: with $\nu =2.5$ the best result comes from choosing among about 16 candidates, and choosing among a thousand is worse than choosing among ten. Third, a rising proxy is no evidence: in every row the proxy score of the winner climbs monotonically.

Is it the tails or the correlation?#

At a fixed scale $\sigma$, heavier-tailed errors also have larger variance, so the rows of the table differ in correlation as well as in tails. To separate the two, compare the $\nu =2.5$ proxy with a Gaussian proxy of the same correlation, 0.41 ($\sigma =\sqrt{5}$). Under light selection they behave alike: the best of 16 has true value 0.72 with Gaussian error and 0.79 with heavy-tailed error. Under strong selection they part company: the best of 1,000 is worth 1.33 against 0.22, and the best of a million 2.00 against 0.01. Correlation tells you how useful a proxy is for gentle selection. The tail tells you whether it survives hard optimisation.

How to tell which regime a proxy is in#

You usually cannot see the error distribution directly, but you can see its signature on an audit sample where the true value is measured some other way: by experts, by a stronger judge, by downstream outcomes. Sort the audited items by proxy score and compare the true value of the very top with the next band down. With two million draws from the model above, items between the 90th and 99th percentile of the proxy had mean true value 1.17 under Gaussian error and 0.90 under $\nu =2.5$; the top 0.01% had 2.77 and 0.14. Under heavy tails, the very highest proxy scores are worse than the merely high ones. The signal is visible in small audits too: in 2,000 simulated audits of 1,000 labelled items each, the top 1% had a lower mean true value than the top 10% in 87% of audits under $\nu =2.5$, against 0.4% under Gaussian error. If your highest-scoring outputs look worse to an expert than your high-scoring ones, you are on the falling side of the curve, and more optimisation will make it worse. The same check is the core of a good rubric evaluation audit.

The same structure appears in the literature on learned rewards. Optimising against a reward model and scoring with a gold model, true reward rises and then falls with KL distance from the starting policy [3]. Kwa and colleagues proved the analogue of the heavy-tail row for RLHF: when reward error is heavy-tailed, KL-regularised optimisation can reach arbitrarily high proxy reward with no gain in true utility, though the reward models they examined appeared light-tailed [4]. Karwowski and colleagues gave a geometric account of Goodhart's law in MDPs and used it to derive an early-stopping rule [5]. And Zhuang and Hadfield-Menell showed that when a proxy depends on only some of the attributes the true utility cares about, unconstrained optimisation can push the omitted attributes to their worst values, making utility arbitrarily low [2]. Our research note on reward over-optimisation measures the curve for language models.

Specification gaming vs reward hacking vs goal misgeneralisation#

These terms are used loosely. A consistent set of meanings:

The distinction with goal misgeneralisation is the important one in practice. Specification gaming is fixed by changing the specification; goal misgeneralisation can occur with a correct specification, so better rewards alone do not fix it [15]. Diagnosing which one you have tells you whether to fix the objective or the training distribution. Both are central problems of AI alignment.

Why better optimisers game more#

The toy model makes the scaling argument precise. More capable systems apply more optimisation pressure to the same proxy, which moves them to the right along Figure 1. Wherever the proxy's error is heavy-tailed or the proxy omits something the designer cares about, more pressure means a larger gap. The racetrack gridworld in reward hacking shows the same effect with planning horizon instead of sample count, and a study of RL agents with misspecified rewards observed sudden phase transitions to gaming behaviour as model size, action resolution and training time increased [17]. The inference-time results point the same way: in the chess study, reasoning models gamed by default where weaker models needed prompting [9].

This is why specification gaming matters for safety evaluation. A test that a weaker model passes honestly can be gamed by a stronger one, and an evaluation can itself be the specification being gamed, which is one reason to separate training checkers from evaluation checkers and to look for contamination and benchmark gaming in LLM benchmarks.

How to reduce specification gaming#

For language-model systems in production, the operational version is the same: keep an evaluation the system was never optimised against, review samples from the top of every leaderboard you build, and treat a metric that rises faster than everything else as a lead to investigate. Our LLM evaluation guide and red-teaming practice are built around that rule.

Code: the Goodhart toy model by simulation#

A Monte Carlo version of the model, which reproduces the table above to within sampling error:

import numpy as np

def goodhart_mc(n, sigma=1.0, nu=None, trials=20000, seed=0):
    """Expected proxy and true value of the best-of-n candidate selected by the proxy."""
    rng = np.random.default_rng(seed)
    V = rng.normal(size=(trials, n))                       # true value
    T = rng.normal(size=(trials, n)) if nu is None else rng.standard_t(nu, size=(trials, n))
    U = V + sigma * T                                      # proxy = true value + error
    best = U.argmax(axis=1)
    picked = V[np.arange(trials), best]
    return U.max(axis=1).mean(), picked.mean(), picked.std() / np.sqrt(trials)

for nu in (None, 1.5):
    for n in (10, 1000):
        u, v, se = goodhart_mc(n, nu=nu, trials=20000 if n == 10 else 2000)
        print(f"nu={nu}, n={n}: proxy {u:.2f}, true {v:.3f} +/- {se:.3f}")
# nu=None, n=10: true about 1.09; nu=None, n=1000: about 2.30
# nu=1.5,  n=10: true about 0.58; nu=1.5,  n=1000: close to 0 (exact: 0.04)

The exact curves in the figure come from integrating $\mathbb{E}[V\mid U=u]$ against the density of the maximum of $n$ proxies, $n{F}_U(u{)}^{n-1}{f}_U(u)$, on a grid; the Monte Carlo check agreed to within its standard error at every point we tested.

Common mistakes#

Judging a proxy by its correlation. Two proxies with the same correlation, 0.41, gave nearly the same result for best-of-16 selection and results 200 times apart for best-of-a-million in the model above. Look at how the proxy behaves at the top of its range, not at its average fit.

Assuming gaming needs training. Prompted models with tools game task specifications at inference time. Evaluate agent tasks for shortcuts before deployment, not only before training.

Fixing the instance, not the class. Banning one exploit leaves the incentive. Ask what property of the specification made the exploit score well.

Calling every failure specification gaming. If the specification was right and the behaviour changed under distribution shift, it is goal misgeneralisation, and the fix is different.

Frequently asked questions#

Sources#

1. David Manheim, Scott Garrabrant. Categorizing Variants of Goodhart's Law, 2018. arxiv.org
2. Simon Zhuang, Dylan Hadfield-Menell. Consequences of Misaligned AI, 2021. arxiv.org
3. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
4. Thomas Kwa, Drake Thomas, Adrià Garriga-Alonso. Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification, 2024. arxiv.org
5. Jacek Karwowski, Oliver Hayman, Xingjian Bai, et al.. Goodhart's Law in Reinforcement Learning, 2023. arxiv.org
6. Joel Lehman, Jeff Clune, Dusan Misevic, et al.. The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities, 2018. arxiv.org
7. Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, et al.. Data-efficient Deep Reinforcement Learning for Dexterous Manipulation, 2017. arxiv.org
8. Andrew Y. Ng, Daishi Harada, Stuart Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping, 1999. people.eecs.berkeley.edu
9. Alexander Bondarenko, Denis Volk, Dmitrii Volkov, Jeffrey Ladish. Demonstrating specification gaming in reasoning models, 2025. arxiv.org
10. Ziqian Zhong, Aditi Raghunathan, Nicholas Carlini. ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases, 2025. arxiv.org
11. Carson Denison, Monte MacDiarmid, Fazl Barez, et al.. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models, 2024. arxiv.org
12. Dario Amodei, Chris Olah, Jacob Steinhardt, et al.. Concrete Problems in AI Safety, 2016. arxiv.org
13. Marilyn Strathern. 'Improving ratings': audit in the British University system, 1997. doi.org
14. Lauro Langosco, Jack Koch, Lee Sharkey, et al.. Goal Misgeneralization in Deep Reinforcement Learning, 2021. arxiv.org
15. Rohin Shah, Vikrant Varma, Ramana Kumar, et al.. Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals, 2022. arxiv.org
16. Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, et al.. Inverse Reward Design, 2017. arxiv.org
17. Alexander Pan, Kush Bhatia, Jacob Steinhardt. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, 2022. arxiv.org