Goodhart's law says that when a measure becomes a target, it ceases to be a good measure. In machine learning it describes why optimising a proxy, such as a reward model score or a benchmark, eventually stops improving the thing the proxy was meant to track.
The idea comes from economist Charles Goodhart's 1975 observation that statistical regularities used for monetary policy tended to break down once policy relied on them; the snappy phrasing above is a later paraphrase usually credited to the anthropologist Marilyn Strathern.
Four variants
Manheim and Garrabrant (2018) separate four mechanisms, each with a different fix:
- Regressional: the proxy equals the goal plus noise, so selecting for extreme proxy values also selects for lucky noise.
- Extremal: the proxy-goal relationship holds in the normal range but breaks at the extremes optimisation pushes into.
- Causal: the proxy is correlated with the goal but does not cause it, so intervening on the proxy does nothing.
- Adversarial: another agent, or the optimised policy itself, exploits the proxy; this is reward hacking.
Worked example: regressional Goodhart
Let true quality and noise be independent standard normals and let the proxy be . Then
Pick the top 1% of items by proxy. In a seeded simulation with a million items, the selected items averaged 3.78 on the proxy but only 1.89 on true quality: half the apparent advantage was noise. Nothing adversarial happened; selection alone did it. More selection pressure widens the absolute gap.
Where it shows up in ML
- RLHF against a learned reward model, measured precisely as reward overoptimization.
- Benchmark-driven development, where repeated selection on a public test set inflates scores; compounded by data contamination.
- Choosing the best of many checkpoints or prompts on one small eval set.
Why it matters
Every training objective and every eval is a proxy. Goodhart's law predicts that the harder you optimise one, the less its score means, so the defence is structural: keep held-out measures that are never optimised, use several proxies that fail differently, limit optimisation strength (for example with a KL penalty), and treat large score gains as claims that need independent confirmation.