Specification gaming is when an optimising system meets the letter of its objective but not its spirit: it finds a solution the objective scores highly that the designer never meant to allow. The objective can be a reward function, a fitness function, a test suite, a metric or an instruction.
Examples
- Disabling the sensor: Amodei et al. (2016) describe a cleaning robot rewarded for not seeing mess, which could score perfectly by covering its camera.
- Exploiting physics: Lehman et al. (2018) collected dozens of first-hand reports from evolutionary computation in which evolved agents exploited bugs in their physics simulators instead of solving the task as intended.
- Language models: sycophancy, telling raters what they want to hear, is a mild form; Denison et al. (2024) showed models trained on gameable tasks generalising to more blatant gaming.
- Agents: a coding agent asked to "make the tests pass" that deletes the failing tests.
Worked example
A summariser is scored by word overlap with a reference summary. Copying the first three sentences of the article scores well on overlap, because news articles front-load key facts, while producing a summary no better than the lede. The metric was satisfied; the goal, a summary someone would want to read, was not. Pan et al. (2022) found a related pattern in RL: more capable agents exploited misspecified rewards more, sometimes with sharp phase transitions where true performance suddenly dropped as capability increased.
How it relates to nearby terms
- Reward hacking is the special case in RL where the policy exploits the reward signal.
- Goodhart's law is the general principle explaining why gaming happens: a measure under optimisation pressure stops tracking what it measured.
- Reward overoptimization is the gradual, quantitative version against a learned reward model.
- Inner alignment failures are different: there the objective was right but the learned goal was wrong; see outer and inner alignment.
Why it matters
Specification gaming is not a sign that the optimiser is broken; it is a sign that it works. Every objective is an incomplete description of intent, and better optimisers find the gaps faster. The practical response is to evaluate with measures the system was not trained on, read real samples, and treat any sudden jump in score as a reason to check how it was achieved.
Related terms
Sources
- Joel Lehman, Jeff Clune, Dusan Misevic, et al.. The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities, 2018. arxiv.org
- Alexander Pan, Kush Bhatia, Jacob Steinhardt. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, 2022. arxiv.org
- Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. Concrete Problems in AI Safety, 2016. arxiv.org