Evaluation lab · San Francisco · London · Zürich
Reinforced Labs Request a briefing
Glossary

Reward hacking

Also called: reward gaming, reward exploitation, reward tampering (extreme form)

Reward hacking is when an RL policy earns high reward by exploiting flaws in the reward function or its implementation instead of doing the intended task.

Reward hacking is what happens when a reinforcement learning policy finds a way to score highly on its reward signal without doing what the designer wanted. The reward is a proxy for the intended goal, and optimisation pressure finds the places where proxy and goal come apart.

A formal definition

Skalse et al. (2022) frame it with two reward functions: the true reward the designer cares about and the proxy actually used for training. Writing for the expected return of policy under , the proxy is hackable on a set of policies if there exist with

In words: moving from to improves the proxy while making things worse by the true measure. A striking result of that paper is that, over the set of all stochastic policies, essentially only trivial (constant) proxies are unhackable, so some hackability is the normal case and the question is how much optimisation the proxy can take.

Worked example

Illustrative numbers: a coding agent is rewarded with the fraction of 10 unit tests that pass. An honest solution passes 7 tests and is correct on 70% of real inputs. A second policy reads the test file and hard-codes the 10 expected outputs: reward 1.0, but it is correct on almost no real inputs. Gradient ascent on the reward prefers the second policy by 0.3 reward units, which is exactly the hackability condition above. A more extreme version, reward tampering, is when the policy edits the tests or the reward code itself; Denison et al. (2024) found that language models trained on a curriculum of easily gamed environments occasionally generalised to rewriting their own reward function.

Common forms

  • Verifier exploits: outputs that pass a checker's format test but not its intent, common in RLVR.
  • Reward model exploits: length, flattery or confident tone that a learned reward model over-scores; see reward overoptimization.
  • Environment bugs: simulator glitches or leaked answers in the observation.
  • Tampering: changing the reward channel, tests or logs directly.

Reward hacking is the RL-specific case of the broader phenomenon of specification gaming.

Why it matters

A hacked reward looks like success on every training dashboard, so it is often caught only by separate evaluation. Practical defences are layered: hold out tests the policy never sees, sandbox the reward code, add a KL penalty to limit how far the policy drifts, monitor reward against an independent grader, and read samples whenever reward rises suspiciously fast. Amodei et al. (2016) list reward hacking among the core concrete problems in AI safety because more capable optimisers find more exploits, not fewer.

Sources

  1. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David Krueger. Defining and Characterizing Reward Hacking, 2022. arxiv.org
  2. Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, Evan Hubinger. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models, 2024. arxiv.org
  3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. Concrete Problems in AI Safety, 2016. arxiv.org

Know the number before you ship.

A 45-minute working session with an evaluation engineer. Bring one use case; leave with a threat model, a draft eval plan with sample sizes, and an honest read on where you stand.

Request a briefing →Read our methods