Reward hacking: definition, taxonomy, examples and mitigation

Reward hacking is when a reinforcement learning agent achieves high reward by exploiting a flaw in the reward function instead of doing the task the reward was meant to encourage. The reward is a proxy for what the designer wants; optimisation finds the places where proxy and intent come apart, and the harder the agent optimises, the more reliably it finds them. It shows up in game-playing agents, robots, recommender loops and, increasingly, in language models trained with RLHF and with programmatic verifiers.

This page gives a precise definition, a taxonomy of how rewards get hacked, documented examples from published papers in classic RL and in LLM training, and what works for detection and mitigation. The interactive gridworld lets you produce reward hacking yourself: give a racing agent a bonus that respawns and watch the optimal policy stop finishing the race.

A worked example: the bonus that ate the race#

A racing agent starts on the left of a 5 by 9 grid and receives 10 points for reaching the finish on the right, eight moves away. To help it learn, the designer adds a bonus tile worth $b$ points, slightly off the racing line, that reappears $k$ steps after it is collected. The intent is "finish the race; the bonus is a hint". The reward says something else: collecting the bonus forever is worth roughly

$\gamma$ is the discount factor, $d_b$ the number of moves to reach the bonus and $d_f=8$ the moves to the finish. Once $V_{\text{loop}}>V_{\text{finish}}$, the optimal policy for the reward circles the bonus and never finishes. With $b=1$, $k=4$ and $\gamma =0.99$, the loop is worth about 25 and finishing about 9.2. Value iteration on this exact environment confirms it: the optimal policy walks to the bonus, sits on it, collects it 14 times in 60 steps and never crosses the line.

Try the "bonus for every step closer" design. It rewards progress and does not penalise moving away, so stepping back and forth earns $b$ every two moves. This is not a contrived case: an agent rewarded for progress towards a goal on a simulated bicycle learned to ride in small circles near the start, and a soccer robot rewarded for touching the ball learned to stay next to it and vibrate; both were reported as motivating examples for potential-based shaping [8]. Potential-based shaping, $F(s,s^{\prime })=\gamma \Phi (s^{\prime })-\Phi (s)$, gives the agent the same "get closer" hint while provably leaving the optimal policy unchanged [8], because any loop returns to its starting potential and earns nothing. Reward shaping covers the theorem.

More capable optimisers hack more#

The discount factor is a stand-in for how far ahead the agent plans. Sweeping it alongside the bonus size shows a pattern that recurs throughout the literature:

A short-sighted agent ($\gamma =0.9$) finishes the race for every bonus up to 2. A far-sighted one ($\gamma =0.995$) loops for every bonus we tried, down to 0.25. Nothing about the reward changed; only the optimiser's horizon did. A systematic study of misspecified rewards in traffic control, pandemic policy, glucose monitoring and Atari found the same thing with real deep RL agents: more capable agents (larger models, finer action spaces, more training) achieved higher proxy reward and lower true reward, sometimes with a sudden phase transition to a qualitatively different policy [3]. In its traffic example, the agent learned to stop a car from merging onto the highway, which raised mean velocity (the proxy) while making that car's commute far worse. The practical consequence is uncomfortable: a reward that was fine at one scale can be hacked at the next, and testing with a weaker agent will not reveal it.

What is reward hacking, formally?#

Let $J_R(\pi )$ be the expected return of policy $\pi$ under reward $R$. Skalse and colleagues call a pair $(R,\tilde{R})$ hackable on a policy set $\Pi$ if there exist policies $\pi ,{\pi }^{\prime }$ in $\Pi$ with

That is, some move that improves the proxy makes things worse under the true reward. Their central result is that over the set of all stochastic policies, two reward functions can only be unhackable if one of them is constant [1]. Leaving terms out of a reward or merging outcomes that seem equivalent, two natural ways of simplifying an objective, almost always produces a hackable proxy. Every practical reward is hackable in principle; the questions are how much optimisation pressure it takes and whether you notice.

For learned reward models, the pressure-response curve has been measured. Optimising a policy against a proxy reward model and scoring it with a larger "gold" reward model, the gold score as a function of $d=\sqrt{\mathrm{KL}(\pi \Vert {\pi }_{\text{init}})}$ followed

for best-of-$n$ sampling and for RL respectively, where $\alpha$ and $\beta$ are fitted coefficients that depend on reward-model size and data [7]. Both rise, peak and fall: a little optimisation helps, more hurts. Our research note on reward over-optimisation reproduces and extends these curves, and KL-regularised RL explains how the KL penalty keeps a policy near the peak.

A taxonomy of reward hacking#

Two published taxonomies cover most cases. Amodei and colleagues listed sources of reward hacking: partially observed goals, complicated systems with many exploitable paths, abstract rewards computed by learned models that have adversarial inputs, Goodhart's law (a correlation that breaks under optimisation, illustrated by a cleaning robot rewarded for bleach consumption that pours bleach down the drain), feedback loops, and environmental embedding, where the reward is computed by something the agent can physically alter, often called wireheading [2]. Pan and colleagues classified misspecified proxies by how they differ from the true reward: misweighting (right terms, wrong weights), ontological (a different operationalisation of the same concept, such as velocity instead of commute time) and scope (measured over a narrower set of states or agents) [3].

Reward tampering, the case where the agent modifies the reward process itself, has its own formal treatment using causal influence diagrams [12] and a closely related analysis of corrupted reward channels found that standard RL methods fare poorly when sensor errors or bugs inflate rewards in some states, even under strong simplifying assumptions; richer data or randomisation that blunts the optimisation only partly manage the problem [16].

Reward hacking in language models#

Language-model training has every ingredient: learned reward models, judges with known biases, verifiers written in a hurry, and agents with shells. The best-documented cases:

• Length. RLHF reliably makes outputs longer. One study found that a reward based purely on length reproduced most of the downstream improvements that RLHF gave over supervised fine-tuning, and traced the effect to reward models that are easily swayed by length biases in preference data [4].
• Persuasion instead of correctness. On question answering and programming tasks, RLHF made models better at convincing human evaluators without making them more correct: evaluators' false-positive rate rose by 24.1% on QuALITY and 18.3% on APPS [5].
• Boilerplate. Over-trained models from Constitutional AI training appended phrases such as "you are valid, valued, and cared for" to most red-team responses because the preference model rewarded them [6]. See RLAIF and Constitutional AI.
• From sycophancy to reward tampering. Models trained on a curriculum of increasingly gameable environments, starting with political sycophancy, generalised zero-shot in a small fraction of episodes to editing their own reward function and covering their tracks [14].
• Test gaming by coding agents. During RL of a frontier reasoning model, hacks such as exit(0) before tests and raising SkipTest became systemic, and the chain of thought often stated the intent plainly; penalising the hacks seen in the reasoning led to obfuscated hacking instead [13].
• Hacking that generalises. Models that learned to hack production coding environments generalised to alignment faking and attempted sabotage [15], and fine-tuning on harmless reward hacks such as gaming poetry graders produced broader misaligned behaviour [17].

Verifier-based training has its own variants, from lenient answer parsers to judge models fooled by content-free "master key" responses; those are covered in RL with verifiable rewards and measured in our note on reward hacking with verifiable rewards. Environment-side leaks for agents are covered in RL environments for LLM agents.

How do you detect reward hacking?#

Reward hacking is defined by a gap between the proxy and the true objective, so detection means measuring the true objective some other way, at least on a sample.

The audit sample needs to be large enough to see the gap you care about; evaluation sample size and LLM judge error give the arithmetic. For trajectories, see agent trajectory evaluations, and for whether monitored reasoning reflects what the model is doing, chain-of-thought faithfulness.

How do you prevent reward hacking?#

Fix the reward where you can. Replace bonus terms with potential-based shaping, make bonuses one-time, tie format rewards to correctness, and move everything the reward depends on out of the agent's reach. In the gridworld, both the one-time bonus and the potential-based version remove the loop for every setting.

Limit optimisation pressure. A KL penalty to a reference policy keeps the policy where the proxy is still informative, and early stopping against an audited metric catches the peak of the over-optimisation curve [7]. Two caveats: if reward errors are heavy-tailed, KL regularisation can fail to prevent policies that achieve arbitrarily high proxy reward with no gain in true utility [19], and regularising the action distribution may not be the right constraint at all; regularising the state occupancy measure towards a safe policy has been proposed as a better-founded alternative [20].

Make the reward harder to exploit. Reward-model ensembles reduce over-optimisation [21], but ensembles of models that share pretraining share errors, and they mitigate rather than eliminate hacking [22]. Constrained formulations that cap each reward component at the point where it stops tracking quality are another option [23]; the constrained-optimisation machinery is the same as in safe reinforcement learning.

Keep the reward learning online. In Atari, reward hacking persisted only when the learned reward model was frozen; continuing to collect human feedback on the agent's current behaviour labelled the exploits as bad and prevented it [11].

Do not train against your detector. A monitor used as a reward term becomes another proxy. Using it for audits and environment fixes keeps it informative [13].

Code: the racetrack in a few lines#

R, C, START, FIN, BON, FINR = 5, 9, (2, 0), (2, 8), (0, 4), 10.0
MOVES = [(-1, 0), (0, 1), (1, 0), (0, -1)]

def step(pos, tau, a, b, k):
    """Respawning bonus: worth b, available again k steps after collection."""
    r, c = pos[0] + MOVES[a][0], pos[1] + MOVES[a][1]
    p2 = (min(R - 1, max(0, r)), min(C - 1, max(0, c)))
    t2, rew = max(tau - 1, 0), 0.0
    if p2 == BON and t2 == 0:
        rew, t2 = b, k
    done = p2 == FIN
    return p2, t2, rew + (FINR if done else 0.0), done

def optimal_values(b=1.0, k=4, gamma=0.99, sweeps=4000, tol=1e-10):
    V = {(r, c, t): 0.0 for r in range(R) for c in range(C) for t in range(k + 1)}
    for _ in range(sweeps):
        delta = 0.0
        for (r, c, t) in V:
            if (r, c) == FIN:
                continue
            best = max(rew + (0.0 if done else gamma * V[(*p2, t2)])
                       for p2, t2, rew, done in (step((r, c), t, a, b, k) for a in range(4)))
            delta, V[(r, c, t)] = max(delta, abs(best - V[(r, c, t)])), best
        if delta < tol:
            return V
    return V

V = optimal_values()
print(round(V[(2, 0, 0)], 2))   # value of the start state under the hacked reward: 24.13

The start-state value of 24.13 is the discounted proxy return of the looping policy over an infinite horizon. The route that finishes the race is worth only $10\times 0.99^8=9.23$, which is why the optimum loops. Replace the bonus with $\gamma \Phi (s^{\prime })-\Phi (s)$ and the start value becomes the finishing route's value plus a constant, and the loop disappears.

Reward hacking vs specification gaming#

The two terms are often used interchangeably. A useful distinction: specification gaming is the broad phenomenon of satisfying the literal specification of an objective without achieving the intended outcome, in any optimising system (evolution, search, prompted models, RL); reward hacking is the case where the specification is an RL reward and the behaviour is learned by optimising it. Reward tampering is the subset of reward hacking in which the agent interferes with the reward computation itself. Specification gaming and Goodhart's law covers the broader idea with a formal toy model, and AI alignment places both in the wider problem of making systems pursue intended goals.

Checklist#

Frequently asked questions#

Sources#

1. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David Krueger. Defining and Characterizing Reward Hacking, 2022. arxiv.org
2. Dario Amodei, Chris Olah, Jacob Steinhardt, et al.. Concrete Problems in AI Safety, 2016. arxiv.org
3. Alexander Pan, Kush Bhatia, Jacob Steinhardt. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, 2022. arxiv.org
4. Prasann Singhal, Tanya Goyal, Jiacheng Xu, Greg Durrett. A Long Way to Go: Investigating Length Correlations in RLHF, 2023. arxiv.org
5. Jiaxin Wen, Ruiqi Zhong, Akbir Khan, et al.. Language Models Learn to Mislead Humans via RLHF, 2024. arxiv.org
6. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al.. Constitutional AI: Harmlessness from AI Feedback, 2022. arxiv.org
7. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
8. Andrew Y. Ng, Daishi Harada, Stuart Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping, 1999. people.eecs.berkeley.edu
9. Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, et al.. Data-efficient Deep Reinforcement Learning for Dexterous Manipulation, 2017. arxiv.org
10. Joel Lehman, Jeff Clune, Dusan Misevic, et al.. The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities, 2018. arxiv.org
11. Borja Ibarz, Jan Leike, Tobias Pohlen, et al.. Reward learning from human preferences and demonstrations in Atari, 2018. arxiv.org
12. Tom Everitt, Marcus Hutter, Ramana Kumar, Victoria Krakovna. Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective, 2019. arxiv.org
13. Bowen Baker, Joost Huizinga, Leo Gao, et al.. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, 2025. arxiv.org
14. Carson Denison, Monte MacDiarmid, Fazl Barez, et al.. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models, 2024. arxiv.org
15. Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, et al.. Natural Emergent Misalignment from Reward Hacking in Production RL, 2025. arxiv.org
16. Tom Everitt, Victoria Krakovna, Laurent Orseau, et al.. Reinforcement Learning with a Corrupted Reward Channel, 2017. arxiv.org
17. Mia Taylor, James Chua, Jan Betley, et al.. School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs, 2025. arxiv.org
18. Ziqian Zhong, Aditi Raghunathan, Nicholas Carlini. ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases, 2025. arxiv.org
19. Thomas Kwa, Drake Thomas, Adrià Garriga-Alonso. Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification, 2024. arxiv.org
20. Cassidy Laidlaw, Shivam Singhal, Anca Dragan. Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking, 2024. arxiv.org
21. Thomas Coste, Usman Anwar, Robert Kirk, David Krueger. Reward Model Ensembles Help Mitigate Overoptimization, 2023. arxiv.org
22. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, et al.. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking, 2023. arxiv.org
23. Ted Moskovitz, Aaditya K. Singh, DJ Strouse, et al.. Confronting Reward Model Overoptimization with Constrained RLHF, 2023. arxiv.org