Reward hacking with verifiable rewards: measuring and hardening the verifier

Reinforcement learning with verifiable rewards replaces a learned reward model with a program that checks the answer: a string match against a reference, a unit-test run, a symbolic equivalence check. It is far harder to hack than a neural reward model, which is why it became the default for reasoning post-training. Harder is not impossible. This note looks at how policies exploit verifiers (format tricks, test gaming, false positives from model-based checkers), shows with a seeded RL simulation how a modest verifier false-positive rate makes training curves rise while true accuracy falls, and describes how to measure verifier precision during training and harden the verifier before the policy finds its holes. It also includes a small real experiment: four common answer checkers run against five scripted exploit strategies.

Four verifiers, five ways to cheat#

We wrote 300 arithmetic questions with integer answers (100 additions, 100 "which is larger, $a$ or $b$?", 100 multiplications) and four answer checkers of the kind that appear in RL codebases:

• Substring match: the reference answer appears anywhere in the response.
• Last number: the last integer in the response equals the reference.
• Any box: some \boxed{...} in the response contains the reference.
• Strict: exactly one \boxed{...}, whose content is an integer equal to the reference, in a response under 4,000 characters.

Then we ran each checker on scripted responses that never attempt the problem. This is a deterministic experiment, not a simulation: the false-positive rate is simply the fraction of the 300 questions on which the checker awarded the point.

Three lessons sit in this table. A checker that looks reasonable on honest outputs (every checker scores honest right and wrong answers perfectly) can award full marks to a degenerate strategy. The "echo" strategy succeeds a third of the time on the substring checker because one question type contains its own answer, which is a property of the data, not the checker. And the strict checker that blocks every exploit also rejects every correct answer written without a box: hardening against false positives creates false negatives. Both kinds of error matter during RL, in different ways.

How policies exploit verifiers#

The term RLVR was introduced with the Tülu 3 post-training recipe, where the policy receives a fixed reward only when a verification function confirms the answer [1]. DeepSeek-R1-Zero was trained with rule-based accuracy rewards (a boxed final answer for maths, compiler and test feedback for code) plus a format reward for wrapping reasoning in think tags, and its authors state that they did not use neural outcome or process reward models because those may suffer from reward hacking in large-scale RL [2]. The group-relative policy optimisation they used was introduced in DeepSeekMath [3]. The exploits that remain fall into three families.

Format exploits. The response satisfies the parser without satisfying the task: several candidate answers, an answer hidden in a list, a reward for a think-tag template that is met by empty tags. Every parser that searches the response rather than reading one designated answer slot is exposed.

Test-case gaming. For code, the verifier is a test run, and the policy can attack the test run instead of the code. Monitoring work on a frontier reasoning model in agentic coding environments describes two hacks that became systemic during training: calling exit(0) to leave the environment before all tests ran, and raising SkipTest from outside the testing framework so tests were skipped. It also shows a chain of thought proposing to "fudge" verification by making a verify function always return true [6]. Earlier experiments showed that models trained on gameable environments can generalise to editing their own reward function [7].

Verifier false positives. Rule-based maths checkers fail mostly in the other direction, rejecting correct answers in unexpected formats, and those false negatives increasingly hinder RL as the policy improves; model-based verifiers fix many false negatives but are highly susceptible to hacking during RL, misclassifying certain response patterns as correct [4]. A systematic study of generative reward models found that "master key" responses with no real content, such as a lone colon or an opener like "Let's solve this problem step by step.", consistently elicit false-positive rewards across many judge models [5].

A model of hacking under a verifier#

To see how a false-positive rate turns into a training pathology, consider one prompt of difficulty $b$. The policy either attempts the problem honestly, and is right with probability $a=\sigma (s-b)$ for skill $s$, or emits an exploit with probability $\pi$. The verifier passes honest correct answers with probability $1-f_n$, honest wrong answers with probability $f_0$, and exploits with probability $f_x$. Exploits are never correct. The rate the training dashboard shows and the rate you care about are

$\pi$ is the exploit propensity, $a$ the honest solve rate, $f_n$ the verifier's false-negative rate, $f_0$ its false-positive rate on honest wrong answers and $f_x$ its false-positive rate on the exploit. With a group-relative policy gradient, a sample's advantage is its reward minus the mean reward of its group, so the exploit is reinforced on a prompt whenever it pays more than honest effort does:

The right-hand side is the expected reward of an honest attempt. On easy prompts it is close to 1 and no exploit with $f_x<1$ can compete. On prompts the policy cannot yet solve it is close to $f_0$, and almost any exploit wins. Hacking therefore starts on the hardest prompts, where honest reward is lowest, and spreads as far as the inequality allows. Raising skill shrinks the region; raising $f_x$ grows it.

Simulated training run#

We simulated this with a pool of 512 prompts whose difficulties are spread evenly over a normal distribution (mean 1, SD 2.5), a skill parameter that saturates at $s=2$ to represent a capability ceiling, and an exploit propensity $\sigma (e_0+e_{1}b)$ that the policy can make depend on difficulty. The exploit starts rare ($e_0=-5$, under 1% of outputs). Each step samples 32 prompts with 8 responses each, scores them with the verifier ($f_n=0.05$, $f_0=0.01$) and applies a group-relative REINFORCE update; 300 steps per run, fixed seed. The figure below re-runs the same simulation in your browser.

All runs start at 36.9% true accuracy. Three features are worth noting. First, there is a threshold: at $f_x=0.1$ the exploit never beats honest effort on enough prompts to spread, while at 0.2 it takes over a fifth of outputs. Second, the dashboard ranks the runs backwards. The worse the verifier, the higher the measured accuracy: 68.2% for the most exploitable verifier against 59.0% for the exact one, whose reading sits below its true accuracy because of its 5% false negatives. Third, hacking starts early, within the first 22 to 43 steps, long before anyone would look at samples by hand.

Patching works if it is done. With $f_x=0.3$ and the verifier fixed at step 150 (exploit pass rate down to 0.01), the measured rate drops overnight from 60.9% to 52.1%, which looks like a regression on the dashboard. The exploit share falls from 30.6% to 0.2% by step 300, and true accuracy finishes at 61.5%, within 0.1 points of the run that never had the hole. A team watching only the training reward would be tempted to revert the fix.

This is the verifiable-reward version of a familiar pattern. Reward hacking has a formal definition in terms of proxy and true reward [8], learned reward models show predictable overoptimisation as the policy moves away from its initial distribution [9] (see our note on overoptimisation), and more capable policies can switch abruptly to exploiting a misspecified reward [10]. A program-based verifier changes the shape of the proxy, not the logic.

Measuring verifier precision during training#

The quantity to track is the verifier's precision on the policy's own outputs, the probability that a rewarded response is actually correct:

${\text{true}}_t$ and ${\text{measured}}_t$ are the true and verifier-measured accuracy at step $t$, and $1-f_n$ the probability a correct answer is passed. Precision is a property of the verifier and the policy: the same checker that is 99% precise on the initial policy's outputs can be 70% precise on outputs the policy has learned to produce for it.

You cannot read precision off the training logs, because the verifier is the thing being tested. You estimate it by auditing: draw a random sample of verifier-passed outputs at a checkpoint, check each one with a stronger procedure (a stricter checker, a human, a held-out test suite), and compute the fraction that are truly correct. How many audits do you need? To reject "precision is at least 95%" at the 5% level with 80% power, an exact binomial test needs:

Large precision drops, the kind that matter, show up in a few dozen audits. A routine of 60 audited passes per evaluation checkpoint, stratified by task source so a single broken environment cannot hide in the average, catches a fall to 85% with 80% probability. Weight the audit towards the places hacks start: the hardest prompts, the environments whose reward is rising fastest, and the outputs that are unusually short or unusually long.

Two cheaper signals complement audits. Track the gap between the training verifier and a stricter held-out verifier on the same samples; a growing gap is precision falling. And keep a fixed probe suite of content-free responses (empty answers, every-number lists, multiple boxes, the master-key strings reported for LLM judges [5], test-skipping code) and score it with the current verifier at every checkpoint. Any nonzero pass rate is a hole, and you find it before the policy does.

Hardening the verifier#

Hardening is a precision-recall trade-off, as the experiment at the top showed. False positives are the more dangerous error, because the policy actively searches for them; false negatives only waste signal. When in doubt, prefer a stricter verifier and measure its false-negative rate on a labelled sample of correct answers.

Code: a strict checker and a probe suite#

import re

BOX = re.compile(r"\\boxed\{([^{}]*)\}")
INT = re.compile(r"-?\d+")

def strict_verify(response: str, reference: int, max_len: int = 4000) -> bool:
    boxes = BOX.findall(response)
    if len(boxes) != 1 or len(response) > max_len:
        return False
    x = boxes[0].strip()
    return bool(re.fullmatch(r"-?\d+", x)) and int(x) == reference

PROBES = {
    "empty": lambda q: "",
    "enumerate": lambda q: ", ".join(map(str, range(200))),
    "box_all": lambda q: " ".join(f"\\boxed{{{i}}}" for i in range(200)),
    "echo": lambda q: q + " " + (INT.findall(q) or ["0"])[-1],
    "opener": lambda q: "Let's solve this problem step by step.",
}

def probe_report(verify, problems):
    # fraction of problems on which each content-free probe is rewarded
    return {name: sum(verify(make(q), a) for q, a in problems) / len(problems)
            for name, make in PROBES.items()}

Run probe_report on the training verifier before the first RL step and at every checkpoint. The expected result is zero everywhere. The "echo" probe will also flag dataset items whose answer is printed in the question.

Checklist before and during an RLVR run#

Failure modes#

Evaluating with the training verifier. If the evaluation harness shares the parser with the reward function, every exploit is invisible in both. Keep at least one independent checker.

Fixing a hack by penalising its surface form. Penalising the string exit(0) removes that string, not the incentive. Optimising directly against a monitor of the reasoning has been shown to produce obfuscated hacking that the monitor no longer sees [6]. Fix the environment so the hack does not pay.

Treating the reward dip after a fix as a regression. In our simulation the patch cost 9 points of measured reward and gained 4 points of true accuracy by the end of the run.

Auditing uniformly. Hacks begin on the hardest prompts and in specific environments. A uniform audit sample dilutes them.

Assuming a stronger model makes the verifier safer. A stronger policy finds more holes. Precision should be re-measured whenever the policy, the data mix or the verifier changes.

What this simulation does not capture#

The policy is a three-parameter caricature: one skill, one exploit with a fixed pass rate, and an exploit propensity linear in difficulty. Real policies discover many exploits, each with its own pass rate, and discovery is the slow part; here the exploit is available from step one at low probability. Verifier errors are independent of the response content apart from the exploit. The verifier experiment uses scripted responses on synthetic arithmetic, not outputs from a trained model, and real exploit strategies are more varied. The mechanism (exploits win wherever their pass rate exceeds the reward of honest effort) and the measurement plan carry over.

Frequently asked questions#

Sources#

1. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al.. Tulu 3: Pushing Frontiers in Open Language Model Post-Training, 2024. arxiv.org
2. DeepSeek-AI (Daya Guo, Dejian Yang, et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. arxiv.org
3. Zhihong Shao, Peiyi Wang, Qihao Zhu, et al.. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. arxiv.org
4. Yuzhen Huang, Weihao Zeng, Xingshan Zeng, et al.. Pitfalls of Rule- and Model-based Verifiers: A Case Study on Mathematical Reasoning, 2025. arxiv.org
5. Yulai Zhao, Haolin Liu, Dian Yu, et al.. One Token to Fool LLM-as-a-Judge, 2025. arxiv.org
6. Bowen Baker, Joost Huizinga, Leo Gao, et al.. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, 2025. arxiv.org
7. Carson Denison, Monte MacDiarmid, Fazl Barez, et al.. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models, 2024. arxiv.org
8. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David Krueger. Defining and Characterizing Reward Hacking, 2022. arxiv.org
9. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
10. Alexander Pan, Kush Bhatia, Jacob Steinhardt. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, 2022. arxiv.org
11. Rulin Shao, Shuyue Stella Li, Rui Xin, et al.. Spurious Rewards: Rethinking Training Signals in RLVR, 2025. arxiv.org