Verifier

A verifier is anything that takes a candidate output and returns a judgment of whether it is correct or acceptable: a unit-test harness, an exact-match answer checker, a proof checker, or a learned model that predicts correctness. Verifiers turn "generate then check" into a training and inference strategy, because checking an answer is often much easier than producing one.

Kinds of verifier

• Programmatic: rules or execution with a known ground truth, such as comparing a final number to the reference or running code against tests. These are the backbone of RL with verifiable rewards.
• Learned: trained models such as an outcome reward model or a process reward model, introduced for maths by Cobbe et al. (2021).
• Model-as-judge: a language model prompted with a rubric; see LLM-as-a-judge.

Measuring verifier quality

A verifier is a binary classifier and has two error types. Its false-positive rate (accepting wrong outputs) matters most, because optimisation actively searches for accepted-but-wrong outputs. If the policy is correct with probability $q$, the verifier's true-positive rate is $\mathrm{TPR}$ and its false-positive rate is $\mathrm{FPR}$, the probability that an accepted output is actually correct is

Worked example

Illustrative numbers: a policy is right 20% of the time and the verifier has TPR 0.95 and FPR 0.05. Then $P(\text{correct}\mid \text{accepted})=0.19/(0.19+0.04)\approx 0.83$. A 5% false-positive rate already means about one accepted answer in six is wrong. If training then pushes the policy towards outputs the verifier mistakenly accepts, the FPR on the policy's own outputs rises well above 5%, which is how reward hacking shows up in practice.

Design checklist

• Measure FPR on adversarial or policy-generated outputs, not only on held-out human answers.
• Normalise answer formats so correct answers are not rejected for trivial reasons (false negatives slow learning).
• Keep a held-out verifier, or a human audit sample, that is never used for training.
• Sandbox code execution so the policy cannot alter the checker.

Why it matters

Verifiers decide what "correct" means for both best-of-n selection and RL training, so their blind spots become the policy's habits. They are also the main reason maths and code improved so quickly under RL: these are the domains where a cheap, reliable verifier exists.

Related terms

• RLVR (reinforcement learning with verifiable rewards)
• Outcome reward model (ORM)
• Process reward model (PRM)
• Best-of-n sampling
• Reward hacking
• LLM-as-a-judge

Sources

1. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman. Training Verifiers to Solve Math Word Problems, 2021. arxiv.org