RLVR (reinforcement learning with verifiable rewards)

Reinforcement learning with verifiable rewards (RLVR) is reinforcement learning on tasks whose answers can be checked automatically, so the reward comes from a verifier program rather than a learned reward model. Typical examples are math problems with known final answers, code with unit tests, and instructions with checkable constraints.

The reward

In its simplest form the reward is binary:

r (x, y) = {10 if verify (x, y) = true otherwise

where $x$ is a problem, $y$ is the full sampled response and $verify$ extracts the final answer and checks it, for example by comparing a number to the reference or running tests in a sandbox. Small format rewards (did the answer appear in the required box?) are sometimes added. Training then uses a policy-gradient method such as PPO or GRPO, usually with a KL penalty or a clipped update.

Worked example

For one math prompt, the model samples 8 solutions and 3 reach the right answer, so the rewards are three 1s and five 0s with mean 0.375. A group-relative method pushes up the three correct solutions and pushes down the five wrong ones. A prompt where all 8 or none are correct gives every sample the same reward, so its advantage is zero and it contributes no learning signal. That is why RLVR datasets are often filtered to problems the current model solves sometimes but not always.

Why verifiable rewards help

No learned proxy to overoptimise. The checker does not drift and does not reward style.
Cheap labels. Any problem with a reference answer or test suite becomes training data.
Outcome-level signal. Credit for intermediate steps is left to the RL algorithm, which makes credit assignment hard on long solutions.

Failure modes

Verifiable does not mean unhackable. Weak checkers accept answers in the wrong form, tests can be special-cased or edited by an agent, and a model can learn to guess when partial credit is absent. These are forms of reward hacking, and they get more likely as the policy gets better at searching for loopholes.

Why it matters

RLVR, a term popularised by Lambert et al. (2024), is a main driver of recent gains in reasoning models. Because the reward is only as trustworthy as the verifier, verifier coverage and hacking checks deserve the same care as the training data, and evaluations should include held-out problems scored with stricter checkers than training used.

Sources

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al.. Tulu 3: Pushing Frontiers in Open Language Model Post-Training, 2024. arxiv.org
Zhihong Shao, Peiyi Wang, Qihao Zhu, et al.. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. arxiv.org