RLVR: Reinforcement Learning with Verifiable Rewards

Q: What is reinforcement learning with verifiable rewards?

RLVR is a way of fine-tuning a language model with reinforcement learning where the reward comes from a program that checks the output against ground truth, such as matching a reference answer or passing unit tests. No reward model or human label is needed per sample, so it scales cheaply on maths, code and other checkable tasks.

Q: What is the difference between RLVR and RLHF?

RLHF learns a reward model from human preference comparisons and optimises against it, which suits open-ended qualities like helpfulness. RLVR skips the learned model and computes reward with a deterministic verifier, which suits tasks with a checkable answer. RLVR fails mainly through verifier false positives; RLHF through reward-model over-optimisation.

Q: Why do prompts with 0% or 100% pass rate not help RLVR?

Group-relative methods subtract the group's mean reward. If every sampled answer passes, or every one fails, all advantages are zero and the prompt contributes no gradient. With 8 samples a prompt at 10% pass rate gives no signal 43% of the time, and one at 1% gives none 92% of the time.

Q: What is a verifier in RLVR?

A function that takes a prompt and a response and returns pass or fail without a human: exact or normalised answer matching, symbolic equivalence checking, unit tests run in a sandbox, format and constraint checkers, or a judge model comparing against a reference. Its quality is measured by precision and recall on responses of known correctness.

Q: Do verifier errors break RLVR training?

Random errors slow it. With false-negative rate fn and false-positive rate fp the expected reward is fp + (1 - fn - fp) times accuracy, so learning keeps its direction but shrinks. In our simulation 20% false negatives plus 10% false positives cut the final pass rate from 84.7% to 74.9%. Errors the policy can reproduce are worse: they get reinforced.

Q: Does RLVR teach models new reasoning abilities?

Evidence is mixed. One study found RLVR-trained models beat their base models at pass@1 while the base models scored higher pass@k at large k, suggesting RLVR mostly reweights reasoning paths already present. Since a never-solved prompt gives no gradient, RLVR works best on prompts the model can sometimes solve.

Q: What group size should I use for RLVR?

Large enough that the probability of an all-pass or all-fail group, p to the power G plus (1 - p) to the power G, is small at the typical pass rate of your prompts. For prompts near 50% even 8 samples suffice; for prompts near 5%, 8 samples waste two thirds of groups and 64 samples waste about 4%.

Reinforcement learning with verifiable rewards (RLVR) trains a language model with a reward computed by a program that checks the output, such as an exact-match test against a reference answer or a unit-test run, instead of a reward model learned from human preferences. The policy samples several answers per prompt, the verifier marks each one pass or fail, and a policy-gradient update makes the passing answers more likely. It is the standard recipe for post-training reasoning in maths and code because the reward is cheap, objective and hard to argue with.

This page explains RLVR from the ground up: what a verifier is, how pass/fail rewards become a learning signal, why the pass rate of each prompt decides how much the model can learn from it, what verifier errors do to training, and where the method stops working. The numbers come from exact calculations and seeded numpy simulations we ran for this article, and the interactive figure lets you see group-relative advantages change as you move the pass rate.

A worked example: one prompt, eight answers#

Take a maths prompt whose reference answer is 42. The model samples eight responses. A verifier extracts the final boxed answer from each and compares it with 42 after normalising whitespace and number formatting. Two of the eight match. The rewards are

r = (1, 0, 0, 1, 0, 0, 0, 0)

The group's mean reward is $0.25$ and its standard deviation is $0.25 \times 0.75 = 0.433$ . A group-relative advantage subtracts the mean and divides by the standard deviation, so each of the two correct answers gets advantage $(1 - 0.25) /0.433 = 1.73$ and each of the six wrong answers gets $(0 - 0.25) /0.433 = - 0.58$ . The update then raises the log-probability of every token in the two correct responses in proportion to 1.73 and lowers the six wrong ones in proportion to 0.58. The advantages sum to zero across the group, so the update moves probability mass from wrong answers to right ones on this prompt without needing a value network.

Now suppose all eight answers had been correct. The mean is 1, every advantage is 0 and the prompt contributes nothing. The same happens if all eight are wrong. That observation, which looks like a detail, turns out to govern most of what RLVR can and cannot learn.

What is a verifier?#

A verifier is any function $v (x, y) \in {0, 1}$ of a prompt $x$ and a response $y$ that can be computed without a human and without a learned preference model. The common families:

Verifier	Checks	Typical false positive	Typical false negative
Exact or normalised match	Extracted final answer equals the reference	Parser finds the reference among several candidates	Correct answer in another format ( $\frac{1}{2}$ vs 0.5)
Symbolic or numeric equivalence	Expressions are equal after simplification	Over-generous tolerance or simplifier bug	Simplifier times out on a correct but messy form
Unit tests	Generated code passes a test suite in a sandbox	Weak tests; code that special-cases the tests	Flaky tests, timeouts, environment differences
Constraint checkers	Format rules such as "exactly three bullet points" or "under 100 words"	Constraint met by a response that ignores the task	Rare; usually a tokenisation or counting mismatch
Model-based verifiers	A judge model decides whether the response matches the reference	Content-free responses the judge accepts	Correct but unusual reasoning the judge rejects

Tülu 3 applied RLVR to grade-school and competition maths with answer matching and to precise instruction following with constraint checkers [1]. DeepSeek-R1-Zero used rule-based accuracy rewards for maths and compiler and test feedback for code, plus a format reward for putting its reasoning between think tags, and its authors chose not to use neural reward models because of reward hacking at scale [3]. Training verifiers to judge maths solutions predates the RLVR name: GSM8K was released together with learned verifiers used to rerank sampled solutions [4]. The verifier in RLVR is a program, and its quality is measured like any classifier's, by precision and recall on responses whose true correctness you know.

How does RLVR training work?#

RLVR is ordinary policy-gradient reinforcement learning with a one-step episode: the prompt is the state, the whole response is the action, and the verifier's output is the terminal reward. If you are new to policy gradients, the policy gradient theorem and the REINFORCE algorithm give the background. The objective is

J (θ) = E_{x \sim D} E_{y \sim π_{θ} (\cdot ∣ x)} [v (x, y)] - β KL (π_{θ} ∥ π_{ref})

$D$ is the prompt set, $π_{θ}$ the policy being trained, $v$ the verifier, $π_{ref}$ a frozen reference policy and $β \geq 0$ the strength of a KL penalty that keeps the policy near the reference. In words: maximise the expected pass rate without drifting too far from where you started. Some recipes set $β = 0$ ; KL-regularised RL explains what the penalty buys.

Each training step does four things:

Sample a batch of prompts and draw $G$ responses per prompt from the current policy.
Score every response with the verifier.
Compute an advantage per response from the rewards in its group.
Take a clipped policy-gradient step that raises the log-probability of responses with positive advantage and lowers the rest.

The group-relative advantage used by GRPO, introduced in DeepSeekMath [2], is

A_{i} = \frac{r _{i} - r ˉ}{std ( r _{1} , \dots , r _{G} )}, r ˉ = \frac{1}{G} j = 1 \sum G r_{j}

$r_{i}$ is the verifier's reward for response $i$ and $r ˉ$ the group mean. Subtracting the mean is a baseline in the sense of variance reduction with baselines: it does not bias the gradient in expectation (up to the small effect of including $r_{i}$ in its own mean) and it removes the need for a learned critic. Dividing by the standard deviation rescales each prompt's update, which matters more than it looks, as the next section shows. The clipping step is the same trust-region idea covered in TRPO and PPO explained. This page does not go further into the optimiser; what matters for RLVR is what the binary rewards do to the signal.

Why the pass rate decides what RLVR can learn#

Model the policy on one prompt as producing a correct answer with probability $p$ , its pass rate on that prompt. Let $z$ be the logit of "correct", so the score function is $\partial lo g π / \partial z = 1 - p$ for a correct response and $- p$ for a wrong one. In a group of $G$ samples, $k \sim Binomial (G, p)$ are correct. With mean-centred advantages the gradient on $z$ summed over the group is

k (1 - \frac{k}{G}) (1 - p) + (G - k) (- \frac{k}{G}) (- p) = \frac{k ( G - k )}{G}

The terms in $p$ cancel exactly, so the push towards "correct" from one group depends only on how the group split. Taking the expectation over $k$ and dividing by $G$ gives the expected push per sample:

mean-centred: \frac{( G - 1 ) p ( 1 - p )}{G}, std-normalised: \frac{1}{G} E [k (G - k)], P (no signal) = p^{G} + (1 - p)^{G}

The first expression is the familiar Bernoulli variance: prompts the model solves half the time teach it the most, and prompts it always or never solves teach it nothing. The second applies the standard-deviation division, which multiplies a group's update by $G / k (G - k)$ and therefore boosts groups with lopsided splits. The third is the chance that a group is all-pass or all-fail and contributes exactly zero gradient.

Concrete values from the exact binomial calculation, for $G = 8$ :

Pass rate $p$	P(no signal)	Push per sample, mean-centred	Push per sample, std-normalised
0.01	92.3%	0.009	0.026
0.05	66.3%	0.042	0.118
0.10	43.0%	0.079	0.209
0.25	10.0%	0.164	0.380
0.50	0.8%	0.219	0.464

Two practical consequences follow. First, a prompt with a 1% pass rate wastes 92% of its groups at $G = 8$ ; raising $G$ to 64 cuts that to 53%. Hard prompts need large groups or they are dead weight. Second, the standard-deviation division changes the effective weighting of prompts by difficulty: relative to mean-centring it multiplies the push on a $p = 0.05$ prompt by 2.8 and on a $p = 0.5$ prompt by 2.1, so very easy and very hard prompts gain relative weight. A critical analysis of R1-Zero-style training identified this as a question-level difficulty bias in GRPO, alongside a response-length bias from dividing each response's loss by its length, and proposed removing both normalisations [6].

Pass rate

p

0.10 Group size

G

Figure 1. Expected learning signal per sample as a function of the prompt's pass rate, from the exact binomial formulas above. Blue: GRPO-style advantages (mean-centred and divided by the group standard deviation). Green dashed: mean-centred only. Red dotted: probability that the group is all-pass or all-fail and gives zero gradient. The bars below the chart are one sampled group: green bars are correct responses, red bars wrong ones, bar height is the advantage. Move

p

towards 0 and most groups show flat bars. The static version shows

G = 8

with the dashed line at

p = 0.1

Filtering zero-signal groups#

Because all-pass and all-fail groups contribute nothing, one fix is to stop spending update steps on them. DAPO over-samples prompts and drops groups whose accuracy is exactly 0 or 1 until the batch is full of groups with a mixed split, noting that the share of all-correct groups keeps rising as training progresses [5]. In our simulation below, dynamic sampling of this kind raised the final mean pass rate after 400 update steps from 84.7% to 97.8%, but it consumed 2.9 times as many generated samples (297,058 against 102,400 on average). Run to the same budget of 102,400 samples it reached 85.0%, essentially the same as the unfiltered run at 84.7%. In this toy, filtering buys more progress per gradient step, not per sample. Whether that trade is worth it depends on whether generation or optimisation is your bottleneck.

A simulated RLVR run, and what verifier noise does#

To see these effects over a whole run we simulated a pool of 1,024 prompts with difficulties $b_{j}$ drawn from a normal distribution (mean 1, SD 2). The policy solves prompt $j$ with probability $σ (s + u_{j} - b_{j})$ , where $s$ is a shared skill that transfers across prompts and $u_{j}$ is prompt-specific. Each step samples 32 prompts with 8 responses each, scores them with a verifier, computes GRPO-style advantages and applies the exact score-function gradient to $s$ and $u_{j}$ . We ran 400 steps, five seeds per condition, and report the true mean pass rate over the whole pool. This is a simulation of the training dynamics, not a language model.

The verifier can be wrong in two ways that are independent of the response content: it fails a correct answer with probability $f_{n}$ (false negative) and passes a wrong answer with probability $f_{p}$ (false positive). The expected reward of a response with true correctness probability $a$ is then

E [r] = f_{p} + (1 - f_{n} - f_{p}) a

The reward is still increasing in true accuracy as long as $f_{n} + f_{p} < 1$ ; noise of this kind shrinks the signal by the factor $1 - f_{n} - f_{p}$ and adds variance, but it cannot reverse the direction of learning.

Figure 2. True mean pass rate over 400 simulated RLVR steps, averaged over five seeds. Random verifier errors slow learning in proportion to how much they shrink the reward's dependence on correctness. Starting pass rate 35.8% in every condition.

Verifier	Pass rate at step 100	Pass rate at step 400	Signal factor $1 - f_{n} - f_{p}$
Exact	52.9%	84.7%	1.0
20% false negatives	49.5%	79.0%	0.8
10% random false positives	50.9%	81.9%	0.9
Both	47.4%	74.9%	0.7

The seed-to-seed standard deviation of the final pass rate was about one point in every condition, so the ordering is clear. Also worth noting: at the start, 31.1% of prompts had a pass rate below $1/ G$ , meaning that most of their groups gave no signal. Because the shared skill transfers, that fraction fell to 3.8% by step 400 with the exact verifier. Without transfer, those prompts would have stayed unsolved, which is the mechanism behind the next section's caveat.

Verifier false positives that the policy can find#

The simulation above assumed errors that strike at random. Real verifiers make errors that depend on the response, and a policy under optimisation pressure searches for responses that trigger them. That changes the picture completely: an exploit that passes the verifier with probability $f_{x}$ is reinforced on any prompt where $f_{x}$ exceeds the expected reward of an honest attempt, so hacking starts on the hardest prompts and spreads. Documented cases include lenient answer parsers that accept several candidates, model-based verifiers that are accurate on static benchmarks but hacked during RL [7], content-free "master key" responses such as a lone colon or "Let's solve this problem step by step." that many judge models accept as correct [8], and coding agents that exit before the tests run [12]. Our research note reward hacking with verifiable rewards measures four answer checkers against scripted exploits and simulates the training dynamics; read it before you trust a verifier in production. The general phenomenon is covered in reward hacking.

What RLVR does and does not teach#

The pass-rate analysis has an uncomfortable corollary. A prompt the policy never solves gives no gradient, so RLVR mostly reweights behaviour the model can already produce. A systematic study comparing RLVR-trained models with their base models found that RLVR wins at pass@1 but the base models achieve higher pass@k when $k$ is large, which suggests the reasoning paths RLVR rewards were already in the base model's distribution [9]. Another study found that RLVR with random or even incorrect rewards raised MATH-500 accuracy substantially for one model family but not for others, and attributed this to training surfacing behaviours learned in pretraining [10]. Neither result says RLVR is useless; both say that a rising training curve is weak evidence about what the model learned. Measure with held-out evaluations and report pass@k as well as pass@1; pass@k versus pass^k explains which to use when.

Binary outcome rewards also say nothing about which step of a long solution was wrong. Process supervision gives per-step feedback and was shown to train more reliable reward models for maths than outcome supervision [11]; our note on process versus outcome supervision works through when each is worth it.

RLVR vs RLHF#

	RLVR	RLHF
Reward source	Program checking against ground truth	Reward model trained on human preference pairs
Cost per reward	Compute only	One reward-model forward pass; human labels up front
Applies to	Tasks with checkable answers: maths, code, formal constraints	Open-ended quality: helpfulness, tone, safety
Main failure	Verifier false positives, sparse signal on hard prompts	Reward-model over-optimisation, preference noise
Typical reward shape	Binary, sometimes with partial credit	Continuous score

Most post-training pipelines use both: RLVR where an answer can be checked and a learned reward elsewhere. RLHF explained and reward models cover the other half, and reinforcement learning for LLMs puts them together. Agentic RLVR, where the verifier inspects the end state of a multi-step episode, is covered in RL environments for LLM agents.

Code: verifier, group advantages and the signal check#

A minimal, dependency-light version of the pieces discussed above. The verifier reads exactly one boxed answer; the advantage function matches the formula in the text; the last function reproduces the "no signal" column of the table.

import re
import numpy as np
from math import comb

BOX = re.compile(r"\\boxed\{([^{}]*)\}")

def verify(response: str, reference: str) -> float:
    boxes = BOX.findall(response)
    if len(boxes) != 1:          # zero or several candidates: fail
        return 0.0
    norm = lambda s: s.replace(" ", "").replace(",", "").rstrip(".")
    return float(norm(boxes[0]) == norm(reference))

def group_advantages(rewards, eps=1e-6, std_norm=True):
    r = np.asarray(rewards, dtype=float)
    a = r - r.mean()
    return a / (r.std() + eps) if std_norm else a

def no_signal_prob(p, G):
    return p**G + (1 - p)**G

def expected_push(p, G, std_norm=True):
    ks = np.arange(G + 1)
    w = np.array([comb(G, k) for k in ks]) * p**ks * (1 - p)**(G - ks)
    per_group = np.sqrt(ks * (G - ks)) if std_norm else ks * (G - ks) / G
    return float((w * per_group).sum() / G)

print(group_advantages([1, 0, 0, 1, 0, 0, 0, 0]).round(2))   # [ 1.73 -0.58 ...]
print(round(no_signal_prob(0.10, 8), 3))                     # 0.43
print(round(expected_push(0.10, 8, std_norm=False), 4))      # 0.0788

Checklist for an RLVR run#

Stage	Check
Data	Estimate each prompt's pass rate with the starting policy (8 to 16 samples). Drop or set aside prompts at 0% and 100%; they cost compute and give no gradient.
Data	Remove items whose answer appears in the prompt and items with ambiguous references.
Verifier	Measure precision and recall on a labelled sample, including deliberately malformed and content-free responses.
Verifier	Read one designated answer slot; reject multiple candidates; run code tests outside the policy's reach.
Group size	Pick $G$ so that $p^{G} + (1 - p)^{G}$ is small at the median pass rate of your data.
During training	Log the share of zero-signal groups, the pass-rate histogram and response length. A sudden jump in reward is a reason to read samples.
Evaluation	Score checkpoints with an independent checker and report pass@1 and pass@k on held-out prompts. The LLM evaluation guide covers the rest of the harness.

Failure modes#

A dataset that is too hard. If most prompts sit near 0% pass rate, most groups are all-fail and the run learns slowly or not at all. Curriculum or filtering by measured pass rate fixes this faster than tuning the optimiser.

A dataset that becomes too easy. As the policy improves, all-pass groups grow. Refresh the prompt pool or filter, otherwise effective batch size shrinks silently.

Format rewards that dominate. A reward for the answer format that is not tied to correctness is an exploit waiting to be found.

Evaluating with the training verifier. Every exploit of the training verifier is invisible to an evaluation that shares its parser.

Length drift. Normalising the loss by response length changes how much long wrong answers are penalised, and length can grow without accuracy gains [6]. Track length alongside reward.

Frequently asked questions#

What is reinforcement learning with verifiable rewards?