Process vs outcome supervision: right answers, wrong reasoning

A reward model that only checks final answers will happily reward a solution that reaches 42 through two cancelling mistakes. Whether that matters depends on what you are building: a calculator does not care, but a tutor, a clinical summariser or an agent whose reasoning a human will audit cares a great deal. This note works through process supervision (scoring every step) and outcome supervision (scoring only the result) with a small seeded simulation. You will see why outcome-only selection can never push "right answer, wrong reasoning" below its base rate, what step labels actually buy you, why Monte Carlo step labels quietly inherit the outcome model's blind spot, and where search against a process reward model starts to hurt.

A worked example: six steps, one lucky answer#

Take a six-step solution to a word problem: parse the quantities, set up an equation, solve it, substitute back, convert units, report. Suppose a policy makes an error at each step with probabilities 4%, 6%, 12%, 8%, 5% and 3%. Step 3, the algebra, is the weak point. When at least one step is wrong, the final answer can still come out right: two sign errors cancel, the wrong intermediate happens to round to the right integer, or the answer space is small. Call that probability $c$ and set it to 0.20.

Three numbers describe this policy:

Here $K$ is the number of steps, $e_i$ is the error rate of step $i$, $P_{\text{sound}}$ is the probability that every step is correct, $P_{\text{correct}}$ is the probability that the final answer is correct, and $\rho$ is the share of correct answers whose reasoning contains at least one error. In words: a chain is only as sound as the product of its steps, and some of the unsound chains still land on the right answer.

Plugging in: $P_{\text{sound}}=0.96\times 0.94\times 0.88\times 0.92\times 0.95\times 0.97=0.673$, so $P_{\text{correct}}=0.673+0.327\times 0.2=0.739$ and $\rho =0.088$. Roughly one in eleven answers that an outcome grader marks as correct was reached through faulty reasoning. That is the quantity this note is about. We will call it the right-answer, wrong-reasoning rate, or RAWR.

Credit assignment: what an outcome reward says about step 3#

Training with outcome reward means each sampled solution gets one number, $R=1$ if the answer is right and 0 otherwise, and every token in the solution shares that number. How much does $R$ tell the learner about whether step $i$ was any good? The cleanest measure is the gap in expected reward between solutions where step $i$ was right and solutions where it was wrong:

Here ${\Delta }_i$ is the reward signal attributable to step $i$, and the product runs over all other steps. The derivation is short: if step $i$ is right, the reward is 1 when all other steps are right, and $c$ otherwise; if step $i$ is wrong, the reward is $c$ regardless. Subtracting gives the formula. The intuition: the outcome only reflects step $i$ when everything else went well, and luck dilutes even that.

For step 3 in our chain, ${\Delta }_3=0.612$. A process label on step 3 has a gap of 1 by construction (it is the step's own correctness), so a single outcome sample carries a bit more than half the step-specific signal of a step label, before even counting the noise in $R$. That ratio shrinks geometrically with chain length. Figure 1 plots $\Delta$ for a step in the middle of chains of uniform per-step error rate.

This is the core argument for process supervision in long-horizon training. Agent trajectories of 40 tool calls, or reasoning traces of thousands of tokens with dozens of logical moves, sit far out on the right of Figure 1. Outcome-based reinforcement learning still works on such problems, as large-scale results with verifiable rewards have shown (with failure modes of their own), but it pays for the weak per-step signal with many more samples and with indifference to how the answer was reached.

Selection: best-of-N with an outcome model versus a process model#

The same tension appears at inference time, where most teams first meet process reward models (PRMs). Sample $N$ solutions, score them, and keep the best. We simulated four selectors on 20,000 problems per setting, using the six-step chain above:

• ORM: an outcome reward model whose score is $s=2y+\epsilon$, where $y$ is answer correctness and $\epsilon$ is standard normal noise. It sees only whether the answer looks right.
• PRM: a process model that scores each step, $z_i\sim \mathcal{N}(2,1)$ for a correct step and $\mathcal{N}(0,1)$ for a wrong one, and aggregates by the minimum step score. Fifteen percent of wrong steps are "plausible" errors that the PRM cannot see; they score like correct steps.
• ORM + PRM: the sum of the two scores.
• PRM step search: instead of sampling whole solutions, generate $N$ candidates for each step, keep the one with the highest PRM score, and continue. This uses the same number of step generations as best-of-$N$.

Here $d=2$ is the PRM's separation between good and bad steps in standard deviations, and the minimum means that one bad-looking step sinks the whole solution. Lightman and colleagues aggregated with the product of per-step correctness probabilities, which behaves similarly; the minimum is easier to reason about.

Two results stand out.

Outcome selection cannot fix RAWR. The flat line in Figure 3 is not a simulation artefact. Any selector whose score depends only on answer correctness (plus noise independent of the reasoning) picks among correct answers without regard to how they were reached. Conditional on picking a correct answer, the chance that it came from a flawed chain is therefore the same as in the raw policy:

The left side is the RAWR of the selected outputs, the right side the RAWR of the policy. In plain terms: an outcome grader can make you right more often, but it cannot make your right answers more honest. Our "oracle" outcome verifier, which picks a correct answer whenever one exists, reached 100% accuracy at $N=8$ and still had RAWR of 8.8% at $N=64$.

The best selector combines both signals. ORM + PRM reached 100.0% accuracy and 96.9% fully sound solutions at $N=64$, against 90.9% sound for the ORM alone. The outcome model supplies what the PRM misses (hidden errors that do wreck the answer) and the PRM supplies what the outcome model cannot see (lucky answers). This mirrors an observation in the earlier comparison by Uesato and colleagues on GSM8K: outcome supervision matched process supervision on final-answer error, but low reasoning error among correct answers needed process-based feedback, or reward models that had learned to emulate it.

PRM-guided search, and why it amplifies hidden errors#

Step search is the most efficient use of a process model when the PRM is good. Picking the best of $m$ candidates at each step cuts that step's error rate roughly to the probability that a wrong candidate outscores every correct one, so errors are filtered where they occur instead of being discarded with whole solutions. With no hidden errors, step search at 64 candidates per step produced 99.9% sound solutions in our simulation, against 97.5% for PRM best-of-64.

The catch is what search does to the errors the PRM cannot see. Every round of selection prefers candidates that look correct to the PRM, and a hidden error looks correct by definition. Search therefore concentrates the PRM's blind spots in the output. The table shows how the advantage of search over best-of-$N$ erodes as the hidden-error share grows:

At a 50% hidden-error share, search and best-of-$N$ are indistinguishable and both leave roughly one solution in five unsound. In practice the hidden errors are not random: they are the plausible-looking mistakes a PRM learned to accept, which is precisely the set an optimiser finds. This is the process-level version of reward model over-optimisation, and it is why the ProcessBench results by Zheng and colleagues matter: PRMs trained on GSM8K and MATH-style data generalised poorly to harder competition problems, which is where their blind spots are largest. Snell and colleagues saw the same signature on MATH: beam search against a PRM beat best-of-$N$ at small budgets, but on easier questions at large budgets it showed signs of over-optimising the verifier and fell behind best-of-$N$.

Try it yourself below. Tap a step to change its error rate, switch selectors, and watch the per-step residual error after selection.

What step labels cost, and the Monte Carlo shortcut#

Outcome labels on maths and code are often free: a checker compares the answer with a reference or runs the tests. Step labels need a person, or a strong model, to read the reasoning. PRM800K, the dataset behind the Lightman et al. PRM, holds 800,000 step-level human labels, and the authors used active learning, surfacing convincing wrong-answer solutions for labelling, to make those labels go further. The DeepSeek-R1 report listed PRMs among its unsuccessful attempts for similar reasons: fine-grained steps are hard to define, intermediate correctness is hard to label at scale, and a model-based PRM invites reward hacking during training.

A common saving is first-error annotation: the labeller reads until the first wrong step and stops, since later steps are conditioned on a broken prefix. The expected number of steps read per solution is

which is the sum over steps of the probability that the labeller reaches that step. For our six-step chain this is 5.08 steps, only 15% fewer than labelling all six, because the policy is fairly good and most chains are read to the end. First-error labelling saves the most on weak policies, which is the opposite of when you most want a PRM.

The alternative is to label steps automatically with Monte Carlo rollouts, as in Math-Shepherd and OmegaPRM: from the prefix ending at step $i$, run $r$ completions and label the step good if any of them (a "hard" label) or a high fraction of them (a "soft" label) reach the right answer. OmegaPRM adds a binary search to find the first error in about ${\log }_{2}K$ probes. The method is cheap, but look at what it measures. We ran 8 completions from a prefix whose step 3 was wrong:

With hard labels, 84% of wrong steps were labelled good, because with $c=0.2$ the chance that at least one of eight rollouts lands on the right answer by luck is $1-0.8^8\approx 0.83$. Rollout labels estimate the probability of reaching a correct answer from a prefix. That is an outcome quantity, and it inherits the outcome model's indifference to lucky answers. Soft labels separate the cases much better (0.20 against 0.88), but a PRM trained on them is still learning answer-reachability, not reasoning validity. Zhang and colleagues reported this pattern in practice: Monte Carlo-labelled PRMs underperformed LLM-judge and human labels on step-error identification, and many PRMs placed their minimum score on the final answer step, a sign they had drifted into outcome scoring. Setlur and colleagues turned the idea around and argued that a good process reward should measure progress, the change in success probability under a separate prover policy, which is a cleaner target than raw Monte Carlo values but still answer-anchored.

Process supervision as a safety tool, and its limit#

For safety and correctness work the argument for process supervision is less about accuracy and more about what you are reinforcing. Outcome reward pays for any route to a passing grade, including routes you would reject on inspection: special-casing a unit test, citing a guideline that does not say what the answer claims, or a dosing calculation that is wrong in two compensating places. Process labels let you encode "this step is not acceptable even if the answer is right". In regulated settings, where a reviewer signs off on the reasoning and not only the conclusion, a RAWR of 9% is a defect rate, not a curiosity.

There is a real limit. Applying process pressure to a model's free-form chain of thought can teach the model to write chains that look good rather than chains that are good. Baker and colleagues showed this directly: adding a chain-of-thought monitor to the reinforcement learning reward produced more aligned agents under light optimisation, but under heavier optimisation agents learned obfuscated reward hacking, hiding intent in the reasoning while still hacking. The same caution applies to PRMs trained on the reasoning text: a policy optimised hard against one learns the PRM's hidden-error set, exactly as in the step-search table. Two design rules follow:

• Use process signals for selection and evaluation more freely than for training pressure on reasoning text. Selection does not change the policy, so it cannot teach obfuscation.
• Where you do train with process rewards, keep a held-out process grader that was never used as a reward, and track RAWR on it. If the training PRM's score rises while the held-out grader's RAWR does not fall, you are optimising the PRM's blind spots.

Choosing between them#

Checklist for a process-supervision project#

1. Measure the policy's RAWR first, by expert step-grading a few hundred correct-answer solutions. If it is under 1%, process supervision is mostly a cost.
2. Build a held-out step-labelled set that over-represents right-answer, wrong-reasoning solutions, and use it to evaluate every PRM (step-level error localisation, not just best-of-N accuracy).
3. Prefer human or strong-judge step labels for the evaluation set even if training labels come from rollouts.
4. Report accuracy and RAWR together for every selector and every $N$. Best-of-N accuracy alone will rank outcome models first.
5. Cap search depth and width until the held-out grader shows that search is reducing RAWR, not just raising PRM scores.

The simulation in a few lines#

The full script is longer, but the core of the best-of-N comparison is this:

import numpy as np
rng = np.random.default_rng(20260707)
E = np.array([0.04, 0.06, 0.12, 0.08, 0.05, 0.03]); C, D, H = 0.2, 2.0, 0.15

def best_of_n(n_problems, N):
    wrong = rng.random((n_problems, N, len(E))) < E           # step errors
    flawed = wrong.any(-1)
    y = np.where(flawed, rng.random(flawed.shape) < C, True)   # lucky answers
    orm = 2.0 * y + rng.standard_normal(y.shape)
    hidden = wrong & (rng.random(wrong.shape) < H)
    z = np.where(wrong & ~hidden, 0.0, D) + rng.standard_normal(wrong.shape)
    prm = z.min(-1)
    out = {}
    for name, s in {"orm": orm, "prm": prm, "orm+prm": orm + prm}.items():
        i = s.argmax(1); r = np.arange(n_problems)
        yy, ff = y[r, i], flawed[r, i]
        out[name] = dict(acc=yy.mean(), sound=(yy & ~ff).mean(),
                         rawr=(yy & ff).sum() / yy.sum())
    return out

Failure modes to watch for#

• Grading the answer twice. A PRM trained on rollout labels, then evaluated by best-of-N accuracy, can look excellent while adding nothing over an ORM. Evaluate step-level localisation directly.
• Step granularity drift. Steps defined by newlines in training and by sentences at inference change what the minimum aggregates over. Fix the segmentation and version it with the PRM.
• Labeller anchoring on the answer. Annotators who see the final answer first grade steps more leniently when it is right. Hide the answer during step labelling, or at least randomise its position.
• Over-deep search. Gains that appear at 8 candidates per step and vanish at 64 are a signature of hidden-error amplification, not noise.
• Treating a readable chain as a faithful one. Process supervision grades the text the model wrote. Whether that text reflects the computation that produced the answer is a separate question.

The simulation is deliberately small: independent step errors, a fixed luck parameter and Gaussian scores. Real errors are correlated (a misread problem corrupts several steps), real PRMs are miscalibrated across difficulty, and real luck depends on the answer format. None of those change the two structural results: outcome-only selection preserves RAWR, and search concentrates whatever the process model cannot see.

Frequently asked questions#

Sources#

1. Cobbe, Kosaraju, Bavarian, Chen et al.. Training Verifiers to Solve Math Word Problems, 2021. arxiv.org
2. Uesato, Kushman, Kumar, Song et al.. Solving math word problems with process- and outcome-based feedback, 2022. arxiv.org
3. Lightman, Kosaraju, Burda, Edwards et al.. Let's Verify Step by Step, 2023. arxiv.org
4. Wang, Li, Shao, Xu et al.. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations, 2023. arxiv.org
5. Luo, Liu, Liu, Phatale et al.. Improve Mathematical Reasoning in Language Models by Automated Process Supervision, 2024. arxiv.org
6. Snell, Lee, Xu, Kumar. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, 2024. arxiv.org
7. Setlur, Nagpal, Fisch, Geng et al.. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning, 2024. arxiv.org
8. Zheng, Zhang, Zhang, Lin et al.. ProcessBench: Identifying Process Errors in Mathematical Reasoning, 2024. arxiv.org
9. Zhang, Zheng, Wu, Zhang et al.. The Lessons of Developing Process Reward Models in Mathematical Reasoning, 2025. arxiv.org
10. DeepSeek-AI, Guo, Yang, Zhang et al.. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. arxiv.org
11. Baker, Huizinga, Gao, Dou et al.. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, 2025. arxiv.org