A process reward model (PRM) is a learned model that assigns a score to every step of a multi-step solution, such as each line of a maths derivation, estimating whether that step is correct. Training on step-level labels is called process supervision, in contrast to the final-answer labels behind an outcome reward model.
How a PRM scores a solution
A solution is split into steps . The PRM outputs . Those step scores must then be combined into one solution score. Two common choices are
The product is the probability that every step is correct if steps are treated as independent; it is the aggregation Lightman et al. (2023) used. The minimum focuses on the weakest step and does not penalise long solutions just for being long.
Worked example
Illustrative numbers: a four-step solution gets step scores 0.95, 0.90, 0.40 and 0.97. The product is and the minimum is 0.40. Either way, the PRM points to step 3 as the likely error, which an outcome model cannot do: it only sees whether the final answer matched.
What the evidence shows
On grade-school maths, Uesato et al. (2022) found that outcome supervision reached similar final-answer error with fewer labels, but that process-based feedback was needed to get the reasoning right: their best system cut reasoning errors among correct-answer solutions from 14.0% to 3.4%. Lightman et al. (2023) collected 800,000 human step labels (released as PRM800K) and found process supervision clearly outperformed outcome supervision for reranking solutions on the MATH dataset, with their process-supervised model solving 78% of a representative test subset.
Costs and caveats
- Step labels are expensive; much later work estimates step correctness automatically, for example from how often rollouts continuing from a step reach the right answer.
- A PRM is still a learned proxy, so a policy optimised against it can learn steps that look correct to the PRM; see reward hacking.
- "Step" boundaries are a design choice and affect results.
Why it matters
PRMs give denser feedback than a single end-of-episode reward, which eases credit assignment and makes them useful both as verifiers for best-of-n reranking and as reward signals. For safety, rewarding the process rather than only the outcome is a way to make a model's reasoning, not just its answers, subject to oversight.
Related terms
Sources
- Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe. Let's Verify Step by Step, 2023. arxiv.org
- Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022. arxiv.org