Outcome reward model (ORM): Definition and Examples

An outcome reward model (ORM) is a learned model that reads a full response and predicts whether its final outcome is correct or good, producing a single score. It is trained only on end-result labels, such as whether a maths answer matches the reference, which makes its training data cheap compared with a process reward model.

How it is trained

Sample many solutions from a generator, label each one $y \in {0, 1}$ by checking the final answer automatically, and train a classifier $r_{ϕ} (x, z)$ on prompt $x$ and solution $z$ with the binary cross-entropy loss

L (ϕ) = - E [y lo g r_{ϕ} (x, z) + (1 - y) lo g (1 - r_{ϕ} (x, z))] .

Here $r_{ϕ}$ is the predicted probability that the solution is correct. Cobbe et al. (2021) introduced this "verifier" setup with the GSM8K dataset and showed that generating many candidates and picking the one the verifier ranked highest improved accuracy substantially, and scaled better with more data than simply fine-tuning the generator.

Worked example

Illustrative numbers: a generator solves a problem correctly 30% of the time. Draw 20 samples and let the ORM pick the top one. If the ORM ranks correct solutions above incorrect ones most of the time, the chance of picking a correct one is far above 30%; with a perfect ORM it equals the chance that at least one of 20 samples is correct, $1 - 0. 7^{20} \approx 0.999$ . Real ORMs fall between those numbers, and the gap is a direct measure of ranking quality.

The false-positive problem

Outcome labels cannot tell a sound derivation from a lucky one. A solution with a wrong step and a correct final answer gets label 1, so the ORM learns to reward some flawed reasoning. Uesato et al. (2022) found outcome supervision gave similar final-answer error to process supervision with less labelling, but more reasoning errors hidden inside correct-answer solutions. Interestingly, they also found that ORMs trained this way partly learned to emulate step-level judgments.

ORM vs a preference reward model

A preference reward model learns relative quality from human comparisons, typically via the Bradley-Terry model. An ORM learns absolute correctness from checkable labels, so it suits domains with a ground truth: maths, code, structured extraction.

Why it matters

ORMs are the cheapest learned verifier to build and power best-of-n reranking and outcome-based RL. Their blind spot, rewarding right answers reached the wrong way, matters most when users rely on the reasoning itself, and it is one more route to reward hacking when a policy is trained hard against them.

Sources

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman. Training Verifiers to Solve Math Word Problems, 2021. arxiv.org
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022. arxiv.org