Reward model

A reward model is a neural network that takes a prompt and a response and returns a single number estimating how much a human would prefer that response. It is trained on preference comparisons and then stands in for the human during RLHF, best-of-n sampling and data filtering.

How it is trained

Most reward models are a language model with its output layer replaced by a scalar head. Given a prompt $x$, a preferred response $y_w$ and a rejected response $y_l$, training minimises the Bradley-Terry negative log-likelihood:

where $r_{\varphi }$ is the reward model with parameters $\varphi$ and $\sigma$ is the logistic function. Only differences in reward matter, so the scale and offset of $r_{\varphi }$ are arbitrary; teams often normalise scores so the reference policy's responses average zero.

Worked example

The model scores the chosen response 1.5 and the rejected one 0.5. The predicted preference probability is $\sigma (1.0)\approx 0.73$ and the loss for that pair is $-\log 0.73\approx 0.31$. Had the scores been swapped, the probability would be 0.27 and the loss 1.31, so the gradient pushes hard on pairs the model gets wrong.

Variants

• Outcome reward models score a whole final answer.
• Process reward models score each intermediate reasoning step.
• Generative or judge-style reward models write a critique before giving a score, much like LLM-as-a-judge.
• Rule-based verifiers replace the learned model entirely when correctness can be checked, as in RLVR.

How to evaluate one

The basic metric is pairwise accuracy on held-out comparisons; on noisy chat preferences it sits well below 100 percent, and it is bounded above by how often annotators agree with each other. Accuracy alone is not enough: check calibration, accuracy per category (safety, coding, factuality), and sensitivity to length and style, because a reward model that likes long answers will teach the policy to ramble.

Why it matters

Whatever the reward model gets wrong, the policy will learn to exploit. Gao et al. (2022) showed that as a policy is optimised further from its start, true quality first rises and then falls while the proxy reward keeps climbing. That is reward overoptimisation, and it makes the reward model one of the most safety-relevant artefacts in the training pipeline.

Related terms

• RLHF (reinforcement learning from human feedback)
• Bradley-Terry model
• Preference data
• Reward overoptimization
• Process reward model (PRM)
• Outcome reward model (ORM)

Sources

1. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. Learning to summarize from human feedback, 2020. arxiv.org
2. Long Ouyang, Jeff Wu, Xu Jiang, et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org
3. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org