Reward models: how they are trained, how to evaluate them, and why accuracy is not enough

A reward model is a model that scores how good an output is, so that another model can be trained or selected to produce better outputs. In language model training it is usually a copy of the language model with a single scalar output, trained on human (or AI) preference comparisons so that the higher-scored of two responses is the one people would pick.

Reward models are the part of RLHF that turns human judgment into a number an optimiser can push on, and they are also used directly for best-of-n sampling, data filtering and evaluation. This guide covers how they are built and trained, how to evaluate them, and the uncomfortable result that a reward model's accuracy on held-out preferences is only loosely related to how good a policy it produces. Two interactive simulations make the points concrete.

What does a reward model do? A worked example#

Take the prompt "Explain why the sky is blue to a ten-year-old" and three candidate answers. Answer A is accurate and pitched right. Answer B is accurate but full of jargon. Answer C is charming and wrong. A trained reward model might score them 1.9, 0.4 and 0.7. Those numbers have no absolute meaning; what they say is that A beats B with probability $\sigma (1.9-0.4)=0.82$, A beats C with probability 0.77, and C narrowly beats B (0.57).

Three uses follow. During RL fine-tuning the policy is rewarded with these scores, so it learns to write more like A. At inference time, best-of-n sampling draws several answers and returns the top-scored one. And for data work, the scores rank or filter candidate training examples. In all three, any systematic mistake (say, a soft spot for charming wrong answers like C) gets amplified, because every use selects for high scores.

Reward model architecture#

The standard design takes a pretrained or supervised fine-tuned language model, removes the layer that predicts the next token, and adds a linear head that reads the final hidden state at the last token and outputs one number. InstructGPT started its reward model from the supervised model with the unembedding layer removed and used a 6B-parameter reward model for a 175B policy, reporting that 175B reward model training could be unstable [1]. Starting from a language model matters: the reward model needs to understand the text as well as the policy does, or the policy will find responses the reward model cannot tell apart.

Variations are common. Separate reward models for separate goals are one: Llama 2 trained a helpfulness reward model and a safety reward model and combined them during RL [6]. Generative reward models, discussed below, have the model write a judgment instead of emitting a scalar. And Direct Preference Optimization has no separate reward model at all; its policy defines an implicit reward through log-probability ratios against the reference model [20].

How is a reward model trained?#

The Bradley-Terry loss#

Given comparisons in which a rater preferred response $y_w$ over $y_l$ for prompt $x$, the reward model is fitted by maximum likelihood under the Bradley-Terry model of paired comparisons [3]:

$\mathcal{D}$ is the comparison dataset and $\varphi$ the reward model's parameters. This is logistic regression on score differences: the loss is small when the preferred response scores well above the other one. Stiennon et al. and InstructGPT both used this loss [2][1]. Because only differences enter, adding any constant to every score leaves the loss unchanged, which is why reward models need an explicit normalisation step (below).

Rankings instead of pairs#

Reading a prompt is a large part of a rater's effort, so it pays to show several responses at once. InstructGPT asked raters to rank between 4 and 9 responses, producing up to 36 comparisons per task, and trained on all comparisons from one prompt as a single batch element: shuffling them as independent examples made the reward model overfit within one pass [1]. Zhu, Jiao and Jordan showed that the maximum likelihood estimator converges both under the full Plackett-Luce ranking model and when a ranking is split into its pairwise comparisons, with the full likelihood asymptotically more efficient [4].

How much is a ranking worth? We simulated it: 16-dimensional linear utilities, rankings drawn from a Plackett-Luce model, a Bradley-Terry fit on all implied pairs, 30 seeds per row (numpy, seeded).

In this toy, 100 four-way rankings were worth as much as 400 single comparisons. Per response read (400 versus 800), the ranking format was about twice as efficient. The implied pairs are correlated, so 600 pairs from rankings are worth far less than 600 independent pairs, but they are cheap.

Margins, normalisation and other details#

Margins. When raters say how much better one response is, the strength can enter the loss. Llama 2 subtracted a margin $m(r)$, a discrete function of the preference rating, inside the sigmoid, so that "significantly better" pairs had to be separated by more; the margin improved accuracy on clearly separable pairs, while a larger margin hurt accuracy on similar ones [6].

Normalisation. Since the loss ignores shifts, the zero point is fixed after training. Ziegler et al. normalised the reward to mean 0 and variance 1 over samples from the reference policy [5]; InstructGPT added a bias so that labeller demonstrations had mean score 0 [1]. The scale matters too: the KL coefficient in RLHF trades reward against KL, so a reward model with a compressed scale effectively gets a larger penalty.

One epoch. InstructGPT reported that its reward model tended to overfit after a single epoch [1]. Reward models have few labels relative to their size, and overconfidence on the training distribution is exactly what an optimiser exploits.

Watch a Bradley-Terry reward model learn#

The figure below fits a linear Bradley-Terry reward model to simulated preferences. Responses have 8 standard-normal features; raters' utility is a fixed linear function of them; each label is drawn from the Bradley-Terry probability and then flipped with probability $\epsilon$. We evaluate on 4,000 fresh pairs.

At the defaults the reward model agrees with fresh noisy labels 73.8 percent of the time, against a ceiling of 74.0 percent for a perfect model, and orders pairs correctly by true utility 96.6 percent of the time. The learned scores are 0.74 times the true log-odds: flip noise compresses the scale. The predicted probabilities are well calibrated in the middle (expected calibration error 0.019) and overconfident at the extremes, because no amount of evidence makes a flipped label less likely than $\epsilon$. With 100 clean pairs the scale is 1.40, overconfident in the other direction; small datasets let a reward model claim more certainty than it has.

The practical reading: accuracy on a noisy validation set is bounded by rater agreement, so "72 percent" might be a near-perfect reward model on hard, contested data or a weak one on easy data. Our research note on preference label noise works through ceilings, flip rates and how many annotators to pay for per pair.

How do you evaluate a reward model?#

Pairwise accuracy and benchmarks#

The default metric is pairwise accuracy: the fraction of held-out comparisons where the chosen response scores higher. RewardBench made this a shared benchmark, with prompt, chosen and rejected trios across chat, hard chat, safety and reasoning, built so that one answer is better for a subtle but verifiable reason such as a bug or a wrong fact [7]. RewardBench 2 moved to a best-of-4 format (one correct response against three incorrect ones, so chance is 25 percent), sourced new human prompts, and reported that models scored about 20 points lower than on the first benchmark while correlating well with downstream use in best-of-n sampling and PPO [8].

Accuracy needs slicing to be useful. Report it by category, on safety pairs where the right answer is a refusal and where it is not, and on pairs where the chosen response is shorter. That last split catches the most common bias; Singhal et al. traced most of RLHF's length growth to reward models that had picked up length correlations in the preference data [21].

Calibration#

Because a Bradley-Terry reward model predicts probabilities, it can be checked like any probabilistic classifier: bin the predicted $P(y_1\succ y_2)$ and compare with observed preference rates, as in Figure 1. Calibration matters when scores are used as confidence, for example to decide which pairs to send for more labels, or when scores from several reward models are combined.

Why accuracy does not predict downstream performance#

The question that matters is how good a policy the reward model produces, and accuracy answers it only loosely. Wen et al. found a weak positive correlation between reward model accuracy and downstream policy performance in a synthetic setting, observed that policies optimised against reward models with similar accuracy could perform quite differently, and showed that the way accuracy is measured changes how well it predicts [9]. Kim et al. likewise reported weak correlations between existing reward model benchmarks and optimised policy performance, and recommended benchmarks with many comparisons per prompt, chosen and rejected responses that differ mainly in correctness, and responses from a variety of models [10].

The reason is that accuracy averages over typical pairs, while optimisation selects from the tail. Here is a minimal demonstration. Two reward models score a pool of 20,000 responses with true quality $g\sim \mathcal{N}(0,1)$. Reward model A has diffuse noise: $r_A=g+0.75e$. Reward model B is less noisy on ordinary responses but has a blind spot: a small fraction of responses carry an "exploit" (think confident formatting that hides a wrong answer) that lowers true quality by 1 and raises B's score by 3. B's noise level is set by bisection so the two have the same pairwise accuracy on random pairs. Best-of-n gold is computed exactly from order statistics: the response at rank $i$ of $M$ is selected with probability $(i/M{)}^n-((i-1)/M{)}^n$.

With a 2 percent exploit rate both models score 79.7 percent pairwise accuracy. At best-of-4 they are nearly indistinguishable (gold gain 0.84 versus 0.80). At best-of-64, A's pick is 1.87 standard deviations above average and B's is 1.42; B's curve is flat from there on (1.44 at $n=128$, 1.41 at $n=1,024$) while A's keeps rising to 2.67. By $n=256$, half of B's picks are exploit responses. Even accuracy restricted to pairs from the top 10 percent of true quality did not separate them in this setup (64.0 versus 64.2 percent). What separates them is the best-of-n curve itself, measured with an independent gold signal, and reading the top-scored outputs.

That is why evaluation should include the optimisation the reward model will face. A best-of-n curve scored by a held-out judge or human labels, at the $n$ or KL budget you plan to use, is closer to the downstream question than any accuracy number. Our research note on reward model overoptimisation shows how to estimate that curve from one batch of samples and how it bends as optimisation pressure grows.

Here is a pairwise accuracy report that at least catches the length bias, with confidence intervals that respect the clustering of comparisons by prompt:

import numpy as np

def rm_report(s_chosen, s_rejected, len_chosen, len_rejected, prompt_id, n_boot=2000, seed=0):
    """Pairwise accuracy of a reward model with a prompt-clustered bootstrap CI,
    split by whether the preferred response is the longer one."""
    correct = (s_chosen > s_rejected).astype(float)
    longer = len_chosen > len_rejected
    prompts = np.unique(prompt_id)
    idx = {p: np.flatnonzero(prompt_id == p) for p in prompts}
    rng = np.random.default_rng(seed)
    def stats(rows):
        c, l = correct[rows], longer[rows]
        return c.mean(), c[l].mean(), c[~l].mean()
    point = stats(np.arange(len(correct)))
    boots = np.array([stats(np.concatenate([idx[p] for p in rng.choice(prompts, len(prompts))]))
                      for _ in range(n_boot)])
    lo, hi = np.nanpercentile(boots, [2.5, 97.5], axis=0)
    names = ["overall", "chosen is longer", "chosen is shorter"]
    return {k: (round(float(m), 3), round(float(a), 3), round(float(b), 3)) for k, m, a, b in zip(names, point, lo, hi)}

# demo: a reward model that partly scores length
rng = np.random.default_rng(1)
n, prompt_id = 3000, np.repeat(np.arange(600), 5)
q_c, q_r = rng.normal(0.5, 1, n), rng.normal(0, 1, n)          # chosen is better on average
l_c, l_r = rng.normal(0, 1, n), rng.normal(0, 1, n)
s_c = q_c + 0.8 * l_c + rng.normal(0, 0.5, n)
s_r = q_r + 0.8 * l_r + rng.normal(0, 0.5, n)
print(rm_report(s_c, s_r, l_c, l_r, prompt_id))

On the demo data the overall accuracy is 60.9 percent (95% interval 59.1 to 62.6), but 78.7 percent when the chosen answer is longer and 42.7 percent, worse than chance, when it is shorter. The clustered interval matters whenever several comparisons share a prompt; our note on clustered standard errors explains why.

Reward model overoptimisation#

Optimising against a reward model raises its score indefinitely, while true quality rises, peaks and falls. Gao et al. measured this with a large "gold" reward model labelling data for smaller proxy reward models, and found smooth functional forms for gold reward against KL distance whose coefficients scale with reward model size [11]. The RLHF guide shows a toy version in which the peak comes from rater biases the reward model faithfully learned, and the KL-regularised RL guide covers the penalty that decides where on the curve a run stops. For the full treatment, including best-of-n versus RL and heavy-tailed errors, see our overoptimisation research note; for how it looks in deployed systems, see reward hacking.

Ensembles#

Training several reward models and optimising a conservative combination (the minimum, or the mean minus a multiple of the disagreement) reduces overoptimisation. Coste et al. found that for best-of-n this practically eliminated overoptimisation in their setup and that for PPO it consistently reduced it [12]. Eisenstein et al. found that ensembles mitigate but do not eliminate reward hacking: members pretrained with different seeds helped more than members differing only in fine-tuning seed, and errors shared by every member survive the ensemble [13]. A length bias learned from the same preference data is exactly such a shared error.

Generative reward models and LLM judges#

Instead of a scalar head, a generative reward model writes its judgment as text. The simplest version is an LLM-as-a-judge prompted to compare two answers; Zheng et al. found strong judges agreeing with human preferences at rates comparable to agreement between humans, alongside position, verbosity and self-preference biases [14]. Trained generative reward models go further. Mahan et al. reported that zero-shot LLM judgments underperformed Bradley-Terry reward models in-distribution, while their trained generative reward model matched Bradley-Terry accuracy in-distribution and beat it out of distribution [15]. Zhang et al. trained verifiers with next-token prediction, so they could reason before judging and use majority voting over several judgments, and reported large best-of-n gains over discriminative verifiers on math and algorithmic tasks [16].

Generative judges inherit the biases of the model doing the judging and cost far more per score. The LLM-as-a-judge guide covers prompting and validation, and our note on judge position and verbosity bias measures the biases.

Verifiers, outcome and process reward models#

When correctness can be checked, the reward model becomes a verifier. Cobbe et al. trained verifiers to judge whether grade-school math solutions were correct and found that best-of-n selection with a verifier gave about the same boost as a 30-fold increase in model size [17]. An outcome reward model scores only the final answer; a process reward model scores each reasoning step. Uesato et al. found that outcome and process supervision reached similar final-answer error on math word problems, but process-based feedback was needed for correct reasoning steps [18]; Lightman et al. found that process supervision trained a more reliable reward model, with which their model solved 78 percent of a representative subset of MATH problems [19]. Our research note on process versus outcome supervision compares the two in detail. Where a program can check the answer outright, no learned model is needed; that is the domain of RL with verifiable rewards, which has its own hacking problems.

Choosing and checking a reward model#

Failure modes#

Spurious features. Length, formatting, hedging and agreement with the user are easy to learn and correlate with preference in typical data. They become the directions a policy exploits.

Distribution shift. A reward model is trained on samples from an earlier policy. As the policy improves, its outputs move away from that distribution and the reward model's accuracy on them is unknown. Iterated collection, with fresh comparisons on current samples, is the standard remedy.

Scale drift. Changing data, noise level or architecture changes the reward scale and so the effective KL penalty. Re-tune the coefficient whenever the reward model changes.

Reward model as evaluator. Reporting a policy's improvement on the reward model it was trained against measures optimisation, not quality. Hold out a different signal: human labels, a different judge, or task success. The broader discipline is in the guide to AI evals.

What the simulations leave out#

All three simulations use low-dimensional linear or Gaussian models. Real reward models are fine-tuned language models whose errors depend on the text in ways no toy captures, and the exploit in Figure 3 is a stylised stand-in for real failure modes such as confident formatting or flattering tone. The simulations show why accuracy and downstream usefulness can come apart; they do not estimate how often that happens with real reward models, which is what the benchmark studies cited above try to measure.

Frequently asked questions#

Sources#

1. Long Ouyang, Jeff Wu, Xu Jiang, et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org
2. Nisan Stiennon, Long Ouyang, Jeff Wu, et al.. Learning to summarize from human feedback, 2020. arxiv.org
3. Ralph Allan Bradley, Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons, 1952. doi.org
4. Banghua Zhu, Jiantao Jiao, Michael I. Jordan. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons, 2023. arxiv.org
5. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, et al.. Fine-Tuning Language Models from Human Preferences, 2019. arxiv.org
6. Hugo Touvron, Louis Martin, Kevin Stone, et al.. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. arxiv.org
7. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, et al.. RewardBench: Evaluating Reward Models for Language Modeling, 2024. arxiv.org
8. Saumya Malik, Valentina Pyatkin, Sander Land, et al.. RewardBench 2: Advancing Reward Model Evaluation, 2025. arxiv.org
9. Xueru Wen, Jie Lou, Yaojie Lu, et al.. Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?, 2024. arxiv.org
10. Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Dongha Lee, Jinyoung Yeo. Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization, 2025. arxiv.org
11. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
12. Thomas Coste, Usman Anwar, Robert Kirk, David Krueger. Reward Model Ensembles Help Mitigate Overoptimization, 2023. arxiv.org
13. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, et al.. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking, 2023. arxiv.org
14. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al.. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. arxiv.org
15. Dakota Mahan, Duy Van Phung, Rafael Rafailov, et al.. Generative Reward Models, 2024. arxiv.org
16. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal. Generative Verifiers: Reward Modeling as Next-Token Prediction, 2024. arxiv.org
17. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al.. Training Verifiers to Solve Math Word Problems, 2021. arxiv.org
18. Jonathan Uesato, Nate Kushman, Ramana Kumar, et al.. Solving math word problems with process- and outcome-based feedback, 2022. arxiv.org
19. Hunter Lightman, Vineet Kosaraju, Yura Burda, et al.. Let's Verify Step by Step, 2023. arxiv.org
20. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023. arxiv.org
21. Prasann Singhal, Tanya Goyal, Jiacheng Xu, Greg Durrett. A Long Way to Go: Investigating Length Correlations in RLHF, 2023. arxiv.org