Preference label noise: annotator agreement, Bradley-Terry reward models and how many labels per pair

Every reward model and every DPO run inherits the quality of its preference labels, and preference labels are noisy: in widely cited RLHF datasets, two trained annotators agree on which response is better only about three times in four. This note separates the two kinds of disagreement that produce that number, shows what each does to a Bradley-Terry reward model and to DPO, and works out how many annotators per pair you should pay for, depending on whether the labels are for training or for evaluation. The numbers come from seeded simulations of a linear reward model, which we ran for this note.

Same agreement rate, different problem#

The InstructGPT paper reports that its training labelers agreed with each other 72.6% of the time (plus or minus 1.5%), and notes that researcher-researcher agreement in the earlier summarisation work was 73% [1][2]. In the MT-Bench study, two human experts agreed 81% of the time once ties were excluded and 63% with ties included [3]. What should a reward-model team conclude from an agreement rate like 73%?

Not much, on its own. Here are two simulated annotation pools that both produce 72.6% agreement:

• Pool A: subtle pairs, careful annotators. The two responses in a pair are usually close in quality, so annotators genuinely differ on many of them, but almost nobody clicks carelessly (flip rate 1.3%).
• Pool B: clear pairs, careless annotators. Pairs are usually far apart, so a careful annotator would agree with the truth almost always, but 9.8% of labels are random flips.

Train a reward model on pool A and you need about 9% more labels than with perfectly clean labels to reach the same accuracy. Train on pool B and you need 90% more. Agreement cannot tell the pools apart. Disagreement on close pairs is mostly harmless, because getting a near-tie wrong costs little; flips on clear pairs teach the model the opposite of the truth. Knowing which pool you have requires a different measurement, which we come to below.

Bradley-Terry with two kinds of noise#

The Bradley-Terry model [4] that underlies reward modelling [5][6] and DPO [7] says that an annotator prefers response $a$ over $b$ with probability

where $r$ is a scalar reward, $\Delta =r(a)-r(b)$ is the reward gap and $\sigma (x)=1/(1+e^{-x})$ is the logistic function. A large gap gives a near-certain preference; a zero gap gives a coin flip. This already contains one kind of disagreement: ambiguity, which is concentrated on pairs with small $|\Delta |$. Add the second kind, flip noise, where with probability $\epsilon$ the label is reversed regardless of the gap (a misclick, a skimmed response, a misunderstood guideline):

$\epsilon$ is the flip rate. At $\epsilon =0$ this is plain Bradley-Terry; at $\epsilon =0.5$ labels are pure noise. Flip noise caps the probability of agreeing with the truth at $1-\epsilon$ even for the clearest pairs. Two annotators labelling the same pair independently agree with probability $\mathbb{E}[p^2+(1-p{)}^2]$, with $p$ the expression above, averaged over the distribution of gaps. Both ambiguity and flips lower that number, which is why agreement alone cannot separate them.

In our simulation responses are 32-dimensional feature vectors, the true reward is linear in the features, and within-pair gaps $\Delta$ are normally distributed with standard deviation 2.5 (the setting of pool A, where ambiguity alone gives 73.8% agreement). The reward model is a linear Bradley-Terry model fitted by maximum likelihood. We score it by gold accuracy: on 20,000 fresh pairs, how often it ranks the truly better response first. Every point averages 12 independent runs.

What flip noise does to a reward model#

It costs data, not (eventually) direction#

With symmetric flip noise, a Bradley-Terry model still learns the right ranking given enough data, because flips do not favour either response. What noise destroys is information per label. Figure 1 shows gold accuracy against the number of singly labelled pairs.

Relative to clean labels, flip rates of 0.1, 0.2 and 0.3 multiply the data needed by 2.2, 4.6 and 9.8. A useful approximation comes from the Fisher information each label carries about the reward gap:

$p_{\epsilon }(\Delta )$ is the labelling probability above, ${\partial }_{\Delta }$ its slope with respect to the gap, and $N_{\epsilon }$ the number of labels needed at flip rate $\epsilon$. A label is informative where the labelling probability changes quickly with the gap and is not already near 0 or 1. Flips flatten the slope by a factor $1-2\epsilon$ and pull the probabilities towards one half. For our setting the formula gives 1.8, 3.5 and 8.2. It understates the simulated cost by 15% to 25%, because the standard Bradley-Terry fit does not know about the flips; treat it as a lower bound. The interactive figure below uses it.

It compresses the reward scale#

The second effect matters for what you do with the reward model afterwards. Fitted on 8,000 clean pairs, the model recovers 95% of the true reward scale (the projection of the learned weights on the true weights). Fitted on 8,000 pairs with $\epsilon =0.2$, it recovers 37%: the model is right about direction but every reward gap is compressed by almost two-thirds, because the fit explains the flips by being less decisive everywhere. A noise-aware likelihood that uses the known $\epsilon$ in place of $\sigma (\Delta )$ recovers 84% of the scale, and edges gold accuracy up from 95.6% to 96.0%. If you tune a KL coefficient or a best-of-$n$ threshold against reward magnitudes, a change in annotation quality silently changes what those settings mean, and it interacts with reward overoptimisation in ways that are hard to see from the reward curve alone.

The same story for DPO#

DPO [7] skips the explicit reward model and fits the policy directly with the Bradley-Terry loss on an implicit reward, $\hat{r}(x,y)=\beta \log {\pi }_{\theta }(y\mid x)/{\pi }_{\text{ref}}(y\mid x)$. The loss is the same logistic loss on the same labels, so flipped labels have the same two effects: more data for the same quality, and compressed implicit reward margins. Two corrections are in use. Label smoothing (sometimes called conservative DPO) replaces the hard target with $1-\epsilon$; it stops the model from becoming arbitrarily confident on any single pair, but in expectation it does not equal the clean loss. Robust DPO [8] builds an unbiased estimate of the clean loss from noisy labels:

$\ell (y_w,y_l)$ is the usual DPO loss with $y_w$ as the preferred response, $\ell (y_l,y_w)$ the loss with the roles swapped, and $\epsilon$ the assumed flip rate. In expectation over the flips, this equals the loss you would have computed on clean labels. Both corrections need an estimate of $\epsilon$, and both assume the noise is symmetric and independent of content. Neither fixes systematic preferences such as a lean towards longer answers, which no amount of averaging removes.

A complementary approach treats each pair individually. Work on reward modelling for RLHF has used an ensemble of reward models to estimate how strongly the data supports each preference, identified pairs that appear mislabeled or ambiguous, and handled them by flipping, label smoothing or adaptive margins [9]. That works when the ensemble is better than the labels on the pairs in question, which is plausible for flips on clear pairs and implausible for ambiguity.

How many annotators per pair?#

This is the question with the biggest budget consequences, and the answer depends on what the labels are for.

For evaluation, repetition raises the ceiling#

When you report reward-model accuracy on a held-out set labelled by humans, the labels are noisy too. Even a perfect reward model that always ranks the truly better response first will "disagree" with every flipped or ambiguous test label. Its measured accuracy is

where $m$ is the number of annotators per test pair (odd), $p_m$ the probability that their majority agrees with the true ordering, and $q=\epsilon +(1-2\epsilon )\sigma (|\Delta |)$ the probability that one annotator does. $A_m$ is a hard ceiling on any model's measured accuracy. With single labels, flip rate 0.1 and our gap distribution, the ceiling is 74.9%. Three annotators per pair raise it to 82.3%, five to 85.9%. A reward model reported at 72% accuracy on singly labelled test pairs may be within three points of perfect.

For training, spread the budget across pairs#

For training, the arithmetic reverses. Under symmetric flip noise, a second label on the same pair is worth less than a first label on a new pair, because new pairs also bring new directions in response space. We gave each configuration a fixed budget of 6,000 labels:

Gold accuracy of the trained reward model, 12 runs each; differences under about half a point are within run-to-run noise. Majority voting is the worst way to use repeated labels: it throws away the vote margin, which carries information about how close the pair is. Using the vote fraction as a soft target recovers most of the loss, and at high noise roughly ties with single labelling. The Fisher formula agrees on the ordering: relative label cost at $\epsilon =0.2$ is 3.5 for one label per pair and 4.9 for three-way majority.

Adjudication#

A common middle path for evaluation sets: two annotators per pair, and a senior reviewer only when they disagree. We modelled the reviewer as more discriminating (twice the sensitivity to the reward gap) with a 3% flip rate, at four times the cost of a regular label.

Adjudication beats five-way majority at about 70% of the cost, and its advantage grows with noise because the reviewer is called in exactly where the regular annotators disagree. It also produces a by-product worth more than the labels: a stream of disagreements with an authoritative resolution, which is the raw material for improving the guidelines.

Measuring the noise in your own data#

To choose between the strategies above you need $\epsilon$, and agreement does not give it to you. Three measurements do.

Sentinel pairs. Seed the queue with pairs whose answer is not in doubt (one response is correct and the other demonstrably wrong, or one follows the instruction and the other ignores it). Errors on sentinels estimate each annotator's flip rate directly, because ambiguity is absent by construction. Five to ten percent of the queue is typical.

Annotator confusion models. With repeated labels, the Dawid-Skene model [10] estimates each annotator's error rates and the latent true label jointly by expectation-maximisation. It separates "this annotator is noisy" from "this pair is hard" without any gold labels, and early crowdsourcing work showed that a handful of modelled non-expert labels can match an expert [11].

Agreement as a function of gap. Once you have a reward model, bin pairs by its predicted gap and plot human agreement in each bin. Pure ambiguity gives agreement near 50% at small gaps rising towards 100% at large gaps. Flip noise shows up as a plateau below 100% even at the largest gaps; the height of the plateau is roughly $1-2\epsilon (1-\epsilon )$.

Code: noise-aware Bradley-Terry and soft labels#

import numpy as np
from scipy.optimize import minimize
from scipy.special import expit

def fit_bt(Z, y, eps=0.0, l2=1e-3):
    """Z: (n, d) feature differences x_a - x_b. y: soft label in [0, 1]
    (vote fraction for a). eps: known flip rate (0 gives standard BT)."""
    n = len(Z)
    def f(w):
        s = expit(Z @ w)
        q = np.clip(eps + (1 - 2 * eps) * s, 1e-12, 1 - 1e-12)
        loss = -np.mean(y * np.log(q) + (1 - y) * np.log(1 - q)) + l2 * w @ w
        dq = (1 - 2 * eps) * s * (1 - s)
        grad = -(Z.T @ ((y / q - (1 - y) / (1 - q)) * dq)) / n + 2 * l2 * w
        return loss, grad
    return minimize(f, np.zeros(Z.shape[1]), jac=True, method="L-BFGS-B").x

def flip_rate_from_sentinels(errors, total):
    # sentinel pairs have no ambiguity, so the error rate estimates eps directly
    return errors / total

Checklist#

Failure modes#

Comparing model accuracy across test sets with different ceilings. A reward model at 70% on singly labelled pairs and one at 80% on triply labelled pairs may be equally good. Always state how the test labels were made.

Forced choices on ties. If the interface forbids "about the same", near-ties become coin flips that look like flip noise. Allow ties, or a strength rating, and drop or down-weight them in training.

Correlated annotators. Annotators trained together share misreadings of the guidelines. Their errors are not independent, so majority voting among them fixes less than the binomial formula suggests.

Position and presentation effects. If the preferred response tends to appear on one side of the screen, a reward model can learn the position signal. Randomise order and log it.

Treating DPO as immune. Removing the explicit reward model removes a place to inspect the noise, not the noise itself.

What this simulation does not capture#

The reward is linear in features, responses are Gaussian and the reward model is correctly specified apart from the noise, so the simulation isolates label effects from representation learning. Flip noise is symmetric, independent across annotators and unrelated to content. Real annotation has annotator-specific rates, content-dependent errors and systematic biases. We did not train a language model or run DPO; the DPO conclusions follow because its loss is the same logistic loss on the same labels, but optimisation effects specific to large models are not modelled.

Frequently asked questions#

Sources#

1. Long Ouyang, Jeff Wu, Xu Jiang, et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org
2. Nisan Stiennon, Long Ouyang, Jeff Wu, et al.. Learning to summarize from human feedback, 2020. arxiv.org
3. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al.. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. arxiv.org
4. Ralph Allan Bradley, Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons, 1952. doi.org
5. Paul Christiano, Jan Leike, Tom B. Brown, et al.. Deep reinforcement learning from human preferences, 2017. arxiv.org
6. Yuntao Bai, Andy Jones, Kamal Ndousse, et al.. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022. arxiv.org
7. Rafael Rafailov, Archit Sharma, Eric Mitchell, et al.. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023. arxiv.org
8. Sayak Ray Chowdhury, Anush Kini, Nagarajan Natarajan. Provably Robust DPO: Aligning Language Models with Noisy Feedback, 2024. arxiv.org
9. Binghai Wang, Rui Zheng, Lu Chen, et al.. Secrets of RLHF in Large Language Models Part II: Reward Modeling, 2024. arxiv.org
10. A. P. Dawid, A. M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm, 1979. doi.org
11. Rion Snow, Brendan O'Connor, Daniel Jurafsky, Andrew Ng. Cheap and Fast, But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks, 2008. aclanthology.org