Bradley-Terry model

The Bradley-Terry model is a probability model for pairwise comparisons: each item has a latent strength, and the chance that one item is preferred over another depends only on the difference between their strengths. It is the standard link between preference data and reward models, and it also underlies Elo-style leaderboards.

Definition

Here $p_i>0$ is the strength of item $i$, $s_i=\log p_i$ is the same strength on a log scale, and $\sigma (z)=1/(1+e^{-z})$ is the logistic function. Adding the same constant to every $s_i$ changes nothing, so scores are identified only up to an offset. Fitting the model to observed wins is maximum likelihood, which is logistic regression on score differences.

Worked example

Three responses have scores $s_A=1.5$, $s_B=0.5$ and $s_C=0$.

• $P(A\succ B)=\sigma (1.0)\approx 0.73$
• $P(B\succ C)=\sigma (0.5)\approx 0.62$
• $P(A\succ C)=\sigma (1.5)\approx 0.82$

The model forces consistency: once A beats B and B beats C by those margins, A versus C is determined. Real preferences are not always transitive, which is one way the model can misfit.

In reward modelling

An RLHF reward model sets $s=r_{\varphi }(x,y)$ and minimises $-\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))$ over labelled pairs. DPO plugs an implicit reward, $\beta \log {\pi }_{\theta }/{\pi }_{\mathrm{ref}}$, into the same likelihood. Model leaderboards built from head-to-head votes fit Bradley-Terry scores and rescale them so that a 400-point gap means 10 to 1 odds, the Elo convention.

Assumptions to keep in mind

• A single scalar quality per item: no room for "better for experts, worse for novices".
• Independent comparisons with no position or rater effects, unless those are added as covariates.
• No ties; extensions such as Rao-Kupper or Davidson handle them.

Why it matters

Almost every preference-based training and ranking method inherits Bradley-Terry's assumptions. When annotators disagree for principled reasons, or when preferences depend on who is asking, a single score averages the disagreement away. Knowing the model helps diagnose why a reward model plateaus near annotator agreement, and why leaderboard gaps of a few points are often inside the noise.

Related terms

• Reward model
• Preference data
• DPO (Direct Preference Optimization)
• RLHF (reinforcement learning from human feedback)
• Item response theory

Sources

1. Ralph Allan Bradley, Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons, 1952. doi.org
2. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei. Deep reinforcement learning from human preferences, 2017. arxiv.org