Preference data is a collection of prompts, each paired with two or more candidate responses and a judgement of which is better. It is the raw material for reward models in RLHF and for direct methods such as DPO.
What a record looks like
The common unit is a triple : a prompt , a chosen response and a rejected response . Richer formats include rankings of responses (which yield pairs), Likert-style margins ("much better" versus "slightly better"), ties, per-criterion labels such as helpfulness and harmlessness, and free-text rationales. Labels may come from people, from a model following written principles (see RLAIF and constitutional AI), or from both.
Why comparisons rather than scores?
People disagree a lot about absolute scores (is this answer a 6 or a 7?) but agree more often on which of two answers is better. A Bradley-Terry model turns those comparisons into a consistent scalar scale.
Worked example: label noise
Illustrative numbers: suppose each annotator picks the truly better response 70 percent of the time, independently. Taking a majority vote of three annotators raises accuracy to
Triple labelling buys about 8 percentage points of label accuracy at three times the cost, and the rate at which annotators disagree is also a useful signal: pairs with split votes are often genuinely ambiguous and can be down-weighted or sent for expert review. Agreement statistics such as Cohen's kappa and inter-annotator agreement should be reported with any preference set.
What makes it good
- Coverage: prompts that match deployment, including hard, borderline and adversarial ones.
- Close pairs: two plausible responses teach more than one excellent and one broken response.
- Clear guidelines: labellers need a written policy for trade-offs, for example when a refusal beats a helpful but risky answer.
- Bias checks: track preference for longer, more confident or first-shown responses, which a reward model will otherwise learn.
Why it matters
The preference set defines what "better" means for the model. Systematic biases in it become systematic behaviours after training, which is how length bias, sycophancy and over-refusal get baked in. For safety work, preference pairs that contrast a safe completion with a harmful compliance or an unnecessary refusal are among the most valuable safety training data.
Related terms
- Reward model
- RLHF (reinforcement learning from human feedback)
- DPO (Direct Preference Optimization)
- Bradley-Terry model
- Inter-annotator agreement
- Safety training data
Sources
- Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. Learning to summarize from human feedback, 2020. arxiv.org
- Long Ouyang, Jeff Wu, Xu Jiang, et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org