Inter-annotator agreement (IAA) is the degree to which independent annotators give the same label to the same item. It is the basic quality signal for any labelled dataset: if trained people cannot agree on a label, a model cannot learn it reliably and an evaluation cannot score it reliably.
How it is measured
- Percent agreement: the share of items where raters match. Easy to read but inflated by chance, especially with imbalanced classes.
- Cohen's kappa: two raters, categorical labels, corrected for chance.
- Fleiss' kappa: a fixed number of raters per item drawn from a larger pool.
- Krippendorff's alpha: any number of raters, missing data, and ordinal or interval scales.
- Correlation (Spearman or Pearson): for continuous scores, though correlation ignores systematic offsets between raters.
Worked example
Illustrative numbers: two raters label 10 responses and agree on 8, so percent agreement is 0.80. Each called half the items Y, so chance agreement is , and kappa is . The gap between 0.80 and 0.60 is the part of the raw number that a coin flip would have produced anyway.
Agreement and gold sets
A gold set is a sample labelled by experts, often by adjudication after disagreement, that serves as ground truth. It has two jobs. First, it audits annotators: each rater's accuracy or kappa against gold flags individuals who misread the guideline. Second, it calibrates automated graders: an LLM judge is measured against the gold set per category before it is allowed to score that category. Gold items should be mixed blind into normal work, refreshed so they do not leak, and versioned with the policy they encode.
Practical guidance
- Report agreement per category, not just overall. Clear-cut categories hide ambiguous ones in an average.
- Treat disagreement as information. Clusters of disagreement point to guideline gaps; fix the guideline, then relabel.
- Human to human agreement is the realistic ceiling for any automated grader on that task.
Why it matters
Every eval score and every reward model inherits the noise in its labels. Preference data with low agreement trains a reward model to fit noise, and a test set with low agreement makes small score differences meaningless. Measuring IAA early is the cheapest way to find out whether a labelling effort will produce a usable signal.
Related terms
Sources
- Ron Artstein, Massimo Poesio. Inter-Coder Agreement for Computational Linguistics, 2008. doi.org