Krippendorff's alpha

Krippendorff's alpha ($\alpha$) is a reliability coefficient that measures how much raters agree beyond chance. Unlike Cohen's kappa, it works with any number of raters, tolerates missing ratings, and supports nominal, ordinal, interval and ratio scales through a choice of distance function.

Formula

$D_o$ is the observed disagreement: the average distance between pairs of labels given to the same item. $D_e$ is the expected disagreement: the average distance between pairs of labels drawn from the pooled set of all labels, ignoring which item they came from. For nominal data the distance is 0 for a match and 1 for a mismatch; for interval data it is the squared difference. So $\alpha =1$ means perfect reliability and $\alpha =0$ means raters agree no more than labels shuffled across items would.

Worked example

Illustrative numbers: two raters mark 10 responses as compliant (Y) or not (N), disagreeing on 2 items. That gives 20 pairable values, 10 of each label. Within items, 4 of the 20 ordered pairs mismatch, so $D_o=4/20=0.20$. Across the pooled labels, a random pair of distinct values mismatches with probability $D_e=(2\times 10\times 10)/(20\times 19)\approx 0.526$. Hence $\alpha =1-0.20/0.526\approx 0.62$. On the same data Cohen's kappa is 0.60; alpha is slightly higher because it estimates expected disagreement without replacement, a small-sample correction.

Interpretation and pitfalls

• Krippendorff suggested $\alpha \ge 0.8$ for data you will draw conclusions from and $\alpha \ge 0.667$ as a floor for tentative ones.
• Pick the distance metric to match the scale. Treating a 1 to 5 harm-severity rating as nominal throws away the fact that 4 versus 5 is a smaller disagreement than 1 versus 5.
• Items rated by only one person carry no reliability information and drop out of the calculation.
• Bootstrap over items to get a confidence interval; a point estimate from 50 items is noisy.

Why it matters

Real annotation projects rarely have the same two raters on every item. Alpha is the statistic that fits how safety data and preference data are actually collected: rotating pools, overlap on a subset, and graded scales. Reporting it per category shows where a labelling policy is clear and where it needs work, and it sets the ceiling any automated grader can be expected to reach.

Related terms

• Cohen's kappa
• Inter-annotator agreement
• Preference data
• Safety training data
• LLM-as-a-judge

Sources

1. Andrew F. Hayes, Klaus Krippendorff. Answering the Call for a Standard Reliability Measure for Coding Data, 2007. doi.org