Cohen's kappa

Cohen's kappa ($\kappa$) is an agreement statistic for two raters assigning items to categories. It reports how far observed agreement exceeds chance agreement, scaled so that 1 means perfect agreement and 0 means no better than chance.

Formula

Here $p_o$ is the fraction of items on which the two raters agree, $p_e$ is the agreement expected if both labelled at random with their own marginal rates, and $p_{A,c}$ and $p_{B,c}$ are the shares of items each rater put in category $c$. The denominator is the most agreement that could possibly be gained above chance, so $\kappa$ is the fraction of that headroom actually achieved.

Worked example

Illustrative numbers: two annotators label 100 model responses as safe or unsafe. Both say unsafe on 40, both say safe on 30, and they split on the other 30. Raw agreement is $p_o=0.70$, which sounds decent. Annotator A called 50 responses unsafe and annotator B called 60 unsafe, so chance agreement is $p_e=0.5\times 0.6+0.5\times 0.4=0.50$. Then $\kappa =(0.70-0.50)/(1-0.50)=0.40$: the raters captured only 40% of the possible above-chance agreement.

Reading the number

• Common rules of thumb treat values above roughly 0.8 as strong and 0.6 to 0.8 as substantial, but these cut-offs are conventions, not tests.
• Kappa depends on prevalence. When one category is rare, $p_e$ is high and kappa can be low even with 95% raw agreement. Always report raw agreement and class balance alongside it.
• For ordinal scales, weighted kappa gives partial credit for near misses. For more than two raters or missing labels, use Krippendorff's alpha or Fleiss' kappa.

Why it matters

Kappa is the standard check that a labelling policy is clear enough to produce consistent data. In AI evaluation it is used twice: between human annotators building a safety dataset, and between a human gold label and an automated grader such as an LLM judge. A judge whose kappa against experts is well below the human to human kappa should not be trusted to score that category on its own. Low kappa usually points to an ambiguous rubric rather than careless raters, so the fix is to rewrite the guideline and relabel.

Related terms

• Inter-annotator agreement
• Krippendorff's alpha
• LLM-as-a-judge
• Preference data
• Safety training data

Sources

1. Jacob Cohen. A Coefficient of Agreement for Nominal Scales, 1960. doi.org