Your LLM judge is a diagnostic test: correcting measured failure rates

A judge is a screening test#

When a model-based grader labels outputs as PASS or FAIL, it behaves exactly like a medical screening test applied to a population. Two numbers characterise it:

If the true failure rate (the prevalence) is $\pi$, the rate the judge reports is

The second term is the problem. Every passing output has a $1-\mathrm{Sp}$ chance of being flagged, and in a mature system almost every output passes. False positives from the large passing majority swamp the true failures.

What that does at realistic failure rates#

We fixed $\mathrm{Se}=0.9$ and $\mathrm{Sp}=0.97$, which is better than many production judges we audit, and varied the true failure rate.

Two practical consequences follow. First, an improving system appears to plateau: once $\pi$ falls below about $1-\mathrm{Sp}$, further real improvement barely moves the dashboard. Second, comparing two systems with the same judge understates the difference between them, because both readings are pulled towards the same floor.

Why κ collapses even though accuracy doesn't#

Teams often validate a judge by reporting Cohen's κ [3] against human labels. κ corrects observed agreement $p_o$ for chance agreement $p_e$:

Holding Se and Sp fixed, accuracy stays near 97% at every prevalence, but κ falls from 0.88 at a 30% failure rate to 0.22 at 0.5% (Figure 2). Nothing about the judge changed. When one class is rare, chance agreement is already very high, so κ has little room left to reward. This is the well-known prevalence dependence of κ, and it explains why a judge validated on a balanced sample looks far worse when it is re-validated on production traffic.

The correction: Rogan–Gladen#

Solving the equation for $q$ gives the estimator Rogan and Gladen proposed for screening surveys [1]:

truncated to $[0,1]$. It needs $\mathrm{Se}+\mathrm{Sp}>1$ (the judge must beat a coin flip), and it becomes unstable as that sum approaches 1. The point estimate is the easy part. The catch is that $\widehat{\mathrm{Se}}$ and $\widehat{\mathrm{Sp}}$ come from a finite human-labelled calibration set, and their uncertainty propagates into $\hat{\pi }$. A delta-method approximation of the variance is

With a low failure rate, $\mathrm{Var}(\widehat{\mathrm{Sp}})$ dominates: $(1-\pi {)}^2\approx 1$, and Sp is estimated from the passing cases in the calibration set. So the precision of the corrected estimate is mostly limited by how well you know the judge's false-positive rate.

Simulation: how many human labels?#

Setup. True failure rate $3\%$; 20,000 production outputs scored by the judge; a calibration set of $m$ outputs sampled from the same traffic and labelled by humans (assumed correct). Se and Sp are estimated from the calibration set, then the Rogan–Gladen estimate is computed. 3,000 replications per $m$.

Findings. The corrected estimate is centred on the truth from $m=100$ on (median 2.98%), but at $m=100$ its 95% band runs from 0.0% to 7.6%, too wide to act on. At $m=400$ the band is 0.9%–5.1%, and at $m=1,600$ it is 2.0%–4.1%. With a random calibration sample at a 3% failure rate, the bottleneck is Se: 400 random cases contain only about 12 true failures. Enriching the calibration set with known failures (stratified sampling, then reweighting) gets the same precision with far fewer labels.

The same arithmetic for pairwise win rates#

Pairwise judges ("is A better than B?") suffer an analogous attenuation. Suppose the judge reports the wrong winner with probability $e$, independent of which response is truly better. If model A truly wins a fraction $w$ of comparisons, the observed win rate is

Errors shrink every observed win rate towards 50%. A true 60% win rate measured with $e=0.15$ appears as $0.15+0.7\times 0.6=57\%$. Two consequences: real improvements look smaller than they are, so you need more comparisons to detect them, and model rankings stay correct while the margins shrink. Position bias is a non-symmetric error. Run each comparison in both orders and count only consistent verdicts, which turns position bias into a measurable "inconsistent" rate rather than a hidden shift.

What we do in practice#

1. Stratify the calibration set. Label roughly equal numbers of judge-PASS and judge-FAIL outputs, then estimate the judge's positive and negative predictive values and reweight to the production mix. Rare failures stop starving the Se estimate.
2. Report the corrected rate with its interval, and show the uncorrected judge rate next to it so nobody compares numbers across methods.
3. Re-calibrate on drift. Se and Sp are properties of the judge on a distribution. A new base model, prompt or user population can shift them. We re-label a small rolling sample every week.
4. Prefer categorical verdicts with cited evidence over 1–10 scores. They are easier to calibrate and audit, and position and verbosity biases [2] are easier to test with swapped-order and length-controlled pairs.

Limitations#

Frequently asked questions#

Sources#

1. W. J. Rogan and B. Gladen. Estimating prevalence from the results of a screening test. American Journal of Epidemiology 107(1):71–76, 1978. doi.org
2. L. Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685. arxiv.org
3. J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1):37–46, 1960. doi.org