Side by side#
| LLM judge | Generalist human | Domain expert | |
|---|---|---|---|
| Marginal cost per label | Fractions of a cent to a few cents | Roughly $0.10–$1 | Roughly $2–$30+ (complex medical or legal items) |
| Latency | Seconds | Hours to days | Days |
| Consistency over time | High for a fixed model and prompt; shifts on model updates | Drifts with fatigue and turnover | Good, but with real disagreement on hard items |
| Known biases | Position, verbosity, self-preference [1] | Fatigue, anchoring, instruction misreading | Specialty-specific priors |
| Can judge subtle domain harm | Category-dependent; must be validated | Often not | Yes: this is the ground truth |
| Auditability | Evidence quotes, reproducible | Rationales if required | Rationales if required |
Cost ranges are our planning estimates for budgeting, not quotes; they vary widely by task length and expertise.
Hybrid routing#
Route by category reliability and judge confidence:
- Every item is graded by the judge.
- Categories where the validated judge matches the human ceiling: accept judge verdicts, and audit a random 1–2% with humans to track drift.
- Items the judge marks
UNSURE, or where two judge runs disagree: route to humans. - High-severity categories (clinical harm, dangerous capability): route all judge-flagged items and a random sample of judge-passed items to domain experts.
The expected human cost per item is then
where is the audit rate, the uncertain-or-disagreeing rate, and the expert-review share for high-severity items. With and , human cost falls by about 92% relative to full human review, and the audit sample still lets you detect judge drift.
When humans are non-negotiable#
- Building the gold set that validates the judge in the first place.
- Categories where judges fail validation. In our experience these are subtle clinical harm, persuasion and manipulation, legal-advice boundaries, and long-horizon agent plans.
- Policy changes. When the rubric changes, humans must re-establish ground truth before the judge is re-validated.
- Regulatory evidence, where a reviewer needs to see qualified human judgement in the loop.
Measuring the ceiling#
Human evaluation isn't error-free. Before comparing a judge with humans, measure human–human agreement on the same items; κ [2] between two experts on hard safety items is often well below 1. A judge that matches the expert–expert κ is as good as adding another expert. A judge that "disagrees with humans 15% of the time" may simply be inside the range where the experts disagree with each other.
Frequently asked questions#
Is LLM-as-a-judge cheaper than human evaluation?
Per label, by two to four orders of magnitude. But it needs a human-labelled calibration set to be trustworthy, and some categories still need human review, so the realistic saving is in routing most items to the judge, not eliminating humans.
How accurate is LLM-as-a-judge compared to humans?
On open-ended chat preference, strong judges reached over 80% agreement with humans in the MT-Bench study, comparable to human–human agreement. Accuracy on specialised safety categories varies and must be measured per category.