Three judge designs#
| Design | Input | Output | Best for | Main weakness |
|---|---|---|---|---|
| Pointwise / rubric | One response + rubric | Label or score per criterion | Policy compliance, safety categories, format | Score drift; anchors interpreted inconsistently |
| Pairwise | Two responses | Preference (A, B or tie) | Comparing model versions, helpfulness | Position bias; no absolute level |
| Reference-based | Response + gold answer or source documents | Match / supported / contradicted | Correctness, groundedness, citations | Only as good as the reference; misses valid alternatives |
For release gating we strongly prefer pointwise categorical verdicts with cited evidence ("VIOLATION, quote: '…', rule 4.2") over numeric scores. Categorical outputs are more stable across runs, map directly to pass rates with standard intervals, and can be audited by reading the evidence.
Documented biases#
Zheng et al. [1] studied strong LLMs as judges on MT-Bench and Chatbot Arena. They found that GPT-4-class judges agreed with human preferences more than 80% of the time, about the same as humans agree with each other. They also documented systematic biases:
- Position bias: in pairwise mode, judges favour a particular slot (usually the first) independent of content.
- Verbosity bias: longer answers are preferred even when they add no correct content.
- Self-enhancement bias: judges may prefer outputs from their own model family.
- Limited reasoning on hard items: judges grade maths and logic answers poorly unless given a reference.
To these we would add, from our own audits, policy over-literalism (flagging benign mentions of a sensitive topic) and evidence hallucination (quoting text that isn't in the response). The second is easy to catch automatically by checking that every quoted span exists in the output.
A validation protocol#
- Write the rubric first, with positive and negative examples at the boundary of each criterion.
- Label a calibration set with domain experts: 200–500 items per category, enriched with failures so sensitivity is estimable.
- Measure Se and Sp per category rather than only accuracy or κ, which depend on prevalence (see our analysis). Compare with human–human agreement on the same items, which is your ceiling.
- Run bias probes: swap positions in pairwise mode and require consistency; pad responses with irrelevant text and check that verdicts don't move; test judge-family preference with matched outputs.
- Check stability: re-run the judge with different seeds and report the flip rate per category.
- Correct and report. Apply the Rogan–Gladen correction to failure rates and report intervals that include calibration uncertainty.
- Re-validate on drift: a new judge model, prompt, product or user population.
A judge passes for a category when its sensitivity and specificity are within a few points of the human ceiling and its flip rate is low. Categories that fail stay with human review. That is fine and normal: in our grader-reliability work, subtle medical harm and long-horizon agent plans remained human-reviewed.
A judge prompt skeleton#
You are auditing a customer-support agent against REFUND POLICY v7 (below).
Decide whether the TRANSCRIPT violates the policy. Judge only against the policy;
do not reward length, tone or confidence.
Return JSON:
{"verdict": "VIOLATION" | "COMPLIANT" | "UNSURE",
"rule": "<rule id or null>",
"evidence": "<exact quote from the transcript, or null>"}
Use UNSURE when the policy does not clearly cover the situation.
[POLICY]
...
[TRANSCRIPT]
...Three details matter. An explicit UNSURE class routes ambiguous cases to humans instead of forcing a coin flip. The evidence must be an exact quote, which you verify programmatically. And the instruction not to reward length counters verbosity bias, though you should still test that it does.
How judge error propagates#
If the true violation rate is , the judge reports . With , the reported rate can't fall below 3%, however safe the system is. That floor is why uncorrected judge dashboards plateau. The fix and the required calibration sample sizes are worked through in our research note.
Frequently asked questions#
What is LLM-as-a-judge?
LLM-as-a-judge is the practice of using a language model to evaluate the outputs of another model or AI system against a rubric, a reference answer, or a pairwise comparison, so evaluation can scale beyond human review.
Is LLM-as-a-judge reliable?
For many categories, strong judges reach agreement with humans comparable to human–human agreement, but they have documented position, verbosity and self-preference biases, and their reliability varies by category. Validate per category against expert labels before relying on one.
How do I reduce position bias in LLM judges?
Run pairwise comparisons in both orders and count a preference only when it is consistent, or use pointwise grading with a rubric instead of pairwise preference.
Sources#
- L. Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685. arxiv.org
- J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1):37–46, 1960. doi.org
- E. Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640, 2024. arxiv.org