LLM benchmarks explained: what they measure, saturation, contamination and how to read leaderboards

LLM benchmarks are shared, public test sets with fixed scoring rules that let anyone compare language models on the same tasks: general knowledge (MMLU, GPQA), maths (GSM8K, MATH), code (HumanEval, SWE-bench), instruction following (IFEval), truthfulness, agentic tool use and human preference (Chatbot Arena). A benchmark score is useful for shortlisting models, but it is a noisy estimate under one particular harness, many popular benchmarks are near saturation or contaminated by training data, and leaderboard rankings among close models are often within the noise.

This guide explains what the major benchmarks measure, how scores are produced, why benchmarks saturate and leak, and how to read a leaderboard without being fooled. Seeded simulations show how ranking reliability depends on benchmark size, answer-key errors, contamination and selective reporting, and you can try all four in the leaderboard reader below.

What do the major LLM benchmarks measure?#

Benchmarks are easiest to understand by the capability they target. The table lists widely cited examples with the facts reported in their papers; "status" notes how informative each still is for comparing strong models.

Two patterns are visible. Benchmarks drift from "too hard" to "solved" in a few years, and each generation of benchmark is a response to the last one's saturation or contamination: MMLU-Pro and Humanity's Last Exam to MMLU, GSM1k to GSM8K, LiveBench and LiveCodeBench to static sets. And the field has moved from single-turn questions to agentic tasks with environments, where scoring checks outcomes rather than text.

How is a benchmark score produced?#

A benchmark is a dataset plus a harness, and the harness matters as much as the data. For multiple-choice tasks, a harness can compare the model's log-probabilities of the answer options or let the model generate an answer and extract a letter; it can use zero or five worked examples in the prompt; it can ask for step-by-step reasoning or not. MMLU-Pro's authors measured that varying the prompt style moved MMLU scores by 4 to 5 points, against about 2 points on MMLU-Pro. Madaan et al. measured how much scores move with training seed alone, which is often comparable to the gaps papers report.

Practical consequence: scores from different papers or model cards are only comparable when the harness, prompt, number of examples, decoding settings and answer extraction are the same. When a leaderboard runs every model itself under one harness, its numbers are comparable to each other but may not match anyone's self-reported numbers. Which metric a benchmark uses (accuracy, exact match, pass@k, win rate) shapes what it rewards; LLM evaluation metrics covers each one.

What is benchmark saturation?#

A benchmark is saturated when the models you want to compare all score near its ceiling, so differences between them are smaller than the noise. Ott et al. mapped 3,765 benchmarks across computer vision and natural language processing and found that a large fraction quickly trended towards near-saturation. The ceiling is usually below 100%, because some items are wrong. Gema et al. re-annotated 5,700 MMLU questions and estimated that 6.49% contain errors, including a majority of the questions they analysed in the virology subset.

Mis-keyed items do two kinds of damage. They cap the score: on a 4-option benchmark where 6.5% of keys are wrong, a model that answers every question correctly scores about $1-0.75\times 0.065\approx 95.1\%$, since it only matches a wrong key by chance a quarter of the time. And they add noise that has nothing to do with ability, compressing real differences.

Simulation. Ten models whose true accuracies on correctly keyed items are spaced exactly one point apart, scored on benchmarks of 300 to 14,000 items, 3,000 replications each. We record how often the truly best model ranks first.

The general rule is that a one-point gap between models needs thousands of items to resolve, and most benchmarks have hundreds to a few thousand. GPQA's 448 questions give a standard error of about 2.4 points at 50% accuracy; a two-point gap on it is not a ranking. For the statistics of comparing two models on the same items, which is more efficient than comparing their separate scores, see statistical significance for evals; for items that share passages or templates, see clustered standard errors. Our research on item response theory shows how to identify which items still discriminate between strong models.

What is benchmark contamination?#

Contamination happens when benchmark items, or close paraphrases, appear in a model's training data. The model can then answer from memory, and its score overstates its ability on new problems. Sainz et al. distinguish levels, from the test set itself appearing in pretraining data to the benchmark's format and answer distribution being learned during fine-tuning, and argue that the extent is unknown for most public benchmarks.

The arithmetic is simple: if a fraction $c$ of items is memorised and the model would otherwise answer correctly with probability $a$, the observed score is

so memorising 10% of items lifts a 60% model to 64% and an 80% model to 82%. That is the size of gap that separates adjacent models on most leaderboards.

The cleanest evidence comes from building a fresh copy of a benchmark. Zhang et al. commissioned GSM1k, a new set matched to GSM8K in style and difficulty, and observed accuracy drops of up to 8% for some model families, with a positive relationship between a model's likelihood of generating GSM8K examples and its drop. Many frontier models showed little sign of overfitting, which is the other half of the lesson: contamination is model-specific, so it can reorder a leaderboard. Our research on contamination detection covers the statistical tests, and membership inference covers the attacks that detect memorised items.

Defences include canary strings embedded in benchmark files so training pipelines can filter them (BIG-bench ships one), keeping test sets private, and live benchmarks that keep adding items created after models' training cutoffs, as LiveBench and LiveCodeBench do.

How should you read an LLM leaderboard?#

Leaderboards add two problems on top of the benchmark's own noise.

Selective disclosure. If a developer can test several private variants and publish only the best score, the published score is a maximum of noisy measurements and is biased upward even when every variant is equally good. With a measurement standard error $s$, the expected inflation from publishing the best of $N$ equal variants is $s$ times the expected maximum of $N$ standard normal draws. We computed it by simulation: 0.85$s$ for 3 variants, 1.54$s$ for 10 and 2.0$s$ for 27. On a 1,000-item benchmark at 80% accuracy, where $s\approx 1.26$ points, testing 10 variants adds about 1.9 points and 27 variants about 2.5 points, with no real improvement at all. Singh et al.'s The Leaderboard Illusion documented undisclosed private testing on a major crowdsourced arena, including one developer testing 27 private variants before a release, together with large asymmetries in how much arena data different providers received.

Gameable judges. Leaderboards that use a model as the judge inherit its biases. Zheng, Pang and colleagues showed that a "null model" returning one constant, carefully crafted response reached an 86.5% length-controlled win rate on AlpacaEval 2.0, and high scores on Arena-Hard-Auto and MT-Bench. Length control helps, as Dubois et al. showed, but does not close every exploit. The guide to LLM-as-a-judge covers the general problem.

A checklist for reading a leaderboard#

1. Is there an interval? If not, compute one: $1.96\sqrt{p(1-p)/n}$ for accuracy. Treat models whose intervals overlap heavily as tied, and remember that a paired comparison on the same items is sharper than overlapping intervals suggest.
2. How many items? Under about 1,000, one-point gaps are noise.
3. Is it near the ceiling? Check published error rates in the benchmark; scores within a few points of the ceiling do not rank.
4. Same harness for everyone? Self-reported scores with different prompts are not comparable.
5. Who chose what to submit? Ask whether developers could test variants privately or retract scores.
6. Could the items be in training data? Prefer benchmarks with private or continuously refreshed test sets, and treat large gains on old public sets with suspicion.
7. Does the task match yours? A coding benchmark says little about a customer-support agent; human-preference arenas reward style as well as substance.

Benchmarks are for shortlisting, evals are for deciding#

Use benchmarks to narrow the field: which models are plausibly strong at the capabilities your application needs. Then run your own evaluation on your own tasks, prompts, tools and policies, because a model that tops a general leaderboard can still mishandle your refund rules or follow an instruction hidden in a retrieved document. Our LLM evaluation guide covers how to build that suite, and what AI evals are places benchmarks within the full taxonomy of evaluations. A private, task-specific eval also avoids contamination by construction, as long as it never leaves your systems.

Benchmarks also matter for safety work, where the same issues apply with higher stakes: a saturated refusal benchmark says nothing about adaptive attackers, which is why red teaming complements evals, and why dangerous capability evaluations put so much effort into eliciting the strongest performance rather than measuring a default one. For RL agents rather than language models, the same reasoning about seeds and aggregates is covered in how to evaluate RL agents.

Code: a leaderboard sanity check#

import numpy as np

def leaderboard_check(scores, n_items, n_variants=1, seed=0, reps=20000):
    """scores: dict model -> published accuracy on the same n_items benchmark.
    Returns 95% half-widths and the expected best-of-N inflation for a given variant count."""
    rng = np.random.default_rng(seed)
    out = {}
    for m, p in sorted(scores.items(), key=lambda kv: -kv[1]):
        half = 1.96 * np.sqrt(p * (1 - p) / n_items)
        out[m] = (round(100 * p, 1), round(100 * half, 1))
    se = np.sqrt(np.mean([p * (1 - p) for p in scores.values()]) / n_items)
    inflation = rng.normal(0, se, (reps, n_variants)).max(1).mean()
    return out, round(100 * inflation, 2)

table, infl = leaderboard_check({"A": 0.874, "B": 0.866, "C": 0.861}, n_items=1000, n_variants=10)
print(table)   # every gap is far inside the +/- 2.1 point half-widths
print(infl)    # about 1.7 points of inflation from publishing the best of 10 variants

Frequently asked questions#

Sources#

1. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, J. Steinhardt. Measuring Massive Multitask Language Understanding, 2020. arxiv.org
2. Y. Wang et al.. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark, 2024. arxiv.org
3. A. P. Gema et al.. Are We Done with MMLU?, 2024. arxiv.org
4. D. Rein et al.. GPQA: A Graduate-Level Google-Proof Q&A Benchmark, 2023. arxiv.org
5. L. Phan et al.. Humanity's Last Exam, 2025. arxiv.org
6. K. Cobbe et al.. Training Verifiers to Solve Math Word Problems, 2021. arxiv.org
7. D. Hendrycks et al.. Measuring Mathematical Problem Solving With the MATH Dataset, 2021. arxiv.org
8. E. Glazer et al.. FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI, 2024. arxiv.org
9. M. Chen et al.. Evaluating Large Language Models Trained on Code, 2021. arxiv.org
10. C. E. Jimenez et al.. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2023. arxiv.org
11. N. Jain et al.. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code, 2024. arxiv.org
12. J. Zhou et al.. Instruction-Following Evaluation for Large Language Models, 2023. arxiv.org
13. S. Lin, J. Hilton, O. Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods, 2021. arxiv.org
14. S. Yao, N. Shinn, P. Razavi, K. Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 2024. arxiv.org
15. W.-L. Chiang et al.. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, 2024. arxiv.org
16. L. Zheng et al.. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. arxiv.org
17. P. Liang et al.. Holistic Evaluation of Language Models, 2022. arxiv.org
18. A. Srivastava et al.. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, 2022. arxiv.org
19. S. Ott, A. Barbosa-Silva, K. Blagec, J. Brauner, M. Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence, 2022. arxiv.org
20. O. Sainz et al.. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark, 2023. arxiv.org
21. H. Zhang et al.. A Careful Examination of Large Language Model Performance on Grade School Arithmetic, 2024. arxiv.org
22. C. White et al.. LiveBench: A Challenging, Contamination-Limited LLM Benchmark, 2024. arxiv.org
23. S. Singh et al.. The Leaderboard Illusion, 2025. arxiv.org
24. X. Zheng, T. Pang, C. Du, Q. Liu, J. Jiang, M. Lin. Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates, 2024. arxiv.org
25. Y. Dubois, B. Galambosi, P. Liang, T. B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators, 2024. arxiv.org
26. L. Madaan et al.. Quantifying Variance in Evaluation Benchmarks, 2024. arxiv.org
27. M. Mazeika et al.. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, 2024. arxiv.org