An LLM benchmark is a fixed, published collection of tasks with a scoring rule, used to compare language models on the same footing. Benchmarks are good for tracking progress and choosing a base model; they are not enough to decide whether a specific system is safe or fit for a specific use.
Kinds of benchmark
- Knowledge and reasoning: multiple-choice exams across subjects, maths word problems, science questions.
- Code: programs checked against unit tests, usually scored with pass@k.
- Agentic: multi-step tasks in a sandbox, scored on the final state or the whole trajectory.
- Safety: harmful-request sets measuring harmful compliance, and benign look-alikes measuring over-refusal.
- Preference arenas: crowd-sourced pairwise votes aggregated into ratings.
Scores have error bars
A benchmark score is a sample mean, with standard error for independent items. For accuracy 0.85 on 1,000 items, the SE is about 1.1 points and the 95% interval is roughly points. Two models 1 point apart on that benchmark are not distinguishable. Items that share a passage or template are not independent, so clustered standard errors are larger still.
How benchmarks fail
- Saturation: once top models score near the ceiling, remaining errors are often label noise and the benchmark stops separating models.
- Contamination: test items leak into training data, so scores measure recall rather than ability.
- Overfitting by selection: many teams tuning against the same public numbers inflate them without improving the underlying skill, a case of Goodhart's law.
- Prompt sensitivity: few-shot format, answer extraction and temperature can move scores by several points.
Item response theory helps diagnose saturated or mislabelled items and build smaller subsets that predict full scores.
Why it matters
Benchmarks are the shared language of model comparison, which is exactly why they get gamed. Treat a public score as a prior, then run your own eval suite built from your data, policies and failure modes. A system can top a leaderboard and still fail the cases your users will send it.