LLM evaluation metrics: accuracy, F1, exact match, pass@k, ECE, win rate, BLEU and ROUGE

LLM evaluation metrics are the functions that turn a model's outputs into numbers: accuracy and F1 for classification-style answers, exact match and token F1 for short answers, pass@k for code, calibration error for confidence, win rate for pairwise comparisons, and overlap scores such as BLEU and ROUGE for text with a reference. The right metric is the one whose number moves when the thing you care about moves, and stays put when it does not.

This guide defines each common metric, shows on real strings and seeded simulations where each one misleads, and ends with a table of when to use which. The metric playground lets you watch accuracy, F1 and calibration error respond to class balance, threshold and overconfidence.

How do you choose an LLM evaluation metric?#

Three questions narrow the choice quickly.

1. What counts as a correct output? If there is one right answer (a label, a number, a function that passes tests), use a correctness metric. If many outputs are acceptable (a summary, an explanation, an email), you need a reference-free judgment, a rubric or a pairwise preference.
2. Is there a trustworthy reference? Overlap metrics compare against references and are only as good as them. Execution-based and rule-based checks need no reference text at all.
3. What decision will the number drive? A release gate needs a metric with a clear threshold and a known error bar. A model selection decision needs a metric that ranks systems the way users would.

The rest of this page goes metric by metric. For how metrics fit into a full pipeline, see what AI evals are; for how to tell whether a difference in any of them is real, see statistical significance for evals.

Accuracy, precision, recall and F1#

When an eval reduces to a yes/no decision (is this output unsafe? did the model flag the fraudulent transaction? is the extracted field present?), count true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN):

Precision asks "when the system says yes, how often is it right?"; recall asks "of the real yeses, how many did it catch?". F1 is their harmonic mean, which is low if either is low.

Accuracy is fine when classes are balanced and errors cost the same. Neither is true for most safety evals. Simulation: we scored 10,000 items with a fixed classifier whose scores for positives sit 1.5 standard deviations above those for negatives, at a fixed threshold. At 50% prevalence it had 76.5% accuracy and 0.75 F1. At 2% prevalence, the same classifier had 83.8% accuracy and an F1 of 0.145, with precision of 8.2%. A system that always answers "safe" would score 97.9% accuracy on that data and catch nothing. Accuracy rose as the classifier became less useful.

Two practical rules follow. Report precision and recall (or the full confusion matrix) for any imbalanced eval, and pick the threshold from the costs of each error type, not by default at 0.5. Threshold-free summaries such as the area under the ROC or precision-recall curve help compare classifiers, but a deployed system runs at one threshold, so report the operating point too. Our note on guardrail thresholds shows how to choose it.

Exact match and token F1 for short answers#

For questions with short factual answers, exact match (EM) scores 1 if the normalised prediction equals the normalised reference and 0 otherwise. SQuAD's normalisation lowercases, strips punctuation and removes the articles "a", "an" and "the". Token F1 treats prediction and reference as bags of words and computes F1 over shared tokens, giving partial credit.

We ran both on a few real strings. Against the reference "Paris", the outputs "Paris" and "paris." both score EM 1. "The answer is Paris" scores EM 0 and F1 0.50; "Paris, France" scores EM 0 and F1 0.67. Against "the Treaty of Paris in 1783", the reordered "1783 Treaty of Paris" gets F1 0.89, but so nearly does the wrong "the Treaty of Paris in 1763" at 0.80. EM punishes harmless verbosity; token F1 rewards near-misses that change the one token that matters.

For chat models that answer in sentences, the fix is usually in the harness, not the metric: ask for a constrained final answer format, extract it with a rule, and apply EM to the extraction. Then track the extraction failure rate separately, because a model that ignores the format will look wrong when it is merely untidy.

pass@k for code and verifiable tasks#

For code, correctness can be checked by running unit tests. pass@k is the probability that at least one of $k$ sampled solutions passes. Estimating it by drawing exactly $k$ samples is noisy, so Chen et al. introduced an unbiased estimator: draw $n\ge k$ samples per problem, count the $c$ that pass, and compute

where $\left(\genfrac{}{}{0ex}{}{n-c}{k}\right)/\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is the probability that a random subset of $k$ of the $n$ samples contains no passing one. The tempting shortcut, plugging the per-problem pass rate $\hat{p}=c/n$ into $1-(1-\hat{p}{)}^k$, is biased downward. Simulation: 164 problems (HumanEval's size) with per-problem success probabilities drawn from a Beta(0.5, 1) distribution, true pass@10 of 71.4%. With $n=10$ samples per problem, the plug-in estimate averaged 65.1%, with $n=20$ 68.3% and with $n=100$ 70.8%. The unbiased estimator averaged 71.4% at every $n$.

pass@k measures capability under search: if you can sample many times and verify, how often does a correct answer exist? It says little about reliability. For that, the complementary metric pass^k asks whether all $k$ attempts succeed; our research note on pass@k vs pass^k shows how far the two diverge for agents.

Calibration and expected calibration error (ECE)#

A model is calibrated if, among answers it gives with 80% confidence, 80% are correct. Calibration matters whenever confidence drives an action: abstaining, escalating to a human, or showing a warning. Expected calibration error bins predictions by confidence and averages the gap between accuracy and confidence in each bin:

where $B_m$ is the set of predictions in confidence bin $m$, $\mathrm{acc}(B_m)$ their accuracy and $\mathrm{conf}(B_m)$ their mean confidence. Naeini, Cooper and Hauskrecht introduced it, and Guo et al. popularised it for neural networks; Kadavath et al. used it to show that large language models can be well calibrated on multiple-choice questions when asked in the right format.

ECE has a bias that most reports ignore: with finite data, even a perfectly calibrated model scores above zero, because each bin's accuracy is a noisy estimate. Simulation: items with latent logits drawn from a normal distribution (mean 1, standard deviation 1.5), true probability of being correct equal to the logistic of the logit, 10 equal-width bins. A perfectly calibrated model measured an average ECE of 7.1 points on 100 items, 4.1 on 300, 2.2 on 1,000 and 1.0 on 5,000. An overconfident model (logits multiplied by 1.5) measured 9.1 on 100 items and 6.0 on 5,000.

So an ECE of 5 on a 200-item eval does not tell you whether the model is miscalibrated. Report ECE with the number of items and bins, show a reliability diagram, and consider the Brier score (mean squared difference between confidence and correctness), which is a proper scoring rule with no binning choices. Nixon et al. and Kumar, Liang and Ma discuss better estimators. Our research on calibration and selective prediction shows what calibration buys when a system can abstain.

Metric playground#

Win rate and pairwise preferences#

For open-ended outputs, it is often easier to say which of two answers is better than to score either one. The win rate of model B against a baseline A is

where $W$ is the number of prompts on which B's answer was preferred, $T$ the number of ties and $N$ the number of prompts. Preferences can come from humans or from a model judge. Across many models, pairwise outcomes are fitted with a Bradley-Terry or Elo model to produce a leaderboard rating, as public arenas do; see LLM benchmarks for how to read those.

Win rates inherit every bias of the judge. Model judges in particular tend to prefer longer answers and the answer shown first, as Zheng et al. documented when introducing MT-Bench. Simulation: two models of equal quality on 500 prompts, where B writes the longer answer on 70% of them, and a judge that shifts its preference towards the longer answer by 10 percentage points. B's expected win rate became 54.0%; with a 20-point length preference, 58.0%. The sampling standard deviation at 500 prompts was 2.2 points, so a 4-point bias is already "significant" and entirely spurious. Dubois et al. introduced length-controlled win rates for AlpacaEval to correct for exactly this, and Zheng, Pang and colleagues showed that even a null model returning one constant response could reach an 86.5% length-controlled win rate by exploiting the judge. Our research on position and verbosity bias measures these effects, and the guide to LLM-as-a-judge covers mitigation: swap positions, control length, and calibrate against human labels.

BLEU, ROUGE and their limits#

BLEU was designed for machine translation. It measures what fraction of the candidate's n-grams (sequences of 1 to 4 words) appear in the reference, combines those precisions by a geometric mean, and multiplies by a brevity penalty so that very short outputs cannot score well:

where $p_n$ is the modified n-gram precision, $w_n=1/4$, $r$ the reference length and $c$ the candidate length. ROUGE was designed for summarisation and is recall-oriented; ROUGE-L scores the longest common subsequence of words between candidate and reference.

Both reward surface overlap, and meaning lives elsewhere. We computed sentence BLEU (with add-one smoothing for higher-order n-grams), ROUGE-L and token F1 for four candidates against the reference "the patient should take the medication twice a day with food":

The negated instruction, which is dangerous, gets the highest ROUGE-L of the four. The correct paraphrase gets the lowest score on every metric. Callison-Burch et al. showed in 2006 that BLEU improvements need not correspond to human judgments, and Novikova et al. found that word-overlap metrics correlate weakly with human ratings for generation tasks. Embedding-based scores such as BERTScore credit paraphrases better but still struggle with negation and numbers.

Use BLEU and ROUGE for what they are good at: tracking regressions in a fixed pipeline over a fixed reference set, at corpus level, where many sentences average out the noise. Do not use them as the primary quality signal for open-ended LLM output, and never for safety-relevant correctness.

Rubric scores, task success and safety metrics#

Much LLM evaluation now uses a grader, often a model, that applies a written rubric and returns a label or score per output. The metric is then a mean of rubric scores, and its quality depends on the grader's agreement with experts; our notes on rubric evals and judge error show how to measure and correct for that. For agents, the natural metric is task success checked against the final environment state, with process metrics (steps, tool errors, policy violations) alongside; see agent trajectory evals. Safety evals add rates with their own subtleties: harmful-compliance rate, over-refusal rate on benign prompts (which must be reported together, as our note on over-refusal shows), and attack success rate, which is meaningless without the attack budget, as the comparison of red teaming and evals explains.

Metric choice can create the effect you are measuring#

Discontinuous metrics turn smooth improvement into apparent jumps. Schaeffer, Miranda and Koyejo showed that several "emergent abilities" of language models, which appeared suddenly at some scale under exact match, became smooth and predictable under continuous metrics such as token edit distance or per-token probability. If exact match requires 10 tokens all correct and per-token accuracy rises smoothly from 80% to 95%, exact match rises from 11% to 60%, slowly and then suddenly. Neither number is wrong, but before concluding that a capability "appeared", check it with a continuous metric.

Which metric should you use?#

Reference implementations#

import re, string
import numpy as np
from collections import Counter

def normalize(s):
    s = s.lower()
    s = "".join(ch for ch in s if ch not in set(string.punctuation))
    s = re.sub(r"\b(a|an|the)\b", " ", s)
    return " ".join(s.split())

def exact_match(pred, ref):
    return float(normalize(pred) == normalize(ref))

def token_f1(pred, ref):
    p, r = normalize(pred).split(), normalize(ref).split()
    common = sum((Counter(p) & Counter(r)).values())
    if common == 0:
        return 0.0
    prec, rec = common / len(p), common / len(r)
    return 2 * prec * rec / (prec + rec)

def pass_at_k(n, c, k):
    """Unbiased pass@k for one problem: n samples, c correct (Chen et al. 2021)."""
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

def ece(conf, correct, bins=10):
    conf, correct = np.asarray(conf), np.asarray(correct, float)
    b = np.minimum((conf * bins).astype(int), bins - 1)
    return sum((b == m).mean() * abs(correct[b == m].mean() - conf[b == m].mean())
               for m in range(bins) if (b == m).any())

print(exact_match("paris.", "Paris"), token_f1("Paris, France", "Paris"))  # 1.0 0.666...

Failure modes to check before trusting a metric#

• The metric rewards the wrong thing. Run it on a handful of deliberately bad outputs (negations, wrong numbers, verbose correct answers). If bad outputs score well, change the metric.
• The harness, not the model, sets the score. Answer extraction, prompt format and stop sequences can move accuracy by several points; MMLU-Pro's authors measured 4 to 5 points of prompt sensitivity on MMLU. Fix and report them.
• A single number hides a trade-off. Pair every safety metric with a helpfulness metric and every precision with a recall.
• No error bar. Every metric above is an estimate from a finite sample. Put an interval on it, and on differences between systems.
• Optimising the metric. Once a metric becomes a training or selection target, systems learn its blind spots. That is specification gaming, and it applies to eval metrics as much as to rewards.

Frequently asked questions#

Sources#

1. P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text, 2016. arxiv.org
2. M. Chen et al.. Evaluating Large Language Models Trained on Code, 2021. arxiv.org
3. K. Papineni, S. Roukos, T. Ward, W.-J. Zhu. BLEU: a Method for Automatic Evaluation of Machine Translation, 2002. aclanthology.org
4. C.-Y. Lin. ROUGE: A Package for Automatic Evaluation of Summaries, 2004. aclanthology.org
5. C. Callison-Burch, M. Osborne, P. Koehn. Re-evaluating the Role of Bleu in Machine Translation Research, 2006. aclanthology.org
6. J. Novikova, O. Dušek, A. Cercas Curry, V. Rieser. Why We Need New Evaluation Metrics for NLG, 2017. arxiv.org
7. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi. BERTScore: Evaluating Text Generation with BERT, 2019. arxiv.org
8. C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger. On Calibration of Modern Neural Networks, 2017. arxiv.org
9. M. P. Naeini, G. F. Cooper, M. Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning, 2015. doi.org
10. J. Nixon, M. Dusenberry, G. Jerfel, T. Nguyen, J. Liu, L. Zhang, D. Tran. Measuring Calibration in Deep Learning, 2019. arxiv.org
11. A. Kumar, P. Liang, T. Ma. Verified Uncertainty Calibration, 2019. arxiv.org
12. S. Kadavath et al.. Language Models (Mostly) Know What They Know, 2022. arxiv.org
13. L. Zheng et al.. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. arxiv.org
14. Y. Dubois, B. Galambosi, P. Liang, T. B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators, 2024. arxiv.org
15. X. Zheng, T. Pang, C. Du, Q. Liu, J. Jiang, M. Lin. Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates, 2024. arxiv.org
16. R. Schaeffer, B. Miranda, S. Koyejo. Are Emergent Abilities of Large Language Models a Mirage?, 2023. arxiv.org
17. Y. Wang et al.. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark, 2024. arxiv.org