Statistical significance for LLM evaluation: paired tests, bootstrap, multiple comparisons and power

A difference between two LLM eval scores is statistically significant when it is larger than the noise you would expect from re-running the eval on a fresh sample of questions and generations. To test it, compute per-question score differences between the two models on the same questions, put a standard error or a paired bootstrap interval on their mean, correct for the number of comparisons you made, and check beforehand that the eval has enough power to detect the gap you care about.

This page is the practical companion to our research notes on eval sample size and clustered standard errors. Those cover Wilson intervals, McNemar's test and design effects in depth; here we cover the rest of the toolkit (paired t, bootstrap and permutation tests, repeated sampling, multiple comparisons and power) and show, with seeded simulations, which choices actually change your conclusions.

A worked example: is 74.2% really better than 72.6%?#

Your new prompt scores 74.2% on a 500-question eval; the old one scored 72.6%. The temptation is to compute a 95% interval for each, about $\pm 3.8$ points, see that they overlap and declare "no difference", or ignore the intervals and ship the 1.6-point gain. Both readings are wrong in instructive ways.

Overlapping individual intervals are the wrong test, because both scores were measured on the same questions. Question difficulty is shared: a hard question tends to be missed by both versions. That shared component cancels when you look at the per-question difference, so the uncertainty of the gap is smaller than the separate error bars suggest. On the other hand, a 1.6-point gap on 500 questions is still well within the noise of most paired designs, as the rest of this page shows. The right procedure is to analyse the differences directly.

Standard errors: the unit of uncertainty#

For an eval with $n$ questions and per-question scores $s_1,\dots ,s_n$ (0 or 1 for pass/fail, or a fraction, or a rubric score), the mean score has standard error

where $\hat{\sigma }$ is the sample standard deviation of the per-question scores. For binary scores this is $\sqrt{\hat{p}(1-\hat{p})/n}$, which at 70% accuracy and 500 questions is 2.0 points. The interval $\bar{s}\pm 1.96\mathrm{SE}$ is the usual 95% interval; near 0% or 100% or with fewer than a few hundred questions, use a Wilson interval instead, as Bowyer, Aitchison and Ivanova argue and as our sample size note shows. If questions come in groups (several per passage, template or conversation), the formula above is too optimistic; use clustered standard errors.

What if the model is sampled several times per question?#

At nonzero temperature, each answer is itself a random draw. If you sample $k$ answers per question and score the average, the law of total variance splits the per-question variance into two parts:

where $p_i$ is the model's true probability of answering question $i$ correctly. The first term is how much questions differ in difficulty; the second is generation noise, which shrinks with $k$. Miller's Adding Error Bars to Evals recommends exactly this resampling to reduce the second term. It does nothing for the first: more samples per question cannot make up for too few questions.

Paired tests: compare differences, not separate scores#

Define the per-question difference $d_i=s_{B,i}-s_{A,i}$. Its mean is the score gap, and its standard error is

where ${\sigma }_A$ and ${\sigma }_B$ are the standard deviations of each model's per-question scores and $\rho$ is the correlation between them across questions. Separate error bars implicitly set $\rho =0$. Real models agree on which questions are easy, so $\rho$ is positive and the paired standard error is smaller. In practice you never need $\rho$: compute the $d_i$, take their standard deviation and divide by $\sqrt{n}$.

Several tests use the $d_i$:

• Paired t-test: $t=\bar{d}/\mathrm{SE}(\bar{d})$, compared with a t or normal critical value. Fine for a few hundred questions or more. Dietterich's classic comparison of tests for classifiers found McNemar's test well behaved and showed that t-tests on overlapping resampled splits give too many false positives.
• McNemar's test: the exact version for binary pass/fail with one sample per question; it uses only the questions where the two models disagree. Covered in our sample size note.
• Paired bootstrap: resample question indices with replacement, recompute $\bar{d}$ each time, and read off percentiles. Efron's bootstrap, applied to paired differences; Koehn popularised it for machine translation, where the metric (BLEU) is computed on the whole corpus and has no simple standard error. Use it for any metric that is not a mean of per-question scores.
• Sign-flip permutation test: under the null hypothesis that the two systems are exchangeable, each $d_i$ is as likely to be positive as negative. Flip signs at random thousands of times and see how often the shuffled mean is as extreme as the observed one. Exact without distributional assumptions; Berg-Kirkpatrick et al. compared it with the bootstrap on NLP metrics.

Simulation: which tests hold their error rate, and which have power?#

Setup (simulation). Each of 500 questions has a difficulty on the logit scale drawn from a normal distribution with standard deviation 1.5 (moderately varied) or 3.0 (very varied), centred at logit(0.7). Model A answers question $i$ correctly with probability $\text{expit}({\theta }_i)$; model B shifts every question by a constant chosen so its mean accuracy is higher by $\delta$. Each model answers each question $k$ times. We ran 1,000 replications per setting and recorded how often each two-sided 5% test declared a difference. "Separate error bars" is a two-sample z-test that treats the two score vectors as independent, which is what comparing two intervals amounts to.

Three findings. First, all three paired tests behave the same: they hold the 5% false-positive rate and have the same power, so choose by convenience, not by folklore. Second, "separate error bars" is not merely less precise; it is badly conservative. With no true gap it almost never fires, and with a real 4-point gain and five samples per question it found the gain 58% of the time against 94% for any paired test. With very varied questions it found it 20% of the time against 99.6%. The more the shared difficulty varies, the more an unpaired analysis throws away. Third, sampling several answers per question helps the paired comparison dramatically, from 11% to 41% power for a 2-point gap, because once shared difficulty cancels, generation noise is most of what remains.

Power: decide the detectable gap before you run the eval#

Power is the probability that your test detects a gap of a given size if it is real. The quantity to fix in advance is the minimum detectable effect (MDE): the smallest gap you could detect with, say, 80% power. For a paired comparison,

where ${\sigma }_d$ is the standard deviation of the per-question differences and 2.8 is 1.96 + 0.84 for a two-sided 5% test at 80% power. With binary scores and one sample per question, ${\sigma }_d$ is typically 0.4 to 0.6; in our simulation it was about 0.55. That gives an MDE of 7.9 to 11.9 points at 200 questions, 5.0 to 7.5 at 500, 3.5 to 5.3 at 1,000 and 2.5 to 3.8 at 2,000. Most of the "improvements" reported on 500-question evals are smaller than that. You can estimate ${\sigma }_d$ from a pilot run of both systems on a few hundred questions.

Multiple comparisons: twenty categories, twenty chances to be fooled#

Real eval suites report many numbers: per category, per language, per difficulty band. Each test at 5% has a 5% chance of a false alarm, and the chances add up. If $m$ independent tests are all null, the probability of at least one false "significant" result is $1-(1-\alpha {)}^m$, which is 64% for 20 tests.

Three standard corrections:

• Bonferroni: test each at $\alpha /m$. Controls the family-wise error rate (FWER), the chance of any false alarm.
• Holm: sort the p-values $p_{(1)}\le \cdots \le p_{(m)}$ and reject while $p_{(j)}\le \alpha /(m-j+1)$. Same guarantee as Bonferroni, never less powerful.
• Benjamini-Hochberg: find the largest $j$ with $p_{(j)}\le j\alpha /m$ and reject the $j$ smallest. Controls the false discovery rate (FDR), the expected share of false alarms among the results you flag, which is the right target when you will investigate every flagged category anyway.

Simulation. Twenty categories of 300 questions each, paired comparison, one sample per question. When nothing changed in any category, uncorrected testing flagged at least one category in 62.4% of runs; Bonferroni, Holm and Benjamini-Hochberg each held this near 5.7%. When three categories had real 8-point gains, uncorrected testing found 71% of them but 23% of everything it flagged was false. Bonferroni found 30%, Holm 30% and Benjamini-Hochberg 36%, with 4% of its flags false.

The power cost is the price of honesty, and the way to pay less is design, not a looser threshold: decide in advance which few categories gate a release, test those strictly, and treat the rest as monitored with FDR control. Our sample size note shows how much larger each gated category must be to keep its power after correction.

The comparisons you did not count#

Multiple comparisons also hide in the workflow. Trying eight prompt templates and reporting the best is eight tests, even if only one is written down. We simulated eight templates that were all exactly as good as the baseline, each scored on the same 300 questions: in 23.7% of runs at least one came out "significantly" different. Choosing a judge prompt, a few-shot set, a decoding temperature or an answer-extraction regex after looking at results does the same thing. The remedies are to fix choices before the final run, keep a held-out question set for the confirmatory comparison, or count the variants and correct for them.

A minimal implementation#

import numpy as np

def paired_stats(a, b, reps=10000, seed=0):
    """a, b: per-question scores of two systems on the SAME questions (same order)."""
    rng = np.random.default_rng(seed)
    d = np.asarray(b, float) - np.asarray(a, float)
    n = len(d)
    se = d.std(ddof=1) / np.sqrt(n)
    # paired bootstrap: resample questions
    boot = d[rng.integers(0, n, (reps, n))].mean(1)
    ci = np.percentile(boot, [2.5, 97.5])
    # sign-flip permutation test, two-sided
    flips = rng.choice([-1.0, 1.0], (reps, n))
    null = np.abs((flips * d).mean(1))
    p = (np.sum(null >= abs(d.mean())) + 1) / (reps + 1)
    return dict(gap=d.mean(), se=se, ci95=ci, p_perm=p)

def holm(pvals, alpha=0.05):
    p = np.asarray(pvals); order = np.argsort(p); m = len(p)
    reject = np.zeros(m, bool)
    for rank, j in enumerate(order):
        if p[j] <= alpha / (m - rank): reject[j] = True
        else: break
    return reject

Which test should I use?#

Common failure modes#

• Reporting significance without size. With 50,000 questions, a 0.3-point gap can be significant and irrelevant. Always report the gap with its interval.
• Ignoring grader noise. A model-based grader adds its own error, which biases rates and can mask or invent differences; our judge error note shows how to correct for it.
• Treating a non-significant result as "no difference". An underpowered eval cannot distinguish "no change" from "change we could not see". Report the MDE alongside.
• Peeking. Running the eval repeatedly as results come in and stopping when $p<0.05$ inflates false positives. Fix $n$ in advance or use a sequential method designed for it.
• Training-run noise. If the systems are two fine-tuning runs, question sampling is not the only randomness; a second seed of the same recipe can move scores too. Madaan et al. measure this seed variance for common benchmarks, and our page on evaluating RL agents shows how to handle it.

Card et al. found that many NLP comparisons in the literature were underpowered, and Dror et al.'s guide catalogues which tests suit which metrics. Their conclusions apply with more force to LLM evals, where test sets are often small and gains are often a point or two. For where significance fits in a full evaluation pipeline, see what AI evals are; for the metrics being compared, see LLM evaluation metrics; and for how public leaderboards handle (or ignore) these questions, see LLM benchmarks. The eval plan template has a section for writing the MDE down before you run anything.

Frequently asked questions#

Sources#

1. E. Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, 2024. arxiv.org
2. P. Koehn. Statistical Significance Tests for Machine Translation Evaluation, 2004. aclanthology.org
3. T. Berg-Kirkpatrick, D. Burkett, D. Klein. An Empirical Investigation of Statistical Significance in NLP, 2012. aclanthology.org
4. R. Dror, G. Baumer, S. Shlomov, R. Reichart. The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing, 2018. aclanthology.org
5. D. Card, P. Henderson, U. Khandelwal, R. Jia, K. Mahowald, D. Jurafsky. With Little Power Comes Great Responsibility, 2020. arxiv.org
6. L. Madaan, A. K. Singh, R. Schaeffer, A. Poulton, S. Koyejo, P. Stenetorp, S. Narang, D. Hupkes. Quantifying Variance in Evaluation Benchmarks, 2024. arxiv.org
7. S. Bowyer, L. Aitchison, D. R. Ivanova. Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints, 2025. arxiv.org
8. Q. McNemar. Note on the sampling error of the difference between correlated proportions or percentages, 1947. doi.org
9. S. Holm. A Simple Sequentially Rejective Multiple Test Procedure, 1979. jstor.org
10. Y. Benjamini, Y. Hochberg. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing, 1995. doi.org
11. T. G. Dietterich. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms, 1998. doi.org
12. B. Efron. Bootstrap Methods: Another Look at the Jackknife, 1979. doi.org