Your benchmark's error bars are too narrow: clustered standard errors for LLM evals

Most benchmark scores are reported with an error bar computed as if every question were an independent draw. They rarely are. Reading-comprehension questions share passages, synthetic suites are stamped out from templates, and many harnesses sample several answers per prompt. When items arrive in correlated groups, the honest standard error can be twice the naive one or more, and a "significant" gap between two models can be nothing but which passages happened to land in the test set. This note shows how big the effect is, how to compute clustered standard errors and paired differences correctly, and how to tell from your own data whether you need them. Every number below comes from a seeded simulation we ran for this article.

A 1,000-question benchmark that is really 100 observations#

Take a reading-comprehension test built the way SQuAD was [1]: annotators read a passage and write several questions about it. Suppose the test set has 100 passages with 10 questions each, so $N=1,000$ items, and a model answers 70% of them correctly. The textbook binomial standard error is $\sqrt{0.7\times 0.3/1000}=1.45$ percentage points, so the 95% interval is about $\pm 2.8$ points.

Now look at how a model actually fails on such a test. If it misreads a dense legal passage, it tends to miss several of that passage's questions. If a passage is about a topic heavily represented in pretraining, it tends to get most of them. Questions inside a passage are positively correlated. A second sample of 100 passages would give a noticeably different score, more different than a fresh draw of 1,000 independent questions would. The binomial formula answers the wrong question: it measures the noise from re-drawing questions while holding the passages fixed, but the passages are also a sample.

The same structure shows up across modern evaluation:

• Shared context. Passages, documents, long-context haystacks, code repositories with many tests, conversations with several turns graded separately.
• Shared templates. Synthetic math or reasoning items generated by filling slots in a few hundred templates; red-team prompts produced by mutating a smaller set of seed attacks.
• Repeated sampling. Running each prompt $k$ times at nonzero temperature and averaging. The prompt is the cluster and the $k$ samples are far from independent.
• Shared annotators or sources. All items written by the same contractor, scraped from the same site or translated by the same pipeline.

The design effect#

Survey statisticians solved this problem for household surveys, where interviewing ten people on the same street gives less information than ten people from ten streets. Kish named the variance inflation the design effect [2]. Write $x_{ki}$ for the 0/1 score on question $i$ in cluster $k$, with $K$ clusters of equal size $m$, so $N=Km$. Let ${\sigma }^2$ be the item variance and $\rho$ the intra-cluster correlation (ICC): the correlation between the scores of two different questions in the same cluster. Then the variance of the benchmark mean is

Here ${\sigma }^2=p(1-p)$ for a 0/1 score with accuracy $p$, $N$ is the number of questions, $m$ the questions per cluster, and $\rho$ the ICC. The derivation is short: the variance of a cluster total is $m{\sigma }^2$ from the diagonal plus $m(m-1)\rho {\sigma }^2$ from the off-diagonal covariances, and the mean averages $K$ independent cluster totals. Intuitively, each extra question in a cluster adds only a fraction $1-\rho$ of new information, because a share $\rho$ of its outcome was already predictable from its siblings.

A useful restatement is the effective sample size

the number of independent questions that would give the same precision. When $\rho =0$ you get $N$ back. When $\rho =1$, every question in a cluster is a copy of the others and $N_{\text{eff}}=K$: you have exactly as many observations as clusters.

Worked example#

With $p=0.70$, $K=100$, $m=10$ and $\rho =0.2$, the design effect is $1+9\times 0.2=2.8$. The effective sample size is $1000/2.8\approx 357$ questions. The standard error grows by $\sqrt{2.8}\approx 1.67$, from 1.45 to 2.42 points, and the 95% half-width grows from $\pm 2.8$ to $\pm 4.8$ points. An ICC of 0.2 is not extreme for items that share a passage or a template; it means that knowing the model got one question in a passage right shifts your belief about a sibling question noticeably, nothing more.

Two features of the formula matter in practice. First, the damage scales with $(m-1)\rho$, so a tiny ICC can still dominate when clusters are large: $\rho =0.02$ with 50 items per cluster gives a design effect of about 2. Second, clusters of unequal size make things worse. A standard approximation is $\mathrm{DEFF}\approx 1+[(1+{\mathrm{CV}}^2)\bar{m}-1]\rho$, where $\bar{m}$ is the mean cluster size and $\mathrm{CV}$ the coefficient of variation of cluster sizes, so a handful of very large clusters can carry the whole variance.

How wrong are naive intervals? A simulation#

Setup (simulation). We generated benchmarks of $N=1,000$ binary items with true accuracy 70%, split into clusters of size $m\in \{1,5,10,20,50\}$. Each cluster has its own accuracy drawn from a beta distribution with mean 0.7 and ICC $\rho$ (the beta-binomial model, where $\rho =1/(a+b+1)$), and items are Bernoulli draws from their cluster's accuracy. For each of 4,000 replications per setting we built two 95% intervals: the naive binomial interval and a clustered interval (defined in the next section) with a $t_{K-1}$ critical value. Coverage is the share of intervals containing 70%.

At $m=10$ and $\rho =0.1$, the naive "95%" interval covers the truth 83.4% of the time. At $\rho =0.2$ it covers 76.0%. With 50 items per cluster and $\rho =0.2$, coverage is 45.0%: the advertised 95% interval is wrong more often than it is right. The simulated coverage tracks the analytic prediction $2\Phi (1.96/\sqrt{\mathrm{DEFF}})-1$ closely (44.9% predicted for that cell), which is what the interactive figure below uses. The clustered interval stays between 94.1% and 95.5% in every cell we ran. Clustered standard errors do not shrink the uncertainty; they report the uncertainty that was always there.

Computing clustered standard errors#

You do not need to know $\rho$ to correct the interval. The clustered sandwich variance estimator of Liang and Zeger [3] sums residuals within each cluster before squaring, so any within-cluster correlation is absorbed automatically:

Here $K$ is the number of clusters, $N$ the number of items, and the inner sum is cluster $k$'s total residual. The factor $K/(K-1)$ is a small-sample correction. If items within a cluster are independent, the cross terms average to zero and the estimator agrees with the naive one; if they are positively correlated, the cluster totals are more spread out and the variance grows to match. For a plain mean this is the same as treating each cluster's mean as one observation, weighted by its size.

The design effect is still worth estimating, because it tells you how much a new benchmark will cost. The one-way ANOVA estimator of the ICC is

where MSB is the mean square between cluster means (scaled by $m$) and MSW is the mean square of items around their own cluster mean. It compares how much cluster averages differ against how much they would differ if items were shuffled at random.

When you have few clusters#

The clustered estimator is a large-$K$ approximation. Its effective degrees of freedom are closer to the number of clusters than to the number of items, a point Cameron and Miller stress in their practitioner's guide [4]. We reran the simulation with $m=10$, $\rho =0.1$, and between 5 and 100 clusters, comparing the naive interval, the clustered interval with a normal critical value, the same with a $t_{K-1}$ critical value, and a cluster bootstrap that resamples whole clusters (400 resamples, 1,000 replications; the other columns use 4,000).

With 5 clusters, even the clustered interval with a normal critical value covers only 84.8%, and the percentile cluster bootstrap does no better. Switching to $t_{K-1}$ recovers most of the gap (92.9%). From about 20 clusters the corrected intervals are close to nominal. With fewer than about 20 clusters, use $t_{K-1}$ critical values at minimum, and consider the wild cluster bootstrap of Cameron, Gelbach and Miller [5], which was designed for exactly this regime. Note that the naive interval does not approach 95% as $K$ grows: its coverage is set by the design effect, not by the sample size.

Comparing two models: pair first, then cluster#

Leaderboards are about differences, and the difference between two models has its own structure. Both models answer the same questions, so the natural statistic is the per-question difference $d_i=a_i-b_i$, where $a_i$ and $b_i$ are the two models' 0/1 scores on question $i$. Its variance is

where ${\sigma }_a^2$ and ${\sigma }_b^2$ are the item-level variances of each model's score, ${\sigma }_{ab}$ is their covariance across questions, and ${\rho }_d$ is the ICC of the differences. Two forces act in opposite directions. Pairing removes shared difficulty: hard questions are hard for both models, so ${\sigma }_{ab}>0$ and the difference is far less noisy than two independent accuracies. Clustering of the difference comes from model-by-cluster interaction: model A happens to be better at legal passages and B at sports, so their gap varies by cluster. That interaction is exactly what changes if you redraw the passages.

Setup (simulation). Two identical models (same expected accuracy, about 71%) answer $K=100$ passages of $m=10$ questions. A question is answered correctly when a latent score exceeds zero; the score is the sum of a shared passage difficulty (s.d. 0.8), a shared question difficulty (s.d. 0.8), a model-specific passage effect with s.d. ${\sigma }_t$, and independent noise. We vary ${\sigma }_t$ and, over 4,000 replications each, count how often three two-sided 5% tests declare a winner: the naive unpaired two-proportion test, a naive paired test on $d_i$, and a paired test with clustered standard errors.

Three things stand out. When there is no model-by-passage interaction (${\sigma }_t=0$), the naive unpaired test is badly conservative: it rejects only 1.4% of the time, because it ignores the shared difficulty that pairing would cancel. That same test then becomes anti-conservative once models differ by passage: at ${\sigma }_t=0.8$, where ${\rho }_d\approx 0.20$, it declares a winner 17.8% of the time. The two errors partly cancel, which is why the unpaired naive test can look fine on one benchmark and fail on another. The naive paired test gets the pairing right but ignores clustering, and its false-win rate climbs from 4.8% to 24.5%, matching the analytic $2(1-\Phi (1.96/\sqrt{{\mathrm{DEFF}}_d}))=24.0\%$. The paired clustered test stays between 4.9% and 5.5% throughout.

Note the modest ICCs involved. A difference ICC of 0.08 already turns a 5% test into a 14% test at $m=10$. Within a single model, scores in the same simulation had ICCs of 0.17 to 0.27, mostly from shared difficulty, which pairing removes; what is left for the difference is the smaller interaction term.

Pairing buys power back#

Clustering widens intervals, but pairing narrows them, and the net is usually favourable. In the same simulation with ${\sigma }_t=0.3$, we gave model A a real advantage and measured one-sided power at the 2.5% level for a clustered unpaired comparison (each model's clustered standard error, combined as if independent) and the paired clustered test.

For a true gap of about 2.2 points, the paired clustered test detects it 22% of the time against 1.5% for the unpaired clustered comparison, because the standard error of the gap falls from 3.3 to 1.9 points. The lesson generalises: never compare two models with separate per-model error bars. Compute the per-question difference, then cluster it. Card and colleagues show that many NLP comparisons are underpowered even before clustering is considered [6], so it is worth knowing which design gives you power back.

Explore the design effect#

The figure below uses the article's formulas directly. The top panel shows the 95% interval for one model at 70% accuracy, naive against clustered. The middle panel draws 40 comparisons of two identical models whose true sampling spread includes the design effect, each with the naive paired interval; red rows are the comparisons a naive test would report as a win. The bottom panel is the analytic false-win rate $2(1-\Phi (1.96/\sqrt{\mathrm{DEFF}}))$ as a function of $\rho$ for the chosen cluster size. For simplicity a single $\rho$ drives both the score and the difference.

Try raising $N$ with $\rho$ and $m$ fixed. Both intervals narrow, but the ratio between them and the false-win rate do not move. Scaling up a benchmark makes the honest interval smaller; it never makes the naive interval honest. Only computing the variance at the cluster level does that, and only reducing $m$ or $\rho$ lowers the price you pay per question.

What to do in practice#

A checklist for benchmark reports#

1. Write down the sampling unit. Ask what would change if someone rebuilt the benchmark: new passages, new templates, new seed prompts, new repositories. That unit is your cluster. Abadie, Athey, Imbens and Wooldridge argue that clustering should follow the sampling or assignment design rather than whatever grouping is available [7]; for a benchmark, the design is how items were generated.
2. Keep the cluster id in the results file. Most harnesses drop it. Without it, nobody can compute the right interval later.
3. Report the ICC and design effect alongside accuracy, so readers can see how much the benchmark is worth in independent items.
4. Compare models on paired differences with clustered standard errors. Use $t_{K-1}$ critical values, and the wild cluster bootstrap if $K$ is below about 20.
5. Treat repeated samples as a cluster level. If you sample $k$ answers per prompt, cluster at the prompt (or at the passage, if prompts are nested in passages; cluster at the coarsest level that was sampled).
6. Plan benchmark size in clusters, not items. When $\rho$ is sizeable, writing a 20th question for a passage buys almost nothing; a new passage with five questions buys a lot.

Code#

The whole correction is a few lines once the cluster id is kept with each score:

import numpy as np
from scipy import stats

def paired_clustered_diff(a, b, cluster):
    """a, b: 0/1 scores of two models on the same items; cluster: id per item."""
    d = np.asarray(a, float) - np.asarray(b, float)
    ids, inv = np.unique(cluster, return_inverse=True)
    K, N = len(ids), len(d)
    resid_totals = np.bincount(inv, weights=d - d.mean())
    se = np.sqrt(K / (K - 1) * (resid_totals ** 2).sum() / N ** 2)
    t = stats.t.ppf(0.975, K - 1)
    return d.mean(), (d.mean() - t * se, d.mean() + t * se)

The same function with $b=0$ gives the clustered interval for a single model. Regression packages that offer clustered sandwich covariance give identical answers for an intercept-only model.

Failure modes#

• Clustering at the wrong level. Clustering by subject on a benchmark whose items were written independently inflates intervals for no reason, and with only a few dozen subjects the estimate is noisy. Clustering by question when passages were the sampled unit under-corrects. Use the generating process, not the folder structure.
• Too few clusters. A benchmark built from 8 templates has, at most, 8 observations' worth of between-template information. No estimator rescues that; report it honestly and add templates.
• Averaging away the structure. Reporting per-category accuracies and then averaging them with a naive interval reintroduces the problem one level up.
• Ignoring other variance sources. Clustering handles item sampling. Seed and sampling noise in the model itself is a separate component; Madaan and colleagues measured seed variance across many models and benchmarks and recommend factoring it in before comparing models [8]. Report both when you can.
• Multiple comparisons. A leaderboard with 20 models has 190 pairs. Correct intervals per pair still produce chance wins somewhere in the table; Dror and colleagues review the testing choices for NLP [9].

Frequently asked questions#

Sources#

1. P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text, 2016. arxiv.org
2. L. Kish. Survey Sampling, 1965. archive.org
3. K.-Y. Liang, S. L. Zeger. Longitudinal data analysis using generalized linear models, 1986. doi.org
4. A. C. Cameron, D. L. Miller. A Practitioner's Guide to Cluster-Robust Inference, 2015. doi.org
5. A. C. Cameron, J. B. Gelbach, D. L. Miller. Bootstrap-Based Improvements for Inference with Clustered Errors, 2008. doi.org
6. D. Card, P. Henderson, U. Khandelwal, R. Jia, K. Mahowald, D. Jurafsky. With Little Power Comes Great Responsibility, 2020. arxiv.org
7. A. Abadie, S. Athey, G. Imbens, J. Wooldridge. When Should You Adjust Standard Errors for Clustering?, 2017. arxiv.org
8. L. Madaan, A. K. Singh, R. Schaeffer, A. Poulton, S. Koyejo, P. Stenetorp, S. Narang, D. Hupkes. Quantifying Variance in Evaluation Benchmarks, 2024. arxiv.org
9. R. Dror, G. Baumer, S. Shlomov, R. Reichart. The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing, 2018. doi.org