How many test cases does an LLM eval need?

The question a release gate is really asking#

A typical release gate compares a candidate against production on a fixed suite and blocks the merge if a category's pass rate falls. Two things can go wrong. The gate can miss a real regression because the suite is too small to see it (a false negative), or it can block a harmless change because sampling noise looked like a drop (a false alarm). Both are a direct function of the number of cases per category, and both can be calculated before you write a single test.

The unit that matters is the category, not the suite. A 10,000-case suite with 40 categories has about 250 cases per category, and the gate's resolution is set by those 250. That one fact explains most noisy eval dashboards we're asked to look at.

Resolution of a single measurement#

Treat each case as a Bernoulli trial with success probability $p$. With $k$ passes out of $n$ cases, the naive estimate is $\hat{p}=k/n$ with standard error $\sqrt{\hat{p}(1-\hat{p})/n}$. Near the pass rates that matter for safety (95–99%), the normal-approximation interval behaves badly: it can exceed 100% and under-covers. The Wilson score interval [2] fixes both problems:

The interval is $\tilde{p}\pm h$, with $z=1.96$ for 95% coverage. Figure 1 plots the half-width $h$ against $n$ for four pass rates.

Read it as a resolution limit. With 200 cases at a 95% pass rate, anything you say finer than about ±3 points is noise. The width scales as $1/\sqrt{n}$, so halving it costs four times the cases. That is why the curves flatten so slowly.

Detecting a regression: power#

For a gate we care about a difference between two versions. For two independent samples of size $n$ each, with pass rates $p_1$ and $p_2=p_1-\delta$, the sample size for a one-sided test at level $\alpha$ with power $1-\beta$ is approximately

Plugging in $p_1=0.95$, $\delta =0.02$, $\alpha =0.05$, $1-\beta =0.8$ gives $n\approx 1,740$ per version. A 5-point drop needs about 340. The $1/{\delta }^2$ term is the whole story: detecting a regression half as large costs four times as many cases.

Does pairing save you?

Running both versions on the same cases allows a paired test. McNemar's test [3] uses only the discordant cases: $n_{10}$ where the old version passed and the new one failed, and $n_{01}$ the reverse. Under the null hypothesis each discordant case is equally likely to go either way:

The common advice is that pairing dramatically reduces the required sample. It reduces the variance of the difference by $2\mathrm{Cov}$, which is large when per-case outcomes are strongly correlated across versions: easy cases pass for both, hard cases fail for both. Miller [1] makes the same argument for LLM evals and recommends paired analysis and clustered standard errors. We wanted to know how big the effect is under a realistic difficulty distribution, so we simulated it.

Simulation: paired vs. unpaired, and the naive rule#

Setup. Each case $i$ has a latent baseline success probability $p_i\sim \mathrm{Beta}(3.8,0.2)$, mean $0.950$. Most cases are easy and a long tail is hard, which matches the shape we see in production safety suites. The candidate model shifts every case by the same amount on the logit scale, $\mathrm{logit}{p}_i^{\prime }=\mathrm{logit}{p}_i+s$, with $s$ solved numerically so the mean drops by exactly $\delta$. Each version is run once per case. We ran 2,000 replications per cell, with a one-sided $\alpha =0.05$ throughout.

Finding 1: pairing helped less than the folklore says. At 1,600 cases the paired test's power was 83% against 79% unpaired, a few points better. The reason is in the setup: with one sample per case and most $p_i$ close to 1, the two versions' outcomes on a case are only weakly correlated, because an easy case mostly passes for both whether or not they share anything. Pairing pays off when per-case outcomes are close to deterministic: temperature 0, or several seeds averaged per case. Even then you should still run the paired test, since it is never worse and it is the correct analysis for the design.

Finding 2: the "block on any 2-point drop" rule is a coin flip at small n. With no real change at all, the naive rule fired in 16% of runs at 200 cases and 9.3% at 400 (Figure 3). A team running ten categories at 200 cases each sees at least one spurious block on about $1-(1-0.16{)}^{10}\approx 83\%$ of merges. That is how gates get disabled. The paired test held its nominal rate (3.2%–4.7%) at every size.

Many categories: the multiple-comparisons tax#

A suite with $m$ independent categories, each gated at level $\alpha$, fires at least one false alarm with probability $1-(1-\alpha {)}^m$. With $m=20$ and $\alpha =0.05$ that is 64%. Two standard fixes:

• Bonferroni: gate each category at $\alpha /m$. Simple and conservative. It raises the required $n$ by roughly the factor ${\left(z_{1-\alpha /m}+z_{1-\beta }\right)}^2/{\left(z_{1-\alpha }+z_{1-\beta }\right)}^2$, which is about 2.2× for $m=20$ at 80% power.
• Tiering: apply strict, individually powered gates only to the handful of high-severity categories (injection, tool misuse, data egress), and track the rest as warnings with a false-discovery-rate procedure. In practice this is the better trade.

A sizing recipe you can use today#

1. For each high-severity category, decide the smallest regression $\delta$ you would block a release for, and the baseline pass rate $p_1$.
2. Compute $n$ from the formula above (or run eval_sample_size.py with your own $p_1$, $\delta$ and difficulty distribution).
3. Run both versions on the same cases and use McNemar's test. Use several seeds per case if outputs are stochastic. This is where pairing earns its keep.
4. Correct for the number of strictly gated categories.
5. Report every category with its interval, and flag categories whose interval is wider than $\delta$ as under-powered rather than "passing".

Limitations#

Frequently asked questions#

Sources#

1. E. Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640, 2024. arxiv.org
2. E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158):209–212, 1927. doi.org
3. Q. McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2):153–157, 1947. doi.org