Eval regression: Definition and Examples

An eval regression is a decline in an evaluation metric from one version of a system to the next that is larger than what sampling noise alone would produce. A CI gate is the automated check that runs the eval suite on every change and blocks a merge or release when a regression, or an absolute threshold breach, is detected.

Telling a regression from noise

A score is an estimate with a standard error. For two independent runs on $n_{1}$ and $n_{2}$ items with pass rates $p^_{1}$ and $p^_{2}$ , the standard error of the difference is

SE (p^_{2} - p^_{1}) = \frac{p ^ _{1} ( 1 - p ^ _{1} )}{n _{1}} + \frac{p ^ _{2} ( 1 - p ^ _{2} )}{n _{2}}

When both versions are scored on the same items, a paired analysis is better: compute the per-item difference and its standard error, or use McNemar's test on items that flipped. Pairing removes item difficulty from the noise and usually shrinks the standard error a lot.

Worked example

Illustrative numbers: the old version passes 82% of 500 safety items and the new one 79%. Treated as independent samples, the SE of the 3-point drop is about 2.5 points, so $z \approx 1.2$ : not distinguishable from noise. Paired on the same items, if 25 items flipped from pass to fail and 10 from fail to pass, McNemar's statistic is $(25 - 10)^{2} / (25 + 10) \approx 6.4$ , which is significant at the 5% level. Same data, opposite conclusion, because the paired test uses the right noise model.

Designing CI gates

Two kinds of rule: absolute floors (for example, harmful compliance must stay below a set rate) and relative rules (no significant drop versus the last release).
Per category: a gain in one category can hide a loss in another inside an average.
Multiple comparisons: with 30 categories checked at the 5% level, one or two will trip by chance. Use a correction or require confirmation on a rerun.
Fixed seeds and pinned graders: a change to the judge prompt or model is itself a change that needs its own regression check.

Why it matters

Safety behaviour regresses quietly. A fine-tune that improves helpfulness can raise harmful compliance, and a guardrail update can raise over-refusal. Without statistically sound gates, teams either block on noise or ship real regressions, and both erode trust in the eval suite.

Sources

Evan Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, 2024. arxiv.org

Telling a regression from noise

Worked example

Designing CI gates

Why it matters

Related terms

Sources

Know the number before you ship.