Item response theory for LLM benchmarks: measuring more with fewer questions

A 1,000-question benchmark does not contain 1,000 equally useful questions. Some are so easy every model gets them, some are so hard no model does, and a few are broken. Item response theory (IRT) gives each question a difficulty and a sharpness, tells you how much each one actually says about a model, and lets you estimate a full-benchmark score from 50 well-chosen items. In our seeded simulation, 50 items picked adaptively estimated the 1,000-item score with an error of 2.9 points, against 6.6 points for 50 random items. This note covers the models, the arithmetic, the simulation and the ways it goes wrong.

A concrete example: two questions, two very different signals#

Suppose you are comparing checkpoints on a multiple-choice benchmark and one evaluation run costs real money, because each item needs a long chain-of-thought generation or a judge call. Consider two items. Item A is answered correctly by about half of the models you have ever run, and the strong models almost always get it while the weak ones almost always miss it. Item B is also answered correctly by about half of the models, but whether a given model gets it right looks like a coin flip: strong and weak models succeed at nearly the same rate, perhaps because the reference answer is wrong or the question is ambiguous.

Plain accuracy treats A and B identically. Each contributes one thousandth of the score. Yet A tells you a great deal about which side of "average" a model sits on, and B tells you almost nothing. IRT makes that difference measurable. It models the probability that a model answers an item correctly as a function of a latent ability of the model and a few parameters of the item. Once the parameters are estimated from a set of previously evaluated models, three things become possible: you can see which items carry information, you can pick a small subset that preserves most of it, and you can estimate a new model's ability (and therefore its expected full-benchmark score) from that subset, with an honest standard error.

This is not a new idea. Psychometrics has used it for standardised tests for decades [1, 2], and it has been applied to NLP test sets for some time [3, 4, 5]. Since 2023 several groups have used it to shrink LLM benchmarks: tinyBenchmarks reported that 100 curated examples suffice to estimate MMLU performance accurately [6], anchor-point selection ranked models with 1 to 30 points [7], and metabench distilled six benchmarks to under 3% of their size [8]. What follows is the machinery underneath, and a simulation that shows where the savings come from and where they stop.

The 1PL, 2PL and 3PL models#

Write ${\theta }_j$ for the ability of model $j$ and $y_{ij}\in \{0,1\}$ for whether model $j$ answered item $i$ correctly. Every IRT model is a choice of item response function $P_i(\theta )=\mathrm{Pr}(y_{ij}=1\mid {\theta }_j=\theta )$, and responses are assumed independent given ability.

One parameter: the Rasch model#

Here $b_i$ is the difficulty of item $i$, measured on the same scale as ability, and $\sigma$ is the logistic function. A model whose ability equals the item's difficulty has a 50% chance of answering correctly. Every item has the same slope, so items differ only in where along the ability axis they switch from "usually wrong" to "usually right". Under this model the raw number of correct answers is a sufficient statistic for ability, which is why it is the one IRT model in which plain accuracy loses nothing.

Two parameters: adding discrimination#

The new parameter $a_i>0$ is the discrimination: the steepness of the curve at $\theta =b_i$. Item A from the example has a large $a_i$; item B has $a_i$ near zero, so its curve is almost flat and the probability of success barely depends on ability. Once items differ in discrimination, the number correct is no longer sufficient: a correct answer on a sharp item says more than a correct answer on a flat one, and the ability estimate weights items accordingly.

Three parameters: guessing#

The lower asymptote $c_i$ is the probability that a model of very low ability still gets the item right, for example by picking one of four options at random ($c_i\approx 0.25$). The form is due to Birnbaum and is treated in the standard texts [1, 2]. For LLM benchmarks it matters most for multiple-choice items scored by option likelihood, where even a tiny model has a floor well above zero. It is also the hardest model to fit: $c_i$ is only identified by the behaviour of weak models, and if your calibration set contains few of them the estimates drift.

Fisher information: what one item tells you#

The quantity that makes item selection possible is the Fisher information an item carries about ability. For the 2PL model it is

where $P_i(\theta )$ is the item's probability of a correct answer at ability $\theta$. Intuitively, an item is informative when its outcome is uncertain ($P$ near one half, so $P(1-P)$ is large) and when that outcome changes quickly with ability ($a_i$ large). An item that every model of ability $\theta$ passes tells you nothing about models near $\theta$. For the 3PL model the expression becomes

with $c_i$ the guessing floor as before. Guessing always reduces information, because a correct answer might be luck. Information adds across items that are conditionally independent, giving the test information $I(\theta )={\sum }_i{I}_i(\theta )$, and the standard error of the maximum-likelihood ability estimate is approximately

In words: the precision you get about a model is the sum of the information of the items you showed it, evaluated at that model's ability. This is the whole argument for small, well-chosen subsets. Precision is not set by how many items you run. It is set by how much information those items carry at the ability you are measuring.

A worked example#

Take a 2PL item with $a=1.5$ and $b=0.5$, and a model with $\theta =1$. Then $a(\theta -b)=0.75$, so $P=\sigma (0.75)=0.679$ and $I=1.5^2\times 0.679\times 0.321=0.49$. If the same item were a four-option question with $c=0.25$, then $P=0.25+0.75\times 0.679=0.759$ and the 3PL formula gives $I=0.33$: a third of the information is lost to the possibility of guessing.

Now compare a broken item with $a=0.1$. Its information can never exceed $0.1^2/4=0.0025$, while the sharp item's peak is $1.5^2/4=0.5625$, 225 times more. Fifty sharp items placed near a model's ability give $I\approx 50\times 0.49\approx 25$ and a standard error of about $1/\sqrt{25}=0.2$ ability units. Getting the same precision from broken items would take thousands of them. That ratio is why random subsets waste most of their budget.

Simulation: how much a small subset can recover#

Setup (simulation, seeded). We generated a synthetic benchmark of 1,000 binary items from a 2PL model. Discriminations were log-normal with median 1, difficulties normal with standard deviation 1.2, and 8.6% of items were made "broken" with discrimination between 0.02 and 0.15, standing in for mislabelled or ambiguous questions. We drew 60 calibration models with abilities from a standard normal, simulated their full response matrix, and fitted a 2PL model to it by joint maximum likelihood with weak priors on the parameters (alternating Newton steps). We then drew 200 new test models, simulated their responses on all 1,000 items, and asked each method to estimate each new model's observed 1,000-item accuracy from a subset of 25, 50 or 100 items. Random subsets were redrawn 20 times. Ability was estimated by the posterior mean under a standard normal prior (EAP), and the predicted full score is the average of $P_i(\hat{\theta })$ over all items. The methods:

• Random, raw accuracy: accuracy on a uniform random subset, the usual practice.
• Random + IRT: the same random subset, scored through the fitted model, so that which items happened to be drawn is taken into account.
• Fixed informative: the items with the highest expected information under the ability distribution of the calibration models, the same subset for every new model.
• Stratified informative: difficulty split into equal-count bins, one most discriminating item per bin.
• Adaptive: computerised adaptive testing; each next item is the one with the most information at the current ability estimate.

Calibration quality. With only 60 calibration models, the fitted parameters are noisy: the correlation with the true values was 0.91 for difficulty and 0.78 for discrimination. Abilities of the calibration models themselves were recovered almost perfectly (correlation 0.995), because each sees 1,000 items. Discrimination is the hardest parameter to pin down, and that matters, because discrimination is exactly what item selection exploits. Still, the fit flagged 71% of the broken items as being in the bottom decile of estimated discrimination, which by itself is a useful audit of a benchmark.

Read across the first row and the scale of the problem is obvious: 50 random items miss the full score by 6.6 points in RMSE, with a 95th-percentile miss of 12.7 points. Simply scoring the same random items through the IRT model cuts that to 5.1 points, because the model knows that a subset which happened to draw easy items should be discounted. Choosing the items helps far more: the fixed informative subset of 50 reaches 3.3 points and the adaptive procedure 2.9. To match the adaptive 50-item error, random sampling with raw accuracy needs more than 100 items; at 100 random items it is still at 4.4.

Informative for whom?#

The fixed informative subset has a blind spot, and Figure 3 shows it. Selecting the items with the highest average information under the calibration population picks sharp items whose difficulties cluster in the middle: the central 90% of the selected difficulties ran from -0.88 to 0.97. The resulting test information peaks at 41.4 near $\theta =0$, almost four times the random subset's 10.6, but at $\theta =-2$ it is 4.1, below the random subset's 5.8. For models far from the middle the "informative" subset is worse than chance.

The last column of the table measures this directly on the 32 test models with ability more than 1.5 standard deviations from the mean. At 50 items the fixed informative subset's error rises to 3.8 points for these models, while the stratified subset, which spreads its items across the difficulty range, holds 3.0. For LLM evaluation this is not a corner case. The model you care most about is usually your newest, strongest checkpoint, which sits at the top edge of the population the items were calibrated on. A subset tuned for the middle of last quarter's leaderboard is tuned for the wrong model.

How many models do you need to calibrate?#

The adaptive method stops improving between 50 and 100 items (2.9 to 2.6 points). The reason is calibration error, not test length. With the true item parameters, the same adaptive 50-item test estimated ability with an RMSE of 0.15 units, roughly half the 0.30 obtained with fitted parameters. To see how quickly this closes, we refitted the item parameters from 30, 60, 150 and 400 calibration models (a separate seeded run) and repeated the adaptive 50-item test.

The error falls from 3.4 points with 30 calibration models to 2.2 with 400, against a floor of 1.3. Two lessons follow. First, a few dozen previously evaluated models already make adaptive testing much better than random sampling. Second, past a few hundred models the remaining error is mostly the noise in the full benchmark score itself; chasing it further is chasing a number that would change if you reran the full benchmark with a different seed. (metabench, for instance, calibrated on responses from more than 5,000 models [8], which is consistent with the small errors it reports.)

Using IRT on a real benchmark#

The fitting itself is a few dozen lines once you have a response matrix of past models by items. The ability estimate for a new model is cheaper still. Here is the EAP estimate and the adaptive selection rule from our simulation, stripped to essentials:

import numpy as np
sig = lambda x: 1 / (1 + np.exp(-x))
grid = np.linspace(-4, 4, 161)
prior = np.exp(-grid**2 / 2); prior /= prior.sum()

def eap(y, a, b):
    """Posterior mean ability given 0/1 responses y on items with params a, b."""
    p = sig(a[None, :] * (grid[:, None] - b[None, :])).clip(1e-9, 1 - 1e-9)
    ll = (y * np.log(p) + (1 - y) * np.log(1 - p)).sum(1)
    post = prior * np.exp(ll - ll.max()); post /= post.sum()
    return (post * grid).sum()

def next_item(theta, a, b, used):
    p = sig(a * (theta - b))
    info = a**2 * p * (1 - p)
    info[used] = -1.0          # never repeat an item
    return int(np.argmax(info))

def predicted_full_score(theta, a, b):
    return sig(a * (theta - b)).mean()

The choice of method depends on what you are trying to do. The table below is how we decide.

Before trusting a reduced benchmark, check:

1. Hold out whole model families. Validate the subset on models whose family and training recipe were absent from calibration. Random held-out models from the same families flatter the method.
2. Check the top edge. Report the error separately for the strongest models in the validation set, as in our tail column.
3. Test unidimensionality. Fit a one-factor model and inspect residual correlations by subject. A benchmark like MMLU [9] spans dozens of subjects; a single $\theta$ may hide a coding ability and a law ability that move independently. Multidimensional IRT or per-subject subsets fix this.
4. Freeze the prompt format. Item parameters are estimated under one template, few-shot set and scoring rule. Change any of them and recalibrate.
5. Keep the standard error. Report $\hat{\theta }$ with $1/\sqrt{I(\hat{\theta })}$, or a bootstrap interval of the predicted full score, not just a point estimate.

Failure modes#

Extrapolation beyond the calibration pool. If no calibration model was as strong as the one you are testing, the hardest items have poorly estimated parameters, and the ability estimate is held up mostly by the prior. EAP estimates are shrunk towards the population mean, so a genuinely better model can look only slightly better. This is the most common way a reduced benchmark misleads in practice, because it fails exactly on the model being celebrated.

Local dependence. IRT assumes that responses are independent given ability. Items that share a passage, a template or a code file violate this: once a model misreads the passage it misses all of its questions. Information is then overcounted and standard errors are too small, the same problem as treating clustered items as independent in an ordinary error bar, discussed in our note on clustered standard errors. Testlet models, or selecting at most one item per cluster, address it.

Contamination masquerading as easiness. If many calibration models saw an item during training, it is estimated as easy and possibly highly discriminating, since contaminated models pass and clean ones do not. Selection then favours precisely the items that measure memorisation. Run contamination checks before calibration, as described in our note on contamination detection, and treat items whose parameters shift sharply between model generations with suspicion.

Overfitting the subset to the leaderboard. A subset chosen to reproduce the scores of a fixed set of models can match those scores closely while generalising poorly to the next one. Selecting on the same models you validate on is a form of test-set reuse. Keep calibration and validation models separate.

Comparability across adaptive runs. Two models evaluated adaptively saw different items. Their $\theta$ estimates are on a common scale only if the item parameters are right. Raw accuracies from adaptive runs are not comparable at all, since a strong model was deliberately given harder items. Publish $\theta$ or the predicted full score, never the accuracy on the adaptive items.

Broken items are information too. Low-discrimination items are not only wasted budget; they are often labelling errors. Rodriguez and colleagues used IRT to surface annotation problems in leaderboard test sets [3]. Before dropping them, read them. Some are ambiguous questions you want fixed in the full benchmark as well.

What to report#

A reduced evaluation should state: the number of items and how they were chosen, the IRT model and how many models calibrated it, the predicted full score with its interval, and the validation error on held-out model families, split by ability. If the full benchmark was also run for a sample of models, show the agreement. The goal is not to replace full evaluation everywhere. It is to spend the full budget where a decision depends on it, and to make the frequent, cheaper measurements honest about their precision.

Frequently asked questions#

Sources#

1. F. M. Lord. Applications of Item Response Theory to Practical Testing Problems, 1980. doi.org
2. F. B. Baker, S.-H. Kim. Item Response Theory: Parameter Estimation Techniques (2nd ed.), 2004. doi.org
3. P. Rodriguez, J. Barrow, A. Hoyle, J. P. Lalor, R. Jia, J. Boyd-Graber. Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?, 2021. aclanthology.org
4. J. P. Lalor, H. Wu, H. Yu. Building an Evaluation Scale using Item Response Theory, 2016. aclanthology.org
5. C. Vania, P. M. Htut, W. Huang, D. Mungra, R. Y. Pang, J. Phang, H. Liu, K. Cho, S. R. Bowman. Comparing Test Sets with Item Response Theory, 2021. aclanthology.org
6. F. Maia Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, M. Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples, 2024. arxiv.org
7. R. Vivek, K. Ethayarajh, D. Yang, D. Kiela. Anchor Points: Benchmarking Models with Much Fewer Examples, 2023. arxiv.org
8. A. Kipnis, K. Voudouris, L. M. Schulze Buschoff, E. Schulz. metabench: A Sparse Benchmark of Reasoning and Knowledge in Large Language Models, 2024. arxiv.org
9. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, J. Steinhardt. Measuring Massive Multitask Language Understanding, 2020. arxiv.org