Item response theory: Definition and Examples

Item response theory (IRT) is a family of statistical models from psychometrics that explain test results with two kinds of latent quantity: an ability for each test taker and parameters for each item. Applied to AI evaluation, the test takers are models and the items are benchmark questions.

The two-parameter logistic model

P (y_{ij} = 1 ∣ θ_{i}) = \frac{1}{1 + e ^{- a_{j} (θ_{i} - b_{j})}}

$y_{ij}$ is 1 if model $i$ answers item $j$ correctly. $θ_{i}$ is the model's ability, $b_{j}$ is the item's difficulty (the ability at which the chance of success is 50%), and $a_{j}$ is its discrimination (how sharply success rises around $b_{j}$ ). Setting every $a_{j}$ equal gives the one-parameter Rasch model; adding a floor $c_{j}$ for guessing gives the 3PL model, useful for multiple choice.

Worked example

Illustrative numbers: a model has ability $θ = 1$ . On an item with $b = 0$ and $a = 1.5$ , its success probability is $1/ (1 + e^{- 1.5}) \approx 0.82$ . On a harder item with $b = 2$ and the same $a$ , it falls to about 0.18. If the hard item instead has low discrimination $a = 0.5$ , the probability is about 0.38: such an item barely separates strong models from weak ones and carries little information.

What IRT gives evaluators

Ability estimates on a common scale even when models were scored on different subsets of items.
Item diagnostics. Items with near-zero or negative discrimination are often mislabelled or ambiguous and can be fixed or dropped.
Efficient subsets. Choosing items with high information near the abilities you care about lets a small subset predict full benchmark scores with modest error.
Saturation checks. When most items have difficulty well below current abilities, a benchmark no longer separates models.

Caveats

IRT assumes a single ability explains performance (or a few, in multidimensional variants) and that items are conditionally independent. Benchmarks that mix skills, or contain near-duplicate items, violate both. Fitting needs many models and many items; with a handful of models the item parameters are poorly identified.

Why it matters

Raw accuracy treats every item as equally informative, which they are not. IRT tells you which items matter, how much of a score difference is real, and how to evaluate faster without losing precision, which matters when an eval suite runs on every checkpoint.

Sources

Felipe Maia Polo et al.. tinyBenchmarks: evaluating LLMs with fewer examples, 2024. arxiv.org