pass@k

pass@k is the probability that at least one of $k$ independent samples from a model solves a task, averaged over tasks. pass@1 is plain per-attempt accuracy; larger $k$ measures what a model can reach when it is allowed several tries and something (a test suite or verifier) picks out the correct one.

The unbiased estimator

Computing pass@k by drawing exactly $k$ samples is high variance. Instead, draw $n\ge k$ samples per task, count the $c$ correct ones, and use

The fraction is the probability that a random subset of $k$ of the $n$ samples contains no correct answer, so one minus it is the probability that the subset contains at least one. Averaging this over tasks gives an unbiased estimate of pass@k. The naive alternative, $1-(1-c/n{)}^k$, is biased low because it plugs a noisy estimate into a nonlinear function.

Worked example

A model generates $n=10$ programs for a task and $c=3$ pass the unit tests. pass@1 is $3/10=0.30$. For pass@5, $\left(\genfrac{}{}{0ex}{}{7}{5}\right)=21$ and $\left(\genfrac{}{}{0ex}{}{10}{5}\right)=252$, so the estimate is $1-21/252\approx 0.917$. A task the model solves only 30% of the time is solved more than 90% of the time with five tries.

Pitfalls

• pass@k with large $k$ assumes a perfect selector. Without one, users see pass@1, not pass@k.
• For reliability questions, the opposite metric matters: the probability that all $k$ attempts succeed, sometimes written pass^k. A model with pass@1 of 0.9 succeeds on all of 5 tries only $0.9^5\approx 0.59$ of the time on a typical task.
• Always report $n$, $k$, the sampling temperature and a confidence interval. Tasks, not samples, are the unit of independence.

Why it matters

pass@k connects evaluation to how models are used and trained. Best-of-n sampling and RL with verifiable rewards both exploit the gap between pass@1 and pass@k, and a widening or narrowing gap is a useful diagnostic of what training changed. For safety, the same arithmetic describes an attacker who retries a jailbreak: a small per-attempt success rate becomes a large one over many attempts.

Related terms

• Verifier
• Best-of-n sampling
• RLVR (reinforcement learning with verifiable rewards)
• LLM benchmark
• Attack success rate (ASR)

Sources

1. Mark Chen et al.. Evaluating Large Language Models Trained on Code, 2021. arxiv.org