pass@k vs. pass^k: measuring agent reliability

Two metrics, two questions#

Let task $i$ have per-attempt success probability $p_i$, and draw tasks from some population. The two metrics ask opposite questions about $k$ independent attempts:

pass@k, popularised for code generation by Chen et al. [1], asks whether at least one of $k$ attempts succeeds. It measures capability under search: if you can sample many times and verify, how often does a correct answer exist? pass^k, introduced with τ-bench by Yao et al. [2], asks whether all $k$ attempts succeed. It measures reliability: if the same customer request arrives $k$ times, does the agent handle it correctly every time? In τ-bench's retail domain, a strong function-calling agent with roughly 61% pass^1 fell below 25% at pass^8 [2].

At $k=1$ the two coincide. They diverge immediately afterwards, in opposite directions, and how fast they diverge depends on something a single-trial eval can't see: how success is distributed across tasks.

The hidden variable: heterogeneity across tasks#

Write the task-level success probabilities as having mean $\mu$ and variance ${\sigma }^2$. For $k=2$,

So for a fixed pass@1 $=\mu$, more variation across tasks gives higher reliability. That sounds backwards until you picture it. A polarised agent solves some tasks every time and others never; a homogeneous agent is a 60% coin flip on everything. The first is predictable, the second is not. For general $k$, with $p\sim \mathrm{Beta}(a,b)$,

which decays like a power law in $k$ rather than geometrically. The homogeneous agent gets the geometric decay ${\mu }^k$.

Simulation: same pass@1, very different reliability#

Setup. Three task populations, all with mean success 0.60: homogeneous (every task $p=0.6$); moderate spread, $\mathrm{Beta}(6,4)$; polarised, $\mathrm{Beta}(0.6,0.4)$. 200,000 tasks each; metrics computed from the closed forms above by Monte Carlo over tasks.

Reading the result. The ordering flips between the two metrics. The homogeneous agent has the best pass@8 (99.9%) and the worst pass^8 (1.7%). The polarised agent is the reverse. A team optimising pass@k with best-of-n sampling and a team optimising customer-facing reliability can therefore prefer different models from the same leaderboard, and both are right for their own use.

Estimating pass^k without bias#

Running exactly $k$ trials and checking whether all succeeded is wasteful and noisy. Chen et al. [1] give an unbiased estimator for pass@k from $n\ge k$ trials with $c$ successes; the same combinatorial argument gives one for pass^k:

Both are averaged over tasks. The pass^k estimator is the probability that $k$ trials drawn without replacement from the $n$ observed are all successes, and its expectation is exactly $p_i^k$. Here is a numerically safe implementation:

from math import comb

def pass_hat_k(n: int, c: int, k: int) -> float:
    """Unbiased estimate of P(all k trials succeed) from c successes in n trials."""
    if c < k:
        return 0.0
    return comb(c, k) / comb(n, k)

# average over tasks: mean(pass_hat_k(n, c_i, k) for c_i in successes)

How many trials? We estimated pass^8 for 100 tasks drawn from the moderate-spread population (true value 4.29%) with $n$ trials per task, 4,000 replications.

Rule of thumb: use $n\approx 3k$ trials per task for a usable pass^k estimate on a 100-task suite. Since pass^k is small, the absolute error looks small, so report the interval as a ratio to the point estimate.

Estimating heterogeneity from your own data#

Since pass^k depends on how success varies across tasks, it helps to estimate that variation directly. With $n$ trials per task and observed success fractions ${\hat{p}}_i=c_i/n$, the sample variance across tasks $S^2$ mixes true heterogeneity with binomial noise:

A method-of-moments Beta fit then follows from $\hat{\mu }$ and ${\hat{\sigma }}^2$, via concentration $\hat{\mu }(1-\hat{\mu })/{\hat{\sigma }}^2-1$, and gives a smooth pass^k curve for any $k$, including values larger than $n$. Treat extrapolation beyond $n$ as a model-based estimate and label it as one.

From pass^k to user experience#

pass^k connects directly to a user-level service objective. If a typical user brings the same kind of task $m$ times a month, the probability that they see at least one failure is $1-\mathbb{E}[p^m]=1-{\text{pass}}^m$. For the moderate-spread population, ${\text{pass}}^8=5.3\%$, so a user who repeats a task type eight times a month has a 95% chance of hitting a failure, even though the "accuracy" reported for the agent is 60%. Writing reliability targets in these terms ("95% of weekly users see no failure") makes the requirement on pass^k explicit and testable.

Which number to put in the launch memo#

• Customer-facing agents (support, claims, booking): report pass^k with $k$ matched to how often a typical customer or workflow repeats the task. An agent at 60% pass^1 and 5% pass^8 will fail every regular user eventually.
• Generate-and-verify pipelines (code with tests, search with a checker): pass@k is the right metric, because a reliable verifier turns any success into a usable one.
• Safety properties (no unauthorised refund, no data egress): the relevant quantity is the per-attempt failure probability compounded over volume, $1-(1-p_{\text{fail}}{)}^N$. We cover this in retry economics of prompt injection.

Limitations#

Frequently asked questions#

Sources#

1. M. Chen et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021. arxiv.org
2. S. Yao, N. Shinn, P. Razavi, K. Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, 2024; ICLR 2025. arxiv.org