Evaluating long-reasoning models: pass@k, majority vote, seeds and token budgets

Long-reasoning models are sampled at non-zero temperature, think for thousands of tokens, and are usually scored on benchmarks with a few dozen problems. That combination breaks the habits most teams carried over from single-answer evals. This note works through what pass@k and majority-vote accuracy actually measure, why they diverge, how much a single seed can mislead you on a 30-problem test, and how to report accuracy against a token budget and a cost per correct answer. Every number comes from a seeded simulation we ran for this note.

A 30-problem benchmark, sampled three ways#

Take a competition-style maths set with 30 problems, the size of one year of AIME. Each problem $i$ has a true per-sample solve rate $p_i$: the probability that one sampled reasoning trace ends in the right answer. In our simulation the 30 rates are spread from 1% to 99% (evenly spaced quantiles of a logit-normal distribution), with a mean of 52.9%. That mean is the quantity most leaderboards call pass@1.

Now score that same model three ways. Draw one sample per problem and the score is a noisy estimate of 52.9%. Draw 64 samples per problem and count a problem as solved if any sample is correct, and the score is 97.8%. Draw 64 samples and take the most common final answer, and the score is 65.7%. Same model, same problems, three numbers spanning 45 points. None of them is wrong. They answer different questions, and a report that does not say which one it used cannot be compared with anything.

This is not hypothetical. The DeepSeek-R1 report [3] gives two AIME 2024 numbers for DeepSeek-R1-Zero side by side: 71.0% pass@1 and 86.7% cons@64 (majority vote over 64 samples). The same paper explains why it moved away from greedy decoding for long-output models (higher repetition and checkpoint-to-checkpoint variability) and instead estimates pass@1 by averaging correctness over $k$ sampled responses per question at temperature 0.6, with $k$ between 4 and 64 depending on test-set size. Those are the right instincts. The rest of this note makes them quantitative.

pass@k: the coverage question#

pass@k asks: if I draw $k$ independent samples, how often is at least one correct? For a single problem with solve rate $p$:

Here $p$ is the per-sample solve rate and $k$ the number of samples. The term $(1-p{)}^k$ is the chance that all $k$ samples fail, so its complement is the chance that at least one succeeds. The benchmark score is the average over problems. Because the curve rises steeply for any $p>0$, pass@k is dominated by how many problems the model can ever solve, not how reliably it solves them.

You rarely know $p$, so you estimate pass@k from $n\ge k$ samples of which $c$ are correct. The naive plug-in $1-(1-c/n{)}^k$ is biased. The standard unbiased estimator from the Codex paper [1] counts subsets:

$n$ is the number of samples drawn for the problem, $c$ the number that were correct, and $\left(\genfrac{}{}{0ex}{}{n-c}{k}\right)/\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is the probability that a random size-$k$ subset of the $n$ samples contains no correct one. Intuitively: you drew more samples than you need and you average over every way of picking $k$ of them.

pass@k is the right metric when a downstream verifier can pick out the correct sample for free: unit tests for code, a proof checker, a numeric answer you can substitute back. Repeated-sampling studies [4] showed that coverage keeps climbing over several orders of magnitude of samples in exactly those settings. Without a verifier, pass@k at large $k$ is an upper bound on something you cannot deploy.

Majority vote: the consensus question#

Majority vote (maj@k, also written cons@k) draws $k$ samples, extracts each final answer, and returns the most frequent one. It is what self-consistency decoding [2] does, and it needs no verifier. What it does need is for the correct answer to be the plurality answer, and that depends on how the wrong answers are distributed, not only on $p$.

The attractor model#

Wrong answers on hard problems are rarely uniform noise. A common sign error or a misread constraint produces the same wrong number again and again. We model this with one parameter: a fraction $a$ of the wrong probability mass lands on a single "attractor" answer, and the rest is scattered over answers that never repeat. Each sample is then correct with probability $p$, equals the attractor with probability $q=a(1-p)$, and is a one-off with probability $r=(1-a)(1-p)$. The count of correct answers $C$, attractor answers $A$ and one-offs $S$ in $k$ samples is multinomial, and the vote is won when $C$ is the largest count (ties broken at random):

$M$ is the winning count (each one-off answer has count 1), $1[C=M]$ says the correct answer is among the leaders, and $T$ is the number of distinct answers tied at $M$. We evaluate this sum exactly and checked it against 4,000 Monte Carlo votes per problem at $k=16$: 65.69% exact against 65.72% simulated.

Two consequences fall out. First, as $k$ grows the vote is decided by the expected shares, so the correct answer wins only if $p>a(1-p)$:

In words: majority vote turns every problem into a hard threshold. Problems above $a/(1+a)$ go to 100%, problems below it go to 0%, and more samples only sharpen the cut. With $a=0.5$ the threshold is $1/3$, and 20 of our 30 problems clear it, so maj@k can never exceed 66.7% however much you spend. Second, with uniform tie-breaking, maj@2 equals pass@1 exactly: two agreeing samples are either both right or both wrong, and a split is a coin flip. Our curve shows 52.9% at both $k=1$ and $k=2$.

The attractor share matters a lot. Holding the 30 problems fixed, maj@64 is 95.4% when wrong answers never repeat ($a=0$), 76.4% at $a=0.25$, 65.7% at $a=0.5$, 58.8% at $a=0.75$ and 55.5% at $a=0.9$. pass@64 is 97.8% in every case, because it ignores what the wrong answers look like. Set the difficulty mix to "concentrated" in Figure 1 and you will see the opposite failure: when most problems sit near $p=0.5$, a modest attractor puts the whole benchmark on one side of the threshold, and maj@k swings by tens of points with small changes in $a$.

Variance across seeds on a small benchmark#

With one sample per problem, the benchmark score is an average of 30 independent Bernoulli draws. Its standard deviation across random seeds is

where $N$ is the number of problems, $p_i$ the solve rate of problem $i$, and $n$ the number of samples averaged per problem. Problems near $p=0.5$ contribute most of the noise; problems the model always or never solves contribute none. For our 30 problems the formula gives 6.9 points at $n=1$, and 20,000 simulated reruns agree (6.93 points). The middle 95% of single-seed scores ran from 40.0% to 66.7% for a model whose true pass@1 is 52.9%.

Averaging $n$ samples shrinks seed noise as $1/\sqrt{n}$: 3.5 points at $n=4$, 1.7 at $n=16$, 0.9 at $n=64$. That is why averaging over many samples per question is the right default for long-reasoning models. But Figure 2 also shows a floor that sampling cannot touch. If the 30 problems are themselves a sample from a larger population of problems you care about, the score carries a question-sampling standard error of $\mathrm{SD}(p)/\sqrt{N}$, which is 6.1 points here. Averaging 64 samples tells you very precisely how the model does on these 30 problems; it tells you nothing extra about the 31st. Report both, and if you compare models, use paired differences on the same problems, which cancels much of the problem-level variance [5]. We worked through paired and clustered standard errors in detail in an earlier note.

How often a better model looks worse#

We gave a second model B the same problems with every logit shifted up by 0.21, which raises true pass@1 by exactly 3 points. Then we scored both models on independent seeds and counted how often B came out strictly behind A.

With one sample per problem, a model that is genuinely 3 points better loses or ties the head-to-head 44% of the time. Even at 16 samples it loses one comparison in ten. Checkpoint selection on a single seed is closer to a lottery than most dashboards admit, a pattern also documented across many benchmarks in seed-variance studies [6].

Majority vote does not rescue you here. We reran the 30-problem benchmark 2,000 times with maj@16 and its seed SD was 4.5 points, against 1.7 points for pass@1 averaged over the same 16 samples. Majority vote compresses each problem into one binary outcome, and problems sitting near the threshold $a/(1+a)$ flip between seeds. If you report cons@64, report its seed spread too.

Token-budget-normalised accuracy#

A long-reasoning model's accuracy is not one number but a function of how many tokens you let it think. Under a hard cap $B$, a trace that has not produced an answer by token $B$ scores zero. If $L$ is the length a trace would naturally run to,

$B$ is the token budget, $p_i$ the uncapped solve rate, and the probability term is the chance a correct trace finishes inside the budget. The curve $\mathrm{Acc}(B)$ rises from zero and saturates at pass@1. Two models can cross: one that thinks long wins at high budgets, one that thinks short wins at low budgets.

We simulated exactly that. The long-thinking model has the solve rates above and a median trace length of 8,000 tokens on easy problems, growing on hard ones and 50% longer for wrong traces (lognormal spread, $\sigma =0.55$). The short-thinking model is 0.6 logits weaker on every problem but runs at a 2,500-token base length. Each point uses 4,000 simulated traces per problem.

The crossover sits between 16k and 32k tokens. A leaderboard that runs everything with a generous cap would rank the long model first by 8.4 points; a product with a 4k-token latency budget would see the opposite ranking by 27 points. The s1 work on budget forcing [7] makes the same point from the other side: the number of thinking tokens is a control knob, and an accuracy figure is meaningless without the setting of that knob. Report $\mathrm{Acc}(B)$ at two or three budgets, or the whole curve, and state the cap you used. The DeepSeek-R1 report, for example, states a 32,768-token generation cap [3].

Cost per correct answer#

Once accuracy depends on tokens and samples, the honest comparison is cost per unit of success. For a strategy that draws $k$ samples per problem:

$k$ is the number of samples per problem, $\mathbb{E}[\min (L,B)]$ the mean tokens consumed per sample under cap $B$, and ${\mathrm{Acc}}_k$ the accuracy of the strategy (pass@1 for $k=1$, maj@k otherwise). Multiply by your price per output token to get money. Prices change; token counts do not, so we report tokens.

Three things stand out. Going from maj@8 to maj@32 on the long model buys 0.4 points for four times the tokens, because the vote has already hit its ceiling. The short model with maj@8 beats the long model's single sample (55.6% against 52.9%) but costs 2.5 times as much per correct answer. And the cheapest correct answer by far comes from the short model with one sample. Which row is "best" depends entirely on what an error costs you, which is why cost should sit next to accuracy in the report, a point made forcefully for agent benchmarks too [8]. A Pareto plot of accuracy against tokens per correct answer is the most informative single figure you can publish.

Code: estimating both metrics from samples#

Given $n$ sampled final answers per problem and the reference answer, this computes unbiased pass@k and an unbiased maj@k by averaging the vote over random size-$k$ subsets.

import numpy as np
from math import comb
from collections import Counter

def pass_at_k(n, c, k):
    # Chen et al. (2021) unbiased estimator
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

def maj_at_k(answers, ref, k, draws=2000, seed=0):
    # Monte Carlo over size-k subsets of the n samples; ties split evenly
    rng = np.random.default_rng(seed)
    answers = np.asarray(answers, dtype=object)
    win = 0.0
    for _ in range(draws):
        sub = Counter(rng.choice(answers, size=k, replace=False))
        top = max(sub.values())
        leaders = [a for a, v in sub.items() if v == top]
        win += (ref in leaders) / len(leaders)
    return win / draws

def seed_sd(p_hat, n):
    # SD of the benchmark mean across seeds, n samples per problem
    p_hat = np.asarray(p_hat)
    return np.sqrt(np.sum(p_hat * (1 - p_hat)) / n) / len(p_hat)

Treat a missing or unparseable final answer as its own value that never matches the reference, but do not let several missing answers "vote" together as one answer. Normalise answers (strip formatting, canonicalise fractions) before counting, or format variation will masquerade as disagreement.

A reporting checklist#

Failure modes we see in practice#

Mixing metrics across rows. A table that puts one model's cons@64 next to another's pass@1 compares a thresholded consensus with a per-sample rate. On our simulated benchmark that alone is worth 13 points.

Greedy decoding for a sampled model. Long-reasoning models are trained and deployed with temperature. A greedy run is one particular path, prone to repetition loops, and its score can move a lot between adjacent checkpoints [3]. If you must report a deterministic number, say so and add the sampled average.

Truncation counted as wrong without saying so. Hitting the cap is a distinct outcome. Report the truncation rate; a jump in it after a training change is a length regression, not a reasoning regression.

Answer extraction drift. Majority vote is only as good as the parser that extracts final answers. A parser that misreads "$\frac{1}{2}$" and "0.5" as different answers splits the correct vote and lowers maj@k while leaving pass@1 untouched.

Over-reading small differences. On 30 problems, a one-problem difference is 3.3 points. Unless the paired interval excludes zero, two models that differ by that much are tied.

What this simulation does not capture#

Samples are independent given the problem; real traces from one model share failure modes, which makes pass@k grow more slowly than $1-(1-p{)}^k$ suggests. One attractor per problem is a simplification; real wrong-answer distributions can have several modes. Trace length is modelled as lognormal and independent of correctness within a problem except for a fixed multiplier for wrong traces. The models are synthetic: the numbers illustrate the arithmetic, and your own benchmark will give different ones. The formulas and the checklist transfer.

Frequently asked questions#

Sources#

1. Mark Chen, Jerry Tworek, Heewoo Jun, et al.. Evaluating Large Language Models Trained on Code, 2021. arxiv.org
2. Xuezhi Wang, Jason Wei, Dale Schuurmans, et al.. Self-Consistency Improves Chain of Thought Reasoning in Language Models, 2022. arxiv.org
3. DeepSeek-AI (Daya Guo, Dejian Yang, et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. arxiv.org
4. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, et al.. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, 2024. arxiv.org
5. Evan Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, 2024. arxiv.org
6. Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, et al.. Quantifying Variance in Evaluation Benchmarks, 2024. arxiv.org
7. Niklas Muennighoff, Zitong Yang, Weijia Shi, et al.. s1: Simple test-time scaling, 2025. arxiv.org
8. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, et al.. AI Agents That Matter, 2024. arxiv.org