Attack success rate is a curve, not a number: power laws for Best-of-N jailbreaks

An attack success rate (ASR) is only meaningful next to the attacker's budget. Resampling attacks such as Best-of-N jailbreaking keep improving as the budget grows, and the aggregate curve often follows a power law. This note derives where that power law comes from, tests two ways of extrapolating ASR from a small budget to a large one, shows how a single-budget comparison can rank two defences the wrong way round, and covers intervals for rare successes. Every number comes from a seeded simulation, labelled as such.

A concrete example: 159 prompts, 100 attempts each#

A red team takes 159 harmful requests and, for each one, generates up to 100 randomly perturbed variants (shuffled characters, random capitalisation, small typos), stopping when a variant gets a harmful answer. This is the shape of Best-of-N (BoN) jailbreaking, introduced by Hughes and colleagues, which reached 89% ASR on GPT-4o and 78% on Claude 3.5 Sonnet with 10,000 augmented samples per request [1]. In our simulation the team sees 3% of requests succeed on the first attempt, 30% within 10, and 67% within 100.

The deployment question is about larger budgets. Sending 10,000 automatically generated variants costs a motivated attacker little. Will ASR at 10,000 be 75%, 90% or 99%? Running 10,000 attempts on every prompt for every model version and every defence configuration is expensive, so teams want to extrapolate. Hughes and colleagues observed that $-\log \text{ASR}$ falls as a power law in $N$ over several orders of magnitude [1], and Schaeffer and colleagues explained why [2]. That explanation also tells you when the extrapolation breaks.

Where the power law comes from#

Each prompt is exponential#

Let prompt $i$ have a per-attempt success probability $p_i$ under the attack's random augmentations. If attempts are independent draws, the probability that at least one of $N$ attempts succeeds is

where $N$ is the number of attempts. The failure probability for a single prompt decays exponentially: after about $1/p_i$ attempts, the prompt is probably broken. Brown and colleagues studied the same repeated-sampling structure on reasoning and coding problems, where coverage keeps rising with the number of samples [3].

The aggregate is polynomial#

The reported number averages over prompts:

where $P$ is the number of prompts. If the $p_i$ are spread across many orders of magnitude, each value of $N$ breaks a new slice of prompts: those with $p_i$ near $1/N$. How fast the aggregate rises therefore depends on how much mass the distribution of $p_i$ has near zero. Model that distribution as $\mathrm{Beta}(a,b)$, with density proportional to $p^{a-1}(1-p{)}^{b-1}$. The expected failure rate has a closed form:

where $B$ is the beta function and $\Gamma$ the gamma function. The approximation uses $\Gamma (b+N)/\Gamma (a+b+N)\approx N^{-a}$. Once ASR is close to 1, $-\log \text{ASR}\approx 1-\text{ASR}$, so

In words: the power-law exponent is the shape of the per-prompt difficulty distribution near zero. A small $a$ means many prompts that almost never break, and a slow climb. This is the distributional account of Schaeffer and colleagues [2], who also showed that estimating the distribution directly forecasts the exponent with much less compute than fitting the curve.

Simulation setup. 159 prompts with $p_i\sim \mathrm{Beta}(0.35,6)$: a mean per-attempt rate of 5.5%, a median of about 2%, and 17% of prompts below one success in a thousand attempts. Attempts are independent Bernoulli draws. Over the observed range the fitted exponent is $k=0.48$, not the asymptotic 0.35, because the approximation only holds once ASR is near 1. That gap is one reason power-law fits drift.

Extrapolating from a small budget#

We compared two estimators of ASR at $N=10,000$ from a budget of 30, 100 or 300 attempts per prompt, over 500 simulated red-team runs each. The power-law method fits a line to $\log (-\log \text{ASR})$ against $\log N$ and extrapolates. The distributional method fits $\mathrm{Beta}(a,b)$ by maximum likelihood to the per-prompt success counts (a beta-binomial model) and evaluates the closed form above. Both use the same attempts.

The true expected ASR at 10,000 averaged 93%. When the population really is Beta-distributed, both methods work: errors of one to two points from budgets 30 to 300 times smaller than the target. The power-law fit carries a small upward bias that does not shrink with budget, because it fits the pre-asymptotic exponent. The beta-binomial fit is unbiased here, which is unsurprising: the simulation draws from the model it assumes.

A worked example: the budget for a target ASR#

Inverting the fitted law gives the attacker budget needed to reach a target success rate $A^{\ast }$:

where $c$ and $k$ are the fitted constants. For the green run in Figure 1, fitted on 100 attempts per prompt, $c=3.75$ and $k=0.48$. Reaching 90% needs $N^{\ast }=(3.75/0.105{)}^{1/0.48}\approx 1,800$ attempts per prompt. Reaching 99% needs $-\log 0.99\approx 0.01$ in the denominator, and $N^{\ast }\approx 250,000$. Each extra "nine" of ASR multiplies the budget by roughly $10^{1/k}$, about 125 here. The exponent, not the intercept, decides how expensive the last prompts are.

The defender reads the same equation the other way. If the threat model is an attacker with a budget of 10,000 automated attempts per request, the quantity to report is ASR at 10,000, and the useful engineering target is the exponent: a defence that halves $k$ does more against a patient attacker than one that halves ASR at a single attempt. That is also why the extrapolation's assumptions matter so much: an error in $k$ of 0.05 changes the 99% budget by a factor of about three.

When the tail stops#

The extrapolation assumes the distribution of $p_i$ continues smoothly towards zero. Real prompt sets break that assumption. Some requests are refused because of a filter the augmentation never gets past, some are effectively impossible because the "harmful" completion is not something the model knows, and some defences block a whole category deterministically. Each creates a mass of prompts with $p_i=0$, which puts a ceiling below 100% on ASR at any budget.

In the red run of Figure 1, 25% of prompts are unbreakable. At budget 100, the power law predicts 83% at 10,000, and the beta-binomial predicts 81%, against an expected 72%. Neither can tell a prompt with $p_i=0$ from one with $p_i=10^{-4}$ after 100 failures, so both treat the unbreakable prompts as merely hard. The fix is not a cleverer curve. It is to run a subset of prompts to a much larger budget, look for a plateau directly, and report the forecast as conditional on the tail continuing. Jones and colleagues took the complementary route for deployment-scale risks, forecasting the largest elicitation probabilities across up to three orders of magnitude more queries than were tested [4].

Comparing defences fairly#

Suppose two defences are measured at one attempt per prompt. Defence A has an ASR of 1.3%; defence B has 2.8%. A looks twice as strong. Now let the attacker resample.

A defence that lowers every prompt's success probability a little looks good at $N=1$ and collapses under resampling, because $1-(1-p{)}^N$ goes to 1 for every $p>0$. A defence that makes a subset of prompts near-impossible looks worse at $N=1$ and holds up. Ranked by attempts needed to reach 50% ASR, A falls at 63 and B at 501. Single-budget comparisons hide this. Rules for comparing:

1. Compare full curves on a log-budget axis, or at least at the budget your threat model assumes, not at $N=1$.
2. Use the same prompts and the same augmentation seeds for both defences, so the comparison is paired and differences in prompt mix cannot masquerade as differences in defence.
3. Hold attacker compute fixed, not attempts, when attacks differ in cost per attempt. An optimised suffix attack [5] and a resampling attack are not comparable per attempt.
4. Use adaptive attacks. Andriushchenko and colleagues reported 100% ASR against a long list of safety-aligned models using simple adaptive methods, with the template and method tuned to each target [6]; a defence evaluated only against a static corpus has not been tested against what it will meet.
5. Standardise the judge. HarmBench [7] and JailbreakBench [8] exist partly because ASR moves by large amounts depending on which classifier decides that an output is harmful.

Large defence evaluations now report effort as well as outcomes. The Constitutional Classifiers work, for example, reported over 3,000 estimated hours of human red teaming alongside its automated evaluations [9]. Effort is the denominator that makes an ASR interpretable.

Confidence intervals for rare successes#

At small budgets, or against strong defences, successes are rare and the textbook interval fails. With $x$ successes in $n$ trials, the Wald interval $\hat{p}\pm z\sqrt{\hat{p}(1-\hat{p})/n}$ collapses to $[0,0]$ when $x=0$. The Wilson interval [10] inverts the score test instead:

where $\hat{p}=x/n$ and $z$ is the normal quantile (1.645 for a 90% interval). It stays sensible at $x=0$, giving an upper limit of about $z^2/n$. The Clopper-Pearson interval [11] inverts the exact binomial test and is conservative. For zero successes, the rule of three [12] gives a quick 95% upper bound of $3/n$: no successes in 3,000 attempts bounds the per-attempt rate below about $10^{-3}$.

We checked coverage of nominal 90% intervals with $n=1,000$ attempts, 20,000 simulated experiments per row:

Attempts are clustered in prompts#

There is a second, larger trap. Attempts on the same prompt share that prompt's $p_i$, so 5,000 attempts spread over 100 prompts are not 5,000 independent trials. We simulated 100 prompts with 50 attempts each, $p_i\sim \mathrm{Beta}(0.35,60)$, and estimated the mean per-attempt rate. A Wilson interval that treats all 5,000 attempts as independent covered the true mean 77% of the time at nominal 90%. A bootstrap that resamples prompts, not attempts, covered 87%: closer, though still short with only 100 heavy-tailed clusters. Report the number of prompts as your sample size, resample prompts, and see our note on clustered standard errors for the analytic version.

Practice: reporting ASR#

The core of both estimators, as used in the simulation:

import numpy as np
from scipy import optimize, special

def powerlaw_extrapolate(n_grid, asr, n_target):
    ok = (asr > 0) & (asr < 1)
    k, logc = np.polyfit(np.log(n_grid[ok]), np.log(-np.log(asr[ok])), 1)
    return np.exp(-np.exp(logc) * n_target ** k)     # k is negative here

def betabinomial_extrapolate(successes, budget, n_target):
    def nll(log_ab):
        a, b = np.exp(log_ab)
        return -(special.betaln(successes + a, budget - successes + b) - special.betaln(a, b)).sum()
    a, b = np.exp(optimize.minimize(nll, np.log([0.5, 5.0]), method="Nelder-Mead").x)
    return 1 - np.exp(special.betaln(a, b + n_target) - special.betaln(a, b))

Failure modes#

Correlated attempts. The exponential per-prompt model assumes independent attempts. Augmentations drawn from a narrow family (only capitalisation, say) produce correlated attempts, and per-prompt failure decays more slowly than $(1-p_i{)}^N$ predicts.

Judge false positives compound with budget. If the harm classifier flags a benign output as harmful with probability $f$, a prompt that is never truly broken still "succeeds" within $N$ attempts with probability $1-(1-f{)}^N$. At $f=10^{-3}$ and $N=10,000$, that is essentially certain. ASR at large budgets is only as good as the judge's false-positive rate, so successes at large budgets need human or second-judge confirmation.

Severity is not binary. A curve of "any harmful answer" mixes a vague hint with a complete procedure. Track ASR per severity tier; the high-severity curve is usually the one with the heavy tail.

Attacker capability changes the curve. Panfilov and colleagues found that attack success with LLM-based attackers depends on the capability gap between attacker and target [13]. Power-law fits from one attacker do not transfer to a stronger one.

Multi-turn and stateful attacks. Resampling a single message is the simplest budget to count. Multi-turn attacks spend their budget on conversation length and branching, and an attacker who learns from each refusal is no longer drawing independent attempts. The per-prompt exponential model then understates success at a given budget, sometimes badly. Count budget in the unit the attacker actually pays for, such as model calls or tokens, and treat curves from adaptive attackers as a separate family rather than extending a resampling curve.

Prompt set drift. The exponent is a property of the prompt distribution. Adding easy prompts raises ASR at every budget and changes the fitted exponent; compare curves only on fixed prompt sets.

The useful mental model is the one in the derivation: every prompt falls exponentially, and the population's curve is a readout of how many prompts sit near zero. Measure that distribution, check whether it has a floor, and the large-budget number stops being a guess.

Frequently asked questions#

Sources#

1. J. Hughes, S. Price, A. Lynch, R. Schaeffer et al.. Best-of-N Jailbreaking, 2024. arxiv.org
2. R. Schaeffer, J. Kazdan, J. Hughes, J. Juravsky et al.. How Do Large Language Monkeys Get Their Power (Laws)?, 2025. arxiv.org
3. B. Brown, J. Juravsky, R. Ehrlich, R. Clark et al.. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, 2024. arxiv.org
4. E. Jones, M. Tong, J. Mu, M. Mahfoud et al.. Forecasting Rare Language Model Behaviors, 2025. arxiv.org
5. A. Zou, Z. Wang, N. Carlini, M. Nasr et al.. Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023. arxiv.org
6. M. Andriushchenko, F. Croce, N. Flammarion. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks, 2024. arxiv.org
7. M. Mazeika, L. Phan, X. Yin, A. Zou et al.. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, 2024. arxiv.org
8. P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko et al.. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models, 2024. arxiv.org
9. M. Sharma, M. Tong, J. Mu, J. Wei et al.. Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming, 2025. arxiv.org
10. E. B. Wilson. Probable Inference, the Law of Succession, and Statistical Inference, 1927. doi.org
11. C. J. Clopper, E. S. Pearson. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial, 1934. doi.org
12. J. A. Hanley, A. Lippman-Hand. If Nothing Goes Wrong, Is Everything All Right? Interpreting Zero Numerators, 1983. doi.org
13. A. Panfilov, P. Kassianik, M. Andriushchenko, J. Geiping. Capability-Based Scaling Trends for LLM-Based Red-Teaming, 2025. arxiv.org