A 2% attack success rate is not low: retry economics of prompt injection

From per-attempt to per-campaign#

Suppose a red team reports that an agent resists injected instructions in 98% of attempts, an attack success rate (ASR) of $p=0.02$. If an attacker can try $n$ independent payloads, for example by sending $n$ emails to an inbox the agent processes, the probability that at least one works is

The approximation shows the scale directly: the attacker needs about $1/p$ attempts to reach a 63% chance of success. At $p=2\%$ that is 50 emails.

Two conditions make this realistic rather than theoretical. Indirect injection costs the attacker nothing per attempt: the payload sits in an email, a web page or a shared document and waits for the agent to read it [1]. And most agent deployments read untrusted content at volume. An inbox assistant processing 2,000 external emails a day gives a 0.1%-ASR payload family roughly two expected successes a day if every email carried a payload.

Does payload diversity change the picture?#

Real attackers don't resend one payload; they vary it. We tested whether diversity matters by drawing each attempt's success probability from $\mathrm{Beta}$ distributions with the same mean. For independent attempts it makes no difference: the expectation of a product of independent terms is the product of the expectations, so $\mathbb{E}[{\prod }_j(1-p_j)]=(1-\bar{p}{)}^n$. Our simulation confirmed it (0.5% mean: 39.5% at 100 attempts with diverse payloads vs. 39.4% for a fixed payload).

What does change the picture is adaptation. An attacker who observes which payloads partly worked and concentrates on them samples from the upper tail of the distribution, not its mean. That is why automated attackers that iterate on feedback, such as gradient-guided suffix search [2] or multi-turn escalation [3], find working attacks far faster than static corpora predict. A per-attempt ASR from a static test set is a lower bound on what an adaptive attacker achieves.

Defence in depth, and the independence trap#

The standard response is layering: an input classifier, a provenance rule and a tool-call policy. If layer $j$ misses with probability $m_j$ and the layers fail independently, the attack gets through all of them with probability ${\prod }_j{m}_j$. For three illustrative layers with miss rates $0.2,0.15,0.1$, that is $0.3\%$: a reassuring number.

Layers aren't independent, though. The same payload property, such as instructions phrased as quoted data or a low-resource language, can defeat a classifier and slip through a provenance heuristic. We modelled the dependence with a Gaussian copula: each layer has a latent standard-normal score $Z_j$, pairwise correlation $\rho$, and misses when $Z_j<{\Phi }^{-1}(m_j)$. Then

where ${\Phi }_3$ is the trivariate normal CDF with equicorrelation matrix ${\Sigma }_{\rho }$. We estimated it with 2 million draws per $\rho$.

Finding. At $\rho =0.4$, a plausible value for two ML-based detectors trained on similar data, the three-layer stack lets through 2.0% of attacks, not 0.3%. At $\rho =0.8$ it is 6.1%, barely better than the weakest layer alone (10%). Layers that share a failure mechanism add much less than their individual miss rates suggest.

Designing layers that actually multiply#

The lesson isn't "don't layer". It is to layer controls with different failure mechanisms, so their failures are as close to independent as possible:

Deterministic controls on the action side are close to uncorrelated with anything an attacker can put in a payload, which is why they carry most of the weight in our guardrails architecture. Two LLM-based detectors in series are the most correlated design we see, and they are common.

Measuring correlation in your own stack

Correlation is cheap to estimate. Run your adversarial corpus through each layer in isolation, record which attacks each one misses, and compute the phi coefficient between layers' miss indicators. If the joint miss rate is well above the product of the marginals, your layers are sharing a failure mode, and the fix is a different kind of control, not a better classifier.

An exposure budget#

Inverting the retry formula gives the per-attempt success rate you can tolerate for a target campaign-level risk $P$ over $N$ exposures:

An inbox agent reading 2,000 external emails a day sees about 730,000 per year. Keeping the yearly chance of even one successful high-impact injection below 1% requires $p_{\max }\approx 0.01/730,000\approx 1.4\times 10^{-8}$ per email, assuming every email could carry a payload. No detector is measured at that level, and none could be: verifying a rate near $10^{-8}$ takes on the order of $10^8$ or more adversarial trials. The only way to meet such a budget is to make the high-impact action unreachable from untrusted content by construction, with provenance rules and deterministic policy. Detection then only has to handle lower-impact outcomes, where budgets of $10^{-3}$ are tolerable.

How to report injection risk honestly#

1. Report ASR per payload family and per channel, with intervals (see sample sizes), not one blended number.
2. Convert to campaign-level risk using your actual exposure: $1-(1-p{)}^N$ with $N$ = untrusted documents processed per day.
3. Report the joint miss rate of your layers measured end-to-end, not the product of individual rates.
4. Re-measure with adaptive attackers. A static corpus understates the risk. Dynamic environments such as AgentDojo [4] make it practical to re-run evolving attacks and defences against realistic agent tasks.

Limitations#

Frequently asked questions#

Sources#

1. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, M. Fritz. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173, 2023. arxiv.org
2. A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, M. Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043, 2023. arxiv.org
3. M. Russinovich, A. Salem, R. Eldan. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack. arXiv:2404.01833, 2024. arxiv.org
4. E. Debenedetti et al. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS 2024 Datasets and Benchmarks. arXiv:2406.13352. arxiv.org