Evaluating agents against adaptive, multi-turn attacks

A defence that blocks 99% of a fixed list of attack prompts has been tested against an attacker who never reads the replies. Real attackers read the replies. They rephrase after a refusal, build context over several turns, and try a different tactic when one stops working. This note sets out how to evaluate an agent or a defence against that kind of attacker: the difference between static and adaptive threat models, a simple model of success over turns, how to account for attack budgets so that numbers are comparable, and an evaluation protocol that does not quietly tune the defence to its own test set. A seeded simulation supplies the numbers.

One defence, two evaluations#

A team ships an email-handling agent with a new prompt-injection defence. Their evaluation suite has 40 attack templates crossed with 50 malicious goals (forward the inbox, change a payment address, exfiltrate a contact list). Against the undefended agent, 7.3% of template-goal attempts succeed. With the defence, 0.43% do. The launch note says the defence cuts attack success by a factor of 17.

A week later a red teamer gets a budget of 50 attempts per goal and permission to look at responses. She breaks 79% of the goals. Nothing about the defence changed; the threat model did. The published record says this pattern is the norm rather than the exception. Li and colleagues found that multi-turn human red teamers exceeded 70% attack success on HarmBench against defences that reported single-digit success rates against automated single-turn attacks. Nasr and colleagues bypassed 12 recent defences against jailbreaks and prompt injection with success above 90% for most, while the majority had originally reported near-zero attack success. Zhan and colleagues bypassed all eight indirect prompt injection defences they tested with adaptive attacks, at over 50% success.

Static and adaptive attackers#

The adversarial-examples literature went through this several years earlier. Athalye, Carlini and Wagner showed that many defences gave a false sense of security because attacks were not adapted to them, and Tramer, Carlini, Brendel and Madry later circumvented thirteen published defences whose authors had attempted adaptive evaluations, then wrote down the methodology an adaptive attack needs. Carlini and colleagues' guide to evaluating defences against adversarial examples made the central point that a defence evaluation is only as strong as the attack used in it. The same applies to language model defences, with two differences: attacks are discrete text, so gradient-based suffix search such as GCG (Zou and colleagues) is only one tool among gradient-free search, human creativity and multi-turn interaction, and the agent's own replies are a rich feedback signal.

A useful way to state a threat model has four parts:

• Knowledge: does the attacker know the defence exists and how it works? Kerckhoffs's principle says to assume yes.
• Access: single turn or multi-turn, black-box replies or log-probabilities, can the attacker plant content the agent will read later?
• Budget: how many turns, conversations, tokens or hours.
• Adaptation: does the next attempt depend on the previous replies?

A static attacker draws attempts from a fixed distribution, such as a benchmark list or a template generator, and does not use the replies. An adaptive attacker conditions on what it has seen. Andriushchenko, Croce and Flammarion summarised their results with "adaptivity is crucial": different models were vulnerable to different templates and API features, and simple random search adapted to each model, with prefilling or transfer where log-probabilities were unavailable, reached 100% success on every model in their study. Crescendo, from Russinovich, Salem and Eldan, is the multi-turn version: start with a benign question about the topic and escalate by referring back to the model's own replies.

Success over turns#

Model a single conversation. On turn $t$ the attacker succeeds with probability $p_t$. A static attacker has $p_t=p_1$ on every turn. An adaptive attacker uses each reply to improve, which we model as a constant gain in log-odds per turn:

Here $p_1$ is the first-turn success probability, $\alpha$ is the adaptation rate (how much the odds improve per turn), and $p_t$ is capped at 0.95. The intuition: each refusal tells the attacker something, and the odds of the next attempt multiply by $e^{\alpha }$. A defence may also detect the attack and end the conversation; let a failed turn be detected with probability $q$. The chance the attack has succeeded by turn $T$ is then

Each term is the probability of surviving the first $t-1$ turns without success or detection and then succeeding on turn $t$.

A worked example with $p_1=0.03$ and $\alpha =0.35$: the odds multiply by $e^{0.35}\approx 1.42$ per turn, so by turn 12 the per-turn success is 59%. Over 12 turns the static attacker succeeds in 31% of conversations and the adaptive one in 97%. With a monitor that ends 10% of failed turns, the figures fall to 19% and 54% (a 200,000-conversation Monte Carlo run gave 53.9% for the last).

The point of Figure 1 is in its first column. A single-turn benchmark sees 3% for both attackers. By turn 12 one of them succeeds about three times as often as the other. Single-turn numbers are not a lower-resolution view of multi-turn risk; they measure a different quantity.

Estimating the curve from your own red-team logs#

The two parameters are measurable. Fit a logistic regression of per-turn success on turn index, with a separate intercept for each goal so that easy and hard goals do not masquerade as adaptation:

Here ${\beta }_g$ is the first-turn log-odds for goal $g$ and $\alpha$ is the shared adaptation rate. Fit it only on turns the attacker actually reached, which is what the conversation logs contain anyway. Three cautions. Human red teamers often front-load benign context, so the first few turns can show negative $\alpha$ before escalation; fit a piecewise slope if your logs show it. Automated attackers have an $\alpha$ that depends on the attacker model and prompt, so record which attacker produced each conversation and fit separately. And a high $\alpha$ is a property of the attacker and the target together: a defence that refuses more consistently across turns lowers $\alpha$ even if it does nothing to $p_1$, which is a real improvement that single-turn metrics cannot show.

For agents, the conversation is not the only channel. In indirect prompt injection the attacker controls content the agent reads (a web page, an email, a calendar invite) rather than the dialogue, and adaptation happens across sessions: the attacker edits the planted content after watching what the agent did. The same model applies with "session" in place of "turn", and the same need to log which version of the planted content each session saw. Our note on evaluating agent trajectories covers how to score what the agent did once an injection lands.

Budget accounting#

Attack success rate without a budget is not a number. A conversation-level success rate depends on how many turns a conversation may last; a goal-level success rate depends on how many conversations the attacker may start. The unit that makes results comparable is the total number of model calls (turns) the attacker spends, and the honest summary is a curve: success against budget. Hughes and colleagues showed for Best-of-N jailbreaking that success keeps rising with the number of sampled prompts and follows power-law-like behaviour over many orders of magnitude, which is why we argued in an earlier note that attack success rate is a curve, not a number.

We simulated an attacker with a budget of $B$ turns in total. A conversation ends on success, on detection, or after 12 turns, and the attacker then starts a new one from scratch. Each point is 20,000 simulated attackers:

Two practical lessons follow. First, a defence that ends suspicious conversations is only as strong as its memory: against an attacker who can restart for free, it converts a 10-turn budget from 86% to 68% success, and makes no difference at 50 turns. Detection that persists across conversations (account-level flags, rate limits, review queues) is what turns a monitor into a cost for the attacker. Second, comparisons between defences are only meaningful at matched budgets, and a single matched budget can rank defences differently from another. Report the curve, or at least two budgets an order of magnitude apart.

How defence evaluations overfit#

The opening example hides a second problem. The defence was developed while looking at the same 40 templates used to evaluate it. We simulated that process. There are 200 attack families; each has a success probability against the undefended agent drawn from a skewed distribution (mean about 6%), and 50 goals vary in difficulty. The defence is tuned on 40 families: it cuts their success by 95% and, because some of what it learned transfers, cuts every other family's success by 30%. We then evaluated it three ways, averaging 20 seeds:

The static number on the development families says the defence is 17 times better. Held-out families say it removes about 30% of success, which is what it actually does to unseen attacks. The adaptive attacker here is modest: it only chooses among existing families, using Thompson sampling on its own successes and failures, and invents nothing new. Even so, with 50 attempts per goal it breaks 79% of goals against the defended agent, against 92% undefended. The defence's real effect shows up as a shift in the budget curve: at 10 attempts per goal, 32% against 49%. That is a worthwhile improvement, and it is a very different claim from "18 times fewer attacks succeed".

An evaluation protocol that does not overfit#

Agent environments make this easier than it used to be. AgentDojo, from Debenedetti and colleagues, was built as an extensible environment rather than a static suite, so that new attacks and defences can be added and adaptive attacks evaluated against them. HarmBench, from Mazeika and colleagues, standardised goals and judging for automated red teaming. Neither removes the need for held-out families and declared budgets.

Reporting with uncertainty#

Attack success is measured over a small number of goals, and attempts on the same goal are correlated, so the goal is the unit of analysis. With $n$ goals and $x$ broken, the Wilson interval is

where $\hat{p}=x/n$ and $z=1.96$ for 95% confidence. It behaves sensibly near 0 and 1, where the textbook normal interval does not. A result of zero broken goals out of 50 has a 95% upper bound of 7.1%, so "0% attack success" on 50 goals is consistent with one goal in fourteen being breakable. If several attempts per goal are pooled at the attempt level, use clustered standard errors with goals as clusters, or the interval will be too narrow.

Code: success-at-budget curves from attack logs#

import numpy as np

def success_at_budget(logs, budgets):
    """logs: list of (goal_id, attempt_index_within_goal, turns_used, success) rows, in attack order.
    Returns, for each budget B in turns, the share of goals broken within B turns,
    with a bootstrap interval over goals."""
    by_goal = {}
    for goal, _, turns, ok in sorted(logs, key=lambda r: (r[0], r[1])):
        spent, first = by_goal.get(goal, (0, None))
        spent += turns
        if ok and first is None:
            first = spent                      # turns spent when the goal first fell
        by_goal[goal] = (spent, first)
    firsts = np.array([f if f is not None else np.inf for _, f in by_goal.values()])
    rng = np.random.default_rng(0)
    out = []
    for B in budgets:
        broken = (firsts <= B).astype(float)
        boots = [rng.choice(broken, len(broken)).mean() for _ in range(2000)]
        out.append((B, broken.mean(), *np.quantile(boots, [0.025, 0.975])))
    return out

Failure modes#

• Single-turn evaluation of a multi-turn product. It measures $p_1$ and nothing about $\alpha$.
• Reporting ASR without a budget. Every ASR is a point on a curve; say which point.
• Evaluating on the development attacks. The table above shows how large that bias can be.
• Conversation-level defences without memory. Ending a conversation costs the attacker one restart.
• Judging success with a drifting judge. A changed judge prompt or model changes the success rate; version it with the results.
• Counting "0%" as proof. Report the upper bound and the budget.

The models here are deliberately simple: a constant log-odds gain per turn, independent detection per turn, attack families with fixed success probabilities, and an adaptive attacker that only reallocates effort among known families. Real attackers also invent new families, which makes the held-out and adaptive numbers optimistic for the defender. The direction of every comparison in this note would only get stronger.

Frequently asked questions#

Sources#

1. Athalye, Carlini, Wagner. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples, 2018. arxiv.org
2. Carlini, Athalye, Papernot, Brendel et al.. On Evaluating Adversarial Robustness, 2019. arxiv.org
3. Tramer, Carlini, Brendel, Madry. On Adaptive Attacks to Adversarial Example Defenses, 2020. arxiv.org
4. Zou, Wang, Carlini, Nasr et al.. Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023. arxiv.org
5. Chao, Robey, Dobriban, Hassani et al.. Jailbreaking Black Box Large Language Models in Twenty Queries, 2023. arxiv.org
6. Mehrotra, Zampetakis, Kassianik, Nelson et al.. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically, 2023. arxiv.org
7. Mazeika, Phan, Yin, Zou et al.. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, 2024. arxiv.org
8. Russinovich, Salem, Eldan. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack, 2024. arxiv.org
9. Andriushchenko, Croce, Flammarion. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks, 2024. arxiv.org
10. Debenedetti, Zhang, Balunović, Beurer-Kellner et al.. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents, 2024. arxiv.org
11. Li, Han, Steneker, Primack et al.. LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet, 2024. arxiv.org
12. Hughes, Price, Lynch, Schaeffer et al.. Best-of-N Jailbreaking, 2024. arxiv.org
13. Zhan, Fang, Panchal, Kang. Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents, 2025. arxiv.org
14. Nasr, Carlini, Sitawarin et al.. The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections, 2025. arxiv.org