LLM jailbreaks explained: why they work, attack families and how success is measured

An LLM jailbreak is an input, or a sequence of inputs, that gets a safety-trained language model to produce content or take actions its developers trained it to refuse. Jailbreaks work because refusal is a learned statistical tendency rather than a hard rule: it competes with the model's drive to be helpful, and it does not generalise to every input the model can understand.

This article explains what counts as a jailbreak, why they work at a mechanistic level, how the main families of attack relate to those mechanisms, and, in the most depth, how jailbreak success is measured and why most published numbers need a budget attached to mean anything. It describes attacks at the level of the research literature and deliberately contains no working attack prompts. For the related but distinct problem of instructions hidden in content a model reads, see prompt injection vs jailbreak.

What is an LLM jailbreak?#

Three ingredients define a jailbreak. There is a policy: a set of behaviours the developer has trained the model to refuse, such as giving uplift for weapons, writing malware or producing sexual content involving minors. There is an attacker, usually the model's own user, who wants one of those behaviours. And there is an input that gets it, where a plain request would have been refused. The attacker controls the prompt directly, which is what separates a jailbreak from prompt injection, where an attacker plants instructions in data that an application feeds to the model, and the victim is usually the application's user rather than the developer's policy.

Jailbreaks matter for two reasons that are worth keeping apart. The first is direct misuse: a jailbroken model can hand over information or carry out actions that cause harm, and for agents with tools the harm can be an action rather than text, which AgentHarm measures for multi-step agent tasks. The second is what jailbreaks say about safety training. If a model's refusal can be removed by rephrasing, the model's capability to produce the harmful output is intact, and any assessment of risk has to be based on that capability rather than on the refusal. This is why dangerous capability evaluations typically test models with safeguards removed.

Why do jailbreaks work?#

Wei, Haghtalab and Steinhardt gave the most widely used account. Safety training fails in two ways.

Competing objectives. A safety-trained model has been trained both to follow instructions and to refuse harmful ones. When an input sets those objectives against each other, for example by framing a harmful request as a task the model has strong pretraining or instruction-following priors to complete, the helpfulness objective can win. Refusal is one tendency among several, not a gate.

Mismatched generalisation. Pretraining gives a model capabilities over a far wider range of inputs than safety training covers. A model that can read an unusual encoding or a low-resource language may never have seen refusal examples in that form, so its capability generalises there and its safety behaviour does not. Yong, Menghini and Bach showed this concretely: translating unsafe requests from a standard benchmark into low-resource languages got a frontier model to engage with them 79% of the time, while high- and mid-resource languages were much less effective.

Later work found mechanisms behind these failures. Qi and colleagues argued that safety alignment is often "shallow": it mainly changes the model's distribution over the first few output tokens (the start of a refusal), so anything that gets the model past those tokens, including prefilled responses, adversarial suffixes or unusual decoding settings, tends to let the rest of the answer follow. Arditi and colleagues found that in 13 open-weight chat models, refusal is mediated by a single direction in the model's activations: erasing it stops the model refusing harmful instructions, and adding it makes the model refuse harmless ones. Both results say the same thing from different angles: refusal is a thin layer on top of capabilities that remain intact underneath.

There is also a theoretical floor. Wolf and colleagues proved, within a formal framework of behaviour expectation bounds, that any behaviour a model exhibits with nonzero probability can be triggered by some prompt, with probability that grows with the prompt's length. An alignment process that attenuates a behaviour without removing it is, in that framework, not safe against adversarial prompting. The result depends on its assumptions, but it explains why "make refusal more likely" is a weaker goal than "remove the capability" or "detect and block the output".

The main families of jailbreak attacks#

Hundreds of named attacks exist; they cluster into a handful of families, each of which exploits one of the mechanisms above. The table describes them at the level of the research literature, which is what a defender or evaluator needs.

Three trends run through the literature. Attacks have become automated: methods such as PAIR need no human creativity, often finding a jailbreak in fewer than twenty queries. They have become adaptive, choosing the next attempt based on the target's responses; Andriushchenko, Croce and Flammarion reached 100% success on every model they studied by adapting simple random search and prompt templates to each target. And human red teamers working over many turns remain strong: Li and colleagues reported above 70% success on HarmBench against defences that reported single-digit success against automated single-turn attacks. The archive's note on adaptive attack evaluations covers why this gap appears.

How are jailbreaks measured?#

The headline metric is attack success rate: of a set of harmful goals, what share did the attack break? Three choices hide inside that number, and published results differ on all three.

The goal set. Which harmful behaviours are attempted, and how hard are they? AdvBench, introduced with GCG, was an early standard; HarmBench (Mazeika and colleagues) standardised behaviours and evaluation across 18 attack methods and 33 target models and defences; JailbreakBench (Chao and colleagues) provides 100 behaviours with a fixed threat model and scoring; StrongREJECT (Souly and colleagues) chose prompts that require specific, harmful information to answer. See AI safety benchmarks for how these compare.

The judge. Who decides whether a response counts as a success? Early work often counted any response that did not refuse. Souly and colleagues found that this overstates jailbreak effectiveness compared with human judgement, partly because many jailbroken responses are useless: jailbreaks that bypass safety training also tended to reduce the model's capabilities, producing "empty" jailbreaks. Their evaluator scores how much useful forbidden information a response contains.

The budget. How many attempts does the attacker get per goal? This is the choice most often left implicit, and it matters most. The rest of this section explains why.

Why attack success depends on the number of attempts#

Model a single goal as having some per-attempt success probability $p$. If attempts are independent, the chance that at least one of $N$ attempts succeeds is $1-(1-p{)}^N$. Goals differ: some are nearly impossible, some fall easily. Let $p$ vary across goals according to a Beta distribution with mean $\mu$ and concentration $\kappa$ (smaller $\kappa$ means more variation between goals). Then the expected share of goals broken within $N$ attempts has a closed form:

where $B$ is the Beta function. For large $N$ the share of goals still unbroken decays as a power law:

with $\Gamma$ the gamma function. The exponent is $\alpha$, the Beta distribution's first shape parameter: the more the hard goals dominate, the slower the decay, but it never stops. Hughes and colleagues observed this power-law-like behaviour empirically: Best-of-N attacks kept improving over many orders of magnitude of samples, reaching 89% and 78% success on two frontier models at 10,000 attempts. The archive's note on ASR power laws fits these curves in detail.

A worked example#

Take a mean per-attempt success of 2% and concentration $\kappa =5$, so $\alpha =0.1$. A seeded simulation of 500 goals agrees with the closed form:

Three lessons follow. First, a single-attempt ASR of 2% and a 10,000-attempt ASR of 54% describe the same model; neither is "the" ASR. Second, a defence that cuts per-attempt success tenfold looks like a tenfold improvement at one attempt (2.0% to 0.2%), but at a budget of 1,000 it only reduces success from 41.8% to 27.0%, which is the undefended model's success at 100 attempts. A per-attempt defence mostly buys a constant factor in attacker effort. Third, when the attacker keeps trying until a judge says yes, the judge's false positive rate compounds: a judge that wrongly flags just 0.1% of failed attempts makes the defended model look 73% broken at 1,000 attempts when its true rate is 27%. Best-of-N style evaluation needs a judge with a very low false positive rate, or human review of every claimed success.

What a jailbreak result should report#

The AI red team report template has fields for each, and the AI red teaming guide covers how to run the exercise that produces them.

Defences, and how to evaluate them#

Defences fall into four groups. Training the model to refuse more reliably, including adversarial training on discovered attacks; HarmBench introduced an efficient adversarial training method alongside its benchmark. Changing representations: circuit breakers (Zou and colleagues) intervene on the internal representations that lead to harmful outputs rather than training refusal on specific inputs, and were designed to hold up against unseen attacks. Filtering inputs and outputs with separate classifiers: Sharma and colleagues trained classifiers on synthetic data generated from a written constitution and reported that over more than 3,000 estimated hours of red teaming no participant found a universal jailbreak that extracted information at a level of detail similar to an unguarded model across most target queries, at the cost of a 0.38 percentage point increase in production refusals and 23.7% inference overhead. Simpler filters such as perplexity checks catch some optimised strings, as Jain and colleagues found, but are bypassed by fluent attacks. System-level controls, such as rate limits, account monitoring and staged access, raise the attacker's cost per attempt, which in the power-law picture is often the most effective lever.

Each claim about a defence is only as good as the attack used to test it. Nasr and colleagues bypassed 12 recent defences against jailbreaks and prompt injection with success above 90% for most, while the majority of those defences had originally reported near-zero attack success. The lesson, familiar from adversarial examples in vision, is that a defence must be evaluated against an attacker who knows the defence and adapts to it, with a stated budget, and that its cost in over-refusal must be measured alongside its benefit (see over-refusal calibration and guardrail thresholds). For layering several defences, the AI safety overview shows why correlated blind spots limit what stacking buys.

Computing ASR at a budget#

import numpy as np
from math import lgamma, exp

def asr_at_n(n, mu, kappa):
    """Share of goals broken within n attempts when per-goal success p ~ Beta(mu*kappa, (1-mu)*kappa)."""
    a, b = mu * kappa, (1 - mu) * kappa
    return 1 - exp(lgamma(b + n) + lgamma(a + b) - lgamma(b) - lgamma(a + b + n))

rng = np.random.default_rng(5)
mu, kappa, goals = 0.02, 5.0, 500
p = rng.beta(mu * kappa, (1 - mu) * kappa, goals)        # each goal's per-attempt success
first = rng.geometric(np.clip(p, 1e-12, 1))               # attempt of first success
for n in (1, 10, 100, 1000, 10000):
    print(n, round(asr_at_n(n, mu, kappa), 3), round(float((first <= n).mean()), 3))
# 1 0.02 0.016
# 10 0.111 0.104
# 100 0.27 0.266
# 1000 0.418 0.428
# 10000 0.538 0.544

To fit the model to your own red-team data, record for each goal the number of attempts until first success (or the budget, if it was never broken) and fit $\alpha$ and $\beta$ by maximum likelihood with censoring. The fitted curve lets you extrapolate, with appropriate caution, to budgets larger than you tested, which is the question a deployment decision usually needs answered.

What to take away#

Jailbreaks are not exotic bugs. They follow from how safety training works: refusal is a learned tendency, shallow in depth and narrow in coverage, layered on capabilities that remain intact. Families of attacks map onto those weaknesses, and the most effective modern attacks are automated and adaptive. Measurement is where most confusion arises: an attack success rate without a budget, a judge with known error rates and an adaptive attacker is not comparable to any other number. For a defender, the practical goals are to raise the cost per attempt, layer defences that fail for different reasons, and report results as curves over attacker effort. For the wider context of where jailbreaks sit among AI risks, see what is AI safety; for testing in practice, red teaming vs evals.

Frequently asked questions#

Sources#

1. Greshake, Abdelnabi, Mishra, Endres, Holz, Fritz. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, 2023. arxiv.org
2. Wolf, Wies, Avnery, Levine, Shashua. Fundamental Limitations of Alignment in Large Language Models, 2023. arxiv.org
3. Wei, Haghtalab, Steinhardt. Jailbroken: How Does LLM Safety Training Fail?, 2023. arxiv.org
4. Zou, Wang, Carlini, Nasr, Kolter, Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023. arxiv.org
5. Shen, Chen, Backes, Shen, Zhang. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models, 2023. arxiv.org
6. Jain, Schwarzschild, Wen, Somepalli et al.. Baseline Defenses for Adversarial Attacks Against Aligned Language Models, 2023. arxiv.org
7. Yong, Menghini, Bach. Low-Resource Languages Jailbreak GPT-4, 2023. arxiv.org
8. Qi, Zeng, Xie, Chen et al.. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!, 2023. arxiv.org
9. Liu, Xu, Chen, Xiao. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models, 2023. arxiv.org
10. Chao, Robey, Dobriban, Hassani, Pappas, Wong. Jailbreaking Black Box Large Language Models in Twenty Queries, 2023. arxiv.org
11. Mehrotra, Zampetakis, Kassianik, Nelson et al.. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically, 2023. arxiv.org
12. Mazeika, Phan, Yin, Zou et al.. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, 2024. arxiv.org
13. Souly, Lu, Bowen, Trinh et al.. A StrongREJECT for Empty Jailbreaks, 2024. arxiv.org
14. Chao, Debenedetti, Robey, Andriushchenko et al.. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models, 2024. arxiv.org
15. Russinovich, Salem, Eldan. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack, 2024. arxiv.org
16. Andriushchenko, Croce, Flammarion. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks, 2024. arxiv.org
17. Zou, Phan, Wang, Duenas et al.. Improving Alignment and Robustness with Circuit Breakers, 2024. arxiv.org
18. Qi, Panda, Lyu, Ma et al.. Safety Alignment Should Be Made More Than Just a Few Tokens Deep, 2024. arxiv.org
19. Arditi, Obeso, Syed, Paleka et al.. Refusal in Language Models Is Mediated by a Single Direction, 2024. arxiv.org
20. Li, Han, Steneker, Primack et al.. LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet, 2024. arxiv.org
21. Andriushchenko, Souly, Dziemian, Duenas et al.. AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents, 2024. arxiv.org
22. Hughes, Price, Lynch, Schaeffer et al.. Best-of-N Jailbreaking, 2024. arxiv.org
23. Sharma, Tong, Mu, Wei et al.. Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming, 2025. arxiv.org
24. Nasr, Carlini, Sitawarin, Schulhoff et al.. The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections, 2025. arxiv.org