A jailbreak is an input designed to make a language model produce content its safety training or usage policy says it should refuse, such as detailed instructions for causing harm. The term covers hand-written prompts, automatically optimised strings and multi-turn conversations.
Why jailbreaks work
Wei et al. (2023) proposed two failure modes that explain most jailbreaks:
- Competing objectives. The model is trained both to be helpful and to be harmless. Prompts that frame the request as role-play, fiction, a test, or a task with a required output prefix ("Start your answer with 'Sure, here is'") push the helpfulness objective above the safety one.
- Mismatched generalisation. Safety training covers less of input space than pretraining. Requests encoded in Base64, translated into a low-resource language, or split across pieces can fall outside what safety training saw while remaining understandable to the model.
Main families
- Manual prompts: personas, hypothetical framing, refusal suppression.
- Optimisation attacks: GCG (Zou et al., 2023) searches for an adversarial suffix by gradient-guided token swaps; such suffixes often transfer between models.
- Attacker-model attacks: PAIR (Chao et al., 2023) uses one model to iteratively rewrite a request until a target model complies, often within about twenty queries.
- Multi-turn and long-context attacks:Crescendo escalates gradually; many-shot jailbreaking fills the context with fake compliant examples.
How jailbreaks are measured
The standard metric is attack success rate: the share of harmful behaviours for which the attack elicits a genuinely harmful completion, judged by a calibrated classifier or human. Two details matter. First, a "success" must be harmful, not merely a non-refusal, or rates are inflated. Second, report ASR at a fixed attempt budget, since an attacker who retries 50 times with a 2% per-attempt success rate succeeds at least once with probability .
Jailbreak vs prompt injection
A jailbreak attacks the model's own policy; prompt injection attacks the application around the model, often to make it misuse tools or leak data. One technique can serve both goals.
Why it matters
Every deployed model has some jailbreak rate above zero, so the useful questions are how much effort a jailbreak takes, what harm it enables, and what other layers catch it. Strong safety evaluations measure ASR against adaptive attacks at several budgets, alongside over-refusal on benign prompts, because making a model refuse everything is not a solution.
Related terms
- Prompt injection
- Crescendo attack
- Many-shot jailbreak
- Attack success rate (ASR)
- AI red teaming
- Adversarial robustness
Sources
- Alexander Wei, Nika Haghtalab, Jacob Steinhardt. Jailbroken: How Does LLM Safety Training Fail?, 2023. arxiv.org
- Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023. arxiv.org
- Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries, 2023. arxiv.org