A many-shot jailbreak is an attack that places dozens or hundreds of fabricated question-and-answer pairs in the prompt, each showing an assistant happily answering a harmful request, followed by the attacker's real request. The model's in-context learning picks up the pattern and continues it, overriding its safety training.
How it works
Language models are strong in-context learners: given examples of a format, they continue it. Safety training teaches a prior to refuse, but a long run of in-context examples is evidence about "how this assistant behaves here", and with enough of it the in-context evidence can outweigh the prior. The attack became practical once context windows grew from a few thousand tokens to hundreds of thousands, which leaves room for hundreds of "shots".
Worked example
Illustrative numbers: an attacker writes 256 short fake exchanges in the form "User: How do I pick a lock? Assistant: Here are the steps: ..." covering many disallowed topics, then ends with "User: [real harmful request] Assistant:". With 5 shots the target model refuses. With 256 it is far more likely to answer. Anil et al. (2024) found that the attack's effectiveness follows a power law in the number of shots, up to hundreds of shots, which means a defence that only shifts the curve buys a constant factor in context length, not immunity.
Why the power law matters
Concretely, the negative log-likelihood the model assigns to the harmful response falls with the number of shots roughly as
where and are constants fitted per model and task. A lower NLL means the harmful continuation is more likely, so the attack keeps getting easier as context grows. Evaluations should therefore report attack success rate as a curve over shot count, not a single number.
Defences
- Fine-tuning on many-shot examples raises the intercept but tends not to remove the trend.
- Classifying and rewriting the prompt before it reaches the model, for example detecting long runs of user/assistant formatted turns inside a single message, is more effective.
- Applications that do not need very long user-supplied context can cap it.
Why it matters
Many-shot jailbreaking shows that capability gains can create new attack surface: the longer context that makes models more useful also makes this jailbreak work. It is a standard case for long-context safety evaluation and for input guardrails that inspect structure, not only keywords.
Related terms
Sources
- Cem Anil, Esin Durmus, Nina Panickssery, Mrinank Sharma, et al.. Many-shot Jailbreaking, 2024. proceedings.neurips.cc