AI red teaming is structured adversarial testing of an AI system: people, automated attackers or both deliberately try to make it produce harmful output, leak data, take unauthorised actions or otherwise violate its policy, so the failures can be fixed before deployment. It tests the whole system, including prompts, retrieval, tools and guardrails, not only the model.
How a red-team exercise runs
- Scope and threat model. Who are the attackers (curious users, fraudsters, competitors), what do they want, and what can they touch?
- Harm taxonomy. A written list of failure categories with examples of what counts as a success, so results can be judged consistently.
- Attack generation. Manual probing by domain experts, plus automated methods: attacker models that rewrite prompts, optimisation attacks, multi-turn escalations such as Crescendo, and planted indirect injections for agents.
- Judging. Each attempt is labelled against the taxonomy by calibrated graders or humans.
- Reporting. Findings with reproductions, severity, attack success rate per category, and recommended mitigations.
- Regression. Successful attacks become permanent test cases in the eval suite.
Worked example
Illustrative numbers: before launching a coding agent with repository write access, a team red-teams for three harms: secret leakage, destructive commands and injection through issue text. Over two weeks they run 600 attempts. Secret leakage succeeds in 2 of 200 (1%); destructive commands in 0 of 200; injection through issue text in 23 of 200 (11.5%). The injection result blocks launch until a tool policy requires confirmation for any command that follows reading untrusted text; a rerun drops it to 1 of 200.
Red teaming vs evals
Evals measure known behaviours on fixed test sets and are repeatable. Red teaming explores for unknown failures and is adaptive. They feed each other: red teaming discovers failure modes, evals track them over time. Frameworks such as HarmBench (Mazeika et al., 2024) sit between the two, running many automated attack methods against a fixed behaviour set.
Why it matters
Benchmarks only test what someone thought to test. Red teaming is how teams find the failures they did not anticipate, with an adversary's creativity. It is also increasingly expected by risk frameworks and regulators as evidence that a system was tested against misuse before release.
Related terms
Sources
- Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, et al.. Red Teaming Language Models with Language Models, 2022. arxiv.org
- Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, Dan Hendrycks. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, 2024. arxiv.org