AI red teaming: a field methodology

What AI red teaming is, and isn't

AI red teaming is structured adversarial testing of an AI system to find ways it can be made to cause harm: leaking data, taking actions it shouldn't, producing dangerous or prohibited content, or bypassing the policies it is supposed to enforce. The output of a good engagement is a set of reproducible findings, ranked by severity, with root causes and fixes, plus tests that make sure those findings never come back.

It is not a contest to make a chatbot say something embarrassing. Those findings have their place, but for most organisations the material risks sit in what the system can do: the tools it holds, the data it can reach and the decisions it makes. Red teaming should spend its time where the damage is.

Step 1: scope and threat model

Every engagement starts with three lists, built with the system owners:

1. Assets: data the system can read (customer records, internal documents, credentials) and actions it can take (payments, emails, deletions, code execution).
2. Actors: who might attack it. Anonymous users, authenticated customers, malicious insiders, and the authors of any content the system ingests (web pages, emails, uploaded files).
3. Unacceptable outcomes, phrased in business terms: "a payout to an account not on file", "one customer sees another's claim history", "the assistant gives a dosing instruction".

Agree on severity definitions up front, in the organisation's own language. A finding's severity should depend on the outcome and how easy it is to reproduce, not on how clever the attack was.

Step 2: choose attack families

Map the threat model to attack families. A typical agent engagement covers most of these:

Public taxonomies such as the OWASP Top 10 for LLM Applications and MITRE ATLAS make useful coverage checklists, and help when findings have to be reported to security leadership.

Step 3: automated sweep

Before humans spend time, run automation. Attacker models generate and adapt thousands of attacks across the families above, using feedback from partial successes to refine the next attempt. The automated sweep does three jobs: it finds the obvious weaknesses quickly, it maps how the system behaves under pressure, and it produces a baseline attack success rate per family that you can track over time.

Automation is breadth. It is very good at variations on known techniques and very bad at business logic: it doesn't know that a refund to a newly added bank account is the dangerous case, or that a claims adjuster's notes field is rendered to the customer. That is what people are for.

Step 4: manual exploitation

Human specialists take the automated map and look for chains: a low-severity information leak that reveals an internal tool name, which makes an injection more convincing, which triggers an action. Real incidents almost always look like this. Good manual testers think like the attacker in the threat model, not like a prompt engineer, and they work the system's actual integrations: a staging environment with real tool schemas, realistic seeded data and sandboxed side effects.

For capability and misuse testing, include domain experts. Judging whether an answer about pathogen enhancement or exploit development provides meaningful uplift needs a virologist or an exploit developer, not a generalist.

Step 5: report findings that get fixed

Each finding needs:

• A reproducible transcript, including all injected content and the exact tool calls made.
• Severity against the agreed definitions, plus a reproduction rate across multiple attempts, since non-deterministic systems may only fail some of the time.
• The root cause: missing tool-call policy, over-broad permissions, no provenance separation, weak system prompt, missing output filtering.
• A recommended control, preferring architectural fixes (permissions, policy, provenance) over prompt patches, which tend to be brittle.

A prompt patch is a promise. A tool-call policy is a control.

Step 6: retest and make it permanent

Once fixes land, retest every finding, and also test variations, because a narrow fix often closes one phrasing and leaves the technique open. Then encode every finding and its variations as cases in your eval suite so a future model swap or prompt edit can't silently reintroduce them. Finally, keep automated red teaming running against staging continuously. Models, prompts and attackers all change, and last spring's report says little about today's system.

How often to red team

• Continuously (automated): nightly against staging, with results tracked as attack success rates per family.
• Before material changes (manual): new tools or permissions, new data sources, a new base model, a new user population.
• Periodically (manual): at least twice a year for high-risk systems, as part of your security programme.

A severity model that works for AI findings

Traditional vulnerability scoring assumes a deterministic exploit. AI findings are probabilistic: an attack might work three times in ten. We score findings on three axes and combine them:

Note that low reliability isn't low risk when an attacker can retry for free. A 2% success rate against a system that processes every inbound email is a near-certainty at scale.

Building an internal AI red team

Organisations with several AI systems in production usually end up with a small internal function. What works:

• Put it in security, staffed with ML literacy. AI red teaming is offensive security work; it needs attacker instincts more than prompt-writing skill.
• Own the attack corpus. The internal team curates the organisation's library of attacks and findings and makes sure they run as automated tests across every AI system.
• Use external specialists for depth and independence: before major launches, for domain misuse testing, and when regulators or customers want an independent assessment.
• Measure outcomes, not activity. Track attack success rate per family over time, time-to-fix for findings, and the share of findings that reappear after being fixed.

Frequently asked questions

Related: Reinforced Red · Prompt injection · LLM evaluation

Keep reading