AI red teaming vs evals: goals, methods, metrics and how they work together

Red teaming and evals answer different questions. An eval measures how often a system behaves well or badly on a fixed, representative set of inputs, and produces a rate with an error bar that you can track across versions. Red teaming searches adaptively for inputs that make the system fail, and produces findings: proof that a failure exists, how it is triggered and how severe it is. You need evals to know how good a system is and red teaming to learn how it breaks, and each finding from red teaming should become a permanent eval case.

This page compares the two side by side, shows with a simulation why neither can stand in for the other, and lays out how to combine them into one testing loop.

Red teaming vs evals at a glance#

What evals are for#

An eval is an instrument. You build a dataset that represents something you care about (customer requests, a policy, a capability), run the system on it, grade each output and aggregate. Because the inputs are fixed, two runs are comparable: if the harmful-compliance rate goes from 1.1% to 2.3% between versions, something changed. That comparability is what makes evals the right tool for release gates, regression detection and tracking improvement over time. The full anatomy is covered in what AI evals are, and the practice in our LLM evaluation guide.

The weakness is the same property seen from the other side. A fixed eval only knows about the inputs it contains. It does not adapt, so it under-represents rare inputs and does not represent adversarial ones at all unless someone put them there. And once a team optimises against a fixed suite, the suite measures the optimisation, a version of the problem described in specification gaming.

What red teaming is for#

Red teaming borrows its name and much of its method from security. A red team starts from a threat model (what assets the system can touch, who might attack it, what outcomes would be unacceptable), chooses attack families, probes, and follows promising leads. Ganguli et al. described early large-scale red teaming of language models and released 38,961 red-team attacks; Perez et al. showed that a language model can itself generate test cases that uncover tens of thousands of offensive replies in a target model. Automated attackers have since multiplied, from optimised adversarial suffixes (Zou et al.) and attacker models that refine a jailbreak over a few dozen queries (PAIR, Chao et al.) to plain resampling of perturbed prompts (Best-of-N, Hughes et al.). Modern engagements on agentic systems focus on what a system can do: tools, data access and actions, with prompt injection through content the system reads as a leading attack path. Our AI red teaming methodology walks through a full engagement, and the red-team report template shows what a good output looks like.

The weakness: a red team's results are not a rate. Its inputs are chosen by the attacker, not sampled from any distribution of interest, so "we found 12 issues" says nothing about how often users will hit them. And because search is open-ended, "we found nothing" is bounded by the team's skill and time. Feffer et al. surveyed red-teaming practice and argued that, when purpose, scope and follow-up are vague, it risks becoming security theater.

The core difference: existence vs rate#

Two short calculations show why the two cannot substitute for each other.

A clean eval only bounds the rate. If an eval of $n$ independent cases shows zero failures, the 95% upper confidence bound on the true failure rate is approximately

the "rule of three", which follows from solving $(1-p{)}^n=0.05$ for $p$. A clean 500-case eval is consistent with a failure rate as high as 0.6%; a clean 10,000-case eval with 0.03%. At a million requests a day, 0.6% is 6,000 failures. And that bound only covers inputs drawn like the eval's. It says nothing about inputs an attacker constructs.

A red team's success rate depends on its budget. If each attempt against a defence succeeds independently with probability $h$, the chance of at least one success within $B$ attempts is

With $h=3\%$, that is 3% for one attempt, 26% within 10, 78% within 50 and 95% within 100. An attack success rate reported without the number of attempts is not a property of the system. Our research on attack success scaling and retry risk shows that, aggregated over many requests with different difficulty, attack success often follows a power law in the budget rather than this single-probability curve, which makes stating the budget even more important.

Simulation: who finds what?#

Setup (simulation). A system has 50 latent failure modes. Forty can be triggered by ordinary traffic, each with a small per-request probability drawn from a log-normal distribution (median 0.02% per request); together they produce a natural failure rate of 1.15%, with the five most frequent modes accounting for 39% of it. The other ten are attack-only: they never fire on natural traffic and need a deliberately constructed input, like a prompt injection hidden in a retrieved document. A static eval samples $n$ requests from natural traffic. A red team makes $B$ attempts; each targets a not-yet-found mode chosen according to the attackers' priors about what is likely to work, and succeeds with a mode-specific probability (median 3% per attempt). We ran 400 replications per configuration.

The pattern is the whole argument. A 2,000-sample eval found 15 of 50 modes on average, but they were the frequent ones, accounting for 60% of the natural failure rate, and it estimated that rate accurately (1.14% on average against a true 1.15%). It found none of the attack-only modes, and no eval size would: at 50,000 samples it still found zero. A 300-attempt red team found 12 modes, including 1.2 of the 10 attack-only ones, but its finds covered only 28% of the natural failure rate, because attackers aim where they expect weakness, not where traffic is. A 1,000-attempt red team found 33 modes, 5.6 attack-only, covering 71% of natural failures. Running both (2,000 samples plus 300 attempts) found 22 modes covering 69% of natural failures, more than either alone at that cost.

Finally, the loop: when we fixed every mode a 1,000-attempt red team found and re-measured, the natural failure rate fell from 1.15% to 0.32% on average. The red team reduced risk; the eval is how you know by how much.

Which metrics belong to which?#

For evals, report rates per category with confidence intervals, and pair every safety rate with a helpfulness rate so the easiest fix (refuse more) does not look like progress. Our notes on sample size and over-refusal cover both, and LLM evaluation metrics covers the metric choices.

For red teaming, report findings, not a score: each with reproduction steps, severity in business terms, root cause and status. Where you do report a rate, report attack success rate as a curve over attempts or at a stated budget, attacker knowledge (black box or white box) and whether attacks adapted to the defence. Our research on adaptive attack evals shows how much non-adaptive testing overstates a defence's strength. Coverage of the threat model (which attack families and which assets were tested) matters more than the count of findings.

Some artefacts sit in between. Benchmarks such as HarmBench package red-teaming attacks and target behaviours into a standardised, repeatable framework so that attack methods and defences can be compared; they are evals built from red-teaming techniques. They measure resistance to known attack methods well and to tomorrow's attacks not at all. The broader catalogue is covered in AI safety benchmarks, and jailbreaks explains the attack side.

When should you use each?#

How red teaming and evals work together#

The two are strongest as one loop.

1. Evals define the baseline. Category rates with intervals show where the system is weak on representative inputs, and production monitoring shows where real users hit trouble.
2. Red teaming targets the gaps. Weak categories, new tools, high-severity assets and anything the eval cannot represent (adversarial inputs, multi-turn manipulation, injected content) get adversarial attention.
3. Findings become cases. Every finding, plus a set of variations around it, is added to the eval suite as permanent regression tests. This is how a red-team engagement keeps paying after it ends.
4. Fixes are verified by evals. The new cases, run on every change, confirm that a fix works and stays working, and the representative suite confirms that the fix did not raise over-refusal or break something else.
5. Repeat with stronger attackers. Adaptive and automated attackers, run periodically, find what the converted cases no longer catch.

Guardrails sit inside this loop too: they are tested by both, and their own thresholds are set from eval data. The comparison of evals and guardrails covers that boundary.

A small calculator for both claims#

import numpy as np
from scipy.stats import beta

def eval_upper_bound(failures, n, conf=0.95):
    """One-sided Clopper-Pearson upper bound on a failure rate from a clean-ish eval."""
    return beta.ppf(conf, failures + 1, n - failures) if failures < n else 1.0

def asr_at_budget(per_attempt, budget):
    """Chance of at least one success within `budget` independent attempts."""
    return 1 - (1 - per_attempt) ** np.asarray(budget)

print(eval_upper_bound(0, 500))           # about 0.0060, the rule of three
print(asr_at_budget(0.03, [1, 10, 50, 100]))  # 0.03, 0.26, 0.78, 0.95

Common mistakes#

• Reporting red-team ASR as a production rate. "Jailbreaks succeed 40% of the time" depends on who attacked, how hard and with what. It is not the rate users will see.
• Treating a passing eval as a security claim. A suite of known attack prompts is a regression test for those attacks. It ages fast once attackers adapt.
• Red teaming without a retest. Findings that are not converted into eval cases tend to return in the next model or prompt version.
• One without the other on a dashboard. A falling harmful-compliance rate with no adversarial testing, or a long findings list with no rates, each tells half the story.
• Testing the model, deploying the system. Both evals and red teaming should target the deployed system: prompts, tools, retrieval and guardrails included. A model-only test misses the failures that live in the integration, such as indirect prompt injection.

Published safety evaluation programmes already treat the two as parts of one system: Weidinger et al. describe a developer's safety evaluation practice that applies a broad set of approaches across risk areas rather than any single method, and NIST's Generative AI Profile (AI 600-1) lists red teaming among its pre-deployment testing practices. Shevlane et al. distinguish dangerous capability evaluations from alignment evaluations, and both draw on the adversarial effort that red teaming supplies.

Frequently asked questions#

Sources#

1. D. Ganguli et al.. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned, 2022. arxiv.org
2. E. Perez et al.. Red Teaming Language Models with Language Models, 2022. arxiv.org
3. M. Feffer, A. Sinha, W. H. Deng, Z. C. Lipton, H. Heidari. Red-Teaming for Generative AI: Silver Bullet or Security Theater?, 2024. arxiv.org
4. M. Mazeika et al.. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, 2024. arxiv.org
5. T. Shevlane et al.. Model evaluation for extreme risks, 2023. arxiv.org
6. A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, M. Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023. arxiv.org
7. P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, E. Wong. Jailbreaking Black Box Large Language Models in Twenty Queries, 2023. arxiv.org
8. J. Hughes et al.. Best-of-N Jailbreaking, 2024. arxiv.org
9. L. Weidinger et al.. Holistic Safety and Responsibility Evaluations of Advanced AI Models, 2024. arxiv.org
10. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), 2024. doi.org