The short version
| Evals | Guardrails | Red teaming | |
|---|---|---|---|
| Question answered | Is this version good and safe enough to ship? | Should this specific request or action be allowed right now? | How can this system be made to fail? |
| When it runs | Before release, in CI; sampled in production | On every live request, in milliseconds | Continuously (automated) and before major changes (manual) |
| Input | A fixed, versioned set of cases | Live traffic | Adaptive, adversarial attempts |
| Output | Scores, regressions, release gate | Allow, modify, block, escalate | Findings, severities, new test cases |
| Main risk if missing | You ship regressions you can't see | One bad request becomes an incident | You only test for failures you already imagined |
Evaluations: the measurement layer
Evaluations run your system against a known set of cases and score the results. Their superpower is comparison: version A versus version B, with statistics that tell you whether a change is real. They are how you decide whether a new model, prompt or retrieval setup is an improvement, and how you prove to a risk committee that it is safe enough.
Their weakness is that they only test what's in the suite. An eval suite is a model of the world, and the world contains things the suite's authors didn't think of. That is why evals need a steady supply of new cases from production incidents and red teaming. Read more in our LLM evaluation guide.
Guardrails: the enforcement layer
Guardrails sit in the request path and make a decision about each input, output and tool call. Their superpower is that they act on reality: they stop the injected instruction in today's email and the payout to an unknown account, even if no eval anticipated that exact case. For agents, the most important guardrails enforce policy on tool calls and track where instructions came from.
Their weakness is that they have to be fast and they make mistakes in both directions. A guardrail whose false-positive rate you haven't measured is a guardrail that will quietly break legitimate users. That measurement is an eval. See the guardrails architecture guide.
Red teaming: the discovery layer
Red teaming is adversarial exploration. Its superpower is finding failure modes nobody has written down yet: the chain of three harmless-looking steps that ends in a data leak, or the new jailbreak technique published last week. Automated red teaming provides breadth and runs continuously; human specialists provide depth and business-logic creativity.
Its weakness is that discovery alone fixes nothing. A red-team report that isn't turned into controls (guardrail policies) and regression tests (eval cases) is a document, not a defence. See our red teaming methodology.
How they work together
The three controls form a loop:
- Red teaming discovers a failure, for example an injected PDF that triggers a payout.
- Guardrails contain it: a tool-call policy denies payouts triggered by retrieved content, deployed in dry-run, measured, then enforced.
- Evals lock it in: the attack and its variations become permanent test cases, so no future change can quietly reopen the hole, and the guardrail's false-positive rate is tracked on every release.
- Production feeds back: anything guardrails block or users report becomes new material for both red teaming and evals.
Red teaming finds it. Guardrails stop it. Evals make sure it stays stopped.
What to adopt first
It depends on where you are.
- Pre-launch, no agent tools: start with evals. You need to know how good the system is and to stop regressions. Add a pre-launch red-team assessment.
- Agent with real permissions (payments, email, data writes): guardrails on tool calls are non-negotiable before launch, along with a red-team engagement focused on indirect injection and excessive agency. Evals in CI follow immediately.
- Regulated industry: all three, because you'll need evidence of testing (evals), evidence of controls (guardrails) and evidence of adversarial assessment (red teaming) for auditors and frameworks such as the EU AI Act and NIST AI RMF.
- Model developer: evals and red teaming dominate, plus safety data to fix what they find in post-training.
Questions to ask any vendor
- Do your evals support agent trajectories and tool-call assertions, or only text outputs?
- What is each grader's measured agreement with expert humans, and can we see it per category?
- Can your guardrails enforce argument-level policy on tool calls, and do they track provenance?
- What is guardrail latency at p50 and p99, and false-positive rate on benign traffic?
- Do red-team findings arrive as reproducible transcripts and as runnable test cases?
- Can everything run inside our cloud account, and do you train on our data?
A worked example: an insurance claims agent
Consider an agent that reads claim submissions, including uploaded PDFs and photos, checks them against the policy, and can approve payouts up to a limit or route them to an adjuster.
What evals cover
A release suite with categories for correct coverage decisions, faithful citation of policy clauses, correct routing above the payout limit, tone with distressed claimants, PII handling, and resistance to injected instructions in documents. Each category has a threshold and an owner. When the team swaps the base model to cut costs, the suite shows that coverage decisions improved slightly but routing accuracy dropped 4 points, which is significant, so the swap waits for a prompt fix.
What guardrails cover
Tool-call policy caps automatic payouts, requires the payee to match the policyholder on file, and denies any payout whose triggering instruction came from document content. Output checks redact other claimants' details. Input checks strip hidden text from PDFs before the model sees them.
What red teaming covers
A red team discovers that a claimant can embed text in a photo's metadata that makes the agent describe damage that isn't visible, inflating the assessed amount. No eval had a case for it, and no guardrail was looking at image metadata. The finding leads to a new sanitisation rule (guardrail), forty new test cases (evals) and a new attack technique in the automated red team's library.
No single control would have been enough. Together they turn an unknown risk into a measured, enforced and permanently tested one.
Cost and effort, honestly
- Evals cost mostly engineering time up front: threat modelling, dataset assembly and grader calibration. A first release suite takes two to four weeks; running it is cheap.
- Guardrails cost integration time and some latency, and need ongoing tuning to keep false positives down. Deterministic tool policies are cheap and high-value; classifier tuning is where the effort goes.
- Red teaming is the most expertise-intensive. Automated red teaming is a platform cost; manual engagements are priced by scope and duration.
Frequently asked questions
Do I need guardrails if I already have evals?
Yes, if your system takes actions or handles sensitive data. Evals tell you how the system behaves on known cases before release; guardrails stop harmful requests and actions in production, including ones no eval anticipated.
Is red teaming the same as evaluation?
No. Red teaming is adversarial discovery of new failure modes; evaluation is repeatable measurement against known cases. Red-team findings should become eval cases.
Which AI safety control is most important for AI agents?
For agents with real permissions, tool-call guardrails with provenance tracking prevent the most damage, because they block harmful actions even when the model has been manipulated. They should be backed by evals and red teaming.
Related: Evals · Guard · Red Teaming
Keep reading
LLM evaluation: the complete guide
Metrics, graders, datasets and CI gates for evaluating LLMs and agents in production.
AI guardrails for agents: an architecture guide
Where to put guardrails, what each layer catches, and how to keep latency under budget.
AI red teaming: a field methodology
How to scope, run, score and fix an AI red-team engagement, from threat model to retest.