AI guardrails for agents: an architecture guide

What AI guardrails are

An AI guardrail is a runtime check that inspects something flowing into or out of a model-based system and decides whether to allow it, modify it, block it or escalate it to a human. Guardrails differ from evaluations in timing: evals measure behaviour before release on a fixed set of cases, while guardrails enforce policy on live traffic, one request at a time.

The term covers a wide range of controls: topic restrictions on a support bot, PII redaction on outputs, jailbreak detection on inputs, spend limits on a payments tool, human approval before an agent deletes anything. What they have in common is that they are independent of the model. A guardrail is useful precisely because it doesn't rely on the model behaving well.

Why agents break the old guardrail model

First-generation guardrails sat at two points: user input and model output. That made sense when the only thing a model could do was produce text for a human to read. Agents change three things.

• Untrusted content enters through tools. An agent reads emails, web pages, PDFs, tickets and API responses. Any of them can carry instructions. An input filter on the user's message never sees them.
• Harm happens through actions. A perfectly polite output can accompany a tool call that wires money to an attacker. Output text filters can't see tool arguments.
• Context accumulates. Multi-turn and multi-step attacks spread a malicious goal across many harmless-looking messages. Per-message checks miss them.

Any architecture for agent guardrails has to cover these blind spots. The rest of this guide is about how.

The four checkpoints

We model guardrails as four checkpoints around the model, all governed by one policy engine and one audit log.

Most incidents we see in red-team engagements pass through ② and ③. That is also where most products are weakest, because those checkpoints require knowing where each piece of text came from and what each tool is allowed to do.

Provenance: knowing who said what

The single most effective defence for agents is tracking the provenance of every token in the context window. When the system knows that a particular instruction came from a retrieved web page rather than from the authenticated user, it can apply very different rules: an instruction from a document should never be able to authorise a payment, change a recipient or reveal the system prompt.

Practically, this means tagging content at the boundary where it enters (user, system, tool result, retrieval, memory) and carrying that tag through to the tool-call checkpoint. Policies can then say things like "deny issue_refund if the triggering instruction has provenance retrieved". In our AgentBreach-2 benchmark, combining provenance with tool-call policy cut successful indirect injections by 92% compared with an undefended agent, where input filtering alone managed about 44%.

Writing guardrail policy as code

Guardrail rules configured by clicking in a vendor console drift, go unreviewed and are impossible to test. Treat them like infrastructure:

• Keep policy in your repository, in a declarative format reviewers can read: tools, limits, allow-lists, thresholds and actions.
• Test policy changes with evals before they go live: measure both what newly gets blocked (false positives on benign traffic) and what still gets through (attack success on an adversarial corpus).
• Roll out in dry-run first. Log decisions without enforcing for a week, review them, then enforce.
• Version everything so every logged decision references the exact policy version that made it.

tools:
  issue_refund:
    max_amount_usd: 500
    require_approval_above: 200
    deny_if_provenance: [retrieved, tool_output]
  send_email:
    recipients: customer_on_ticket_only
output:
  pii: { action: redact, types: [card, ssn, iban] }

Choosing the right action

Blocking is only one option, and often not the best one. A good guardrail layer supports a graded set of responses:

1. Allow and log: for low-risk signals worth tracking.
2. Transform: redact PII, strip hidden HTML, remove URLs from outputs.
3. Constrain: drop the agent's permissions to read-only for the rest of the session.
4. Escalate: require human approval before a high-impact tool call executes.
5. Block: return a safe response and raise a security event.

Match the action to the cost of a mistake in each direction. Blocking a benign customer is a support cost; executing a fraudulent payout is an incident.

Staying inside a latency budget

Guardrails that add half a second to every turn get switched off. Production budgets are usually 20–50 ms of added latency at the median. To get there:

• Run checks in parallel with each other and, where safe, with the model call itself. Input checks can race the first model call and cancel it if they fire.
• Use small, specialised classifiers for high-volume checks, and reserve large-model judgement for the rare ambiguous cases.
• Inspect streams in chunks so output checks don't delay time-to-first-token.
• Put deterministic rules first. Spend limits and recipient allow-lists cost microseconds and stop the most damaging actions.
• Deploy close to the agent, as a sidecar or in-VPC proxy, to avoid extra network hops.

Measuring guardrail effectiveness

Every guardrail has two error rates, and you need both. Attack success rate on an adversarial corpus tells you what still gets through. False-positive rate on a representative sample of benign production traffic tells you what you break. Track both per category and per policy version, and re-measure whenever the model, the prompt or the policy changes. Continuous red teaming keeps the adversarial corpus fresh, because attackers adapt and a static test set goes stale within months.

Guardrails for MCP servers and tool ecosystems

As agents connect to more tools through standard protocols such as the Model Context Protocol, the tool layer becomes the main attack surface. Each new MCP server adds capabilities, and also adds a new channel through which untrusted content flows back into the model. Three practices keep this manageable:

• Inventory and pin tools. Keep an allow-list of approved servers and versions. A tool whose description or schema changes silently can change what the agent believes it is allowed to do.
• Treat tool descriptions as untrusted input. Tool metadata is read by the model and can carry injected instructions. Scan it on registration and on every change.
• Enforce policy at the tool boundary, through a sidecar next to the tool server or a gateway in front of it, so every call is checked regardless of which agent or framework made it.

This also simplifies ownership. The team that owns a payments tool owns its policy, and every agent that uses the tool inherits the same limits automatically.

A rollout plan that won't break production

1. Week 1, observe: deploy in log-only mode on all four checkpoints. Collect a week of decisions on real traffic.
2. Week 2, tune: review blocks with product owners, adjust thresholds, and add allow-list exceptions for legitimate edge cases.
3. Week 3, enforce the deterministic rules: tool-call limits and recipient restrictions go live first. They have near-zero false positives and the highest impact.
4. Week 4, enforce classifiers on the categories where measured precision is acceptable, and keep the rest in flag-for-review.

From then on, every policy change goes through the same pipeline as code: pull request, eval run against adversarial and benign corpora, dry-run, enforce.

Frequently asked questions

Related: Reinforced Guard · Prompt injection defense · Evals vs. guardrails

Keep reading