AI guardrails are runtime controls that sit around a model and check what goes in and out of it: user inputs, retrieved documents, model outputs and tool calls. Each check returns a decision (allow, modify, block or escalate to a human) according to a written policy.
Where guardrails sit
- Input checks:prompt injection and jailbreak detection, topic restrictions, personal data detection.
- Context checks: scanning retrieved or tool-returned content for indirect injection before it reaches the model.
- Output checks: harmful content, leaked secrets or system prompt text, unsupported claims, unsafe links.
- Action checks:tool-call policies on the name and arguments of each call, such as spend limits and recipient allow-lists.
Checks range from deterministic rules (regular expressions, allow-lists, schema validation) to classifiers and model-based judges. Rules are cheap and predictable; classifiers catch paraphrase but have error rates that must be measured.
Worked example
Illustrative numbers: a banking assistant has three guardrails. An input classifier flags injection attempts with a threshold chosen so that the false-positive rate on real customer messages is under 1%. An output filter redacts account numbers that do not belong to the logged-in customer. A tool policy allows transfers only to saved payees and sends anything above a set amount to human approval. Even if an injection gets past the classifier and the model tries to send money to a new account, the tool policy stops it. Layering means no single check must be perfect.
Policy as code
Mature teams write guardrail rules as versioned configuration files in the same repository as the application, reviewed and tested like code. This makes changes auditable, lets CI run the eval suite against every policy change, and avoids silent edits in a console.
Limits
Guardrails reduce risk but do not make a model safe. Classifiers can be evaded by adaptive attackers, they add latency, and their accuracy drifts as traffic changes. They should be evaluated with attack success rate and FPR on realistic data, before launch and continuously after.
Why it matters
Model behaviour is probabilistic and can change with every update. Guardrails give a deployer controls it owns, can test and can explain to an auditor, independent of the model vendor's training choices. For agents with real permissions, action-level guardrails are often the difference between an embarrassing output and an incident.
Related terms
- Guardrail false-positive rate
- Attack success rate (ASR)
- Tool-call policy
- Prompt injection
- Human-in-the-loop
- AI red teaming
Sources
- Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, et al.. Baseline Defenses for Adversarial Attacks Against Aligned Language Models, 2023. arxiv.org