● Reference
AI safety glossary.
44 terms from evaluation, guardrails, red teaming and post-training, defined plainly and linked to our deeper guides.
- Adversarial robustness
- How well a model or system maintains intended behaviour when inputs are deliberately crafted to make it fail.
- Agent
- An AI system that plans and takes actions, usually by calling tools such as APIs, browsers, databases or code execution, in pursuit of a goal.
- Agent trajectory
- The full sequence of steps an agent takes in a run: reasoning artefacts, tool calls, arguments, tool results and the final response. Trajectory evaluation scores the whole sequence, not just the final answer.
- AI guardrails
- Runtime controls that inspect inputs, retrieved content, outputs and tool calls of an AI system and allow, modify, block or escalate them according to policy. See AI guardrails guide.
- AI red teaming
- Structured adversarial testing of an AI system to discover how it can be made to cause harm. See AI red teaming methodology.
- Attack success rate (ASR)
- The share of adversarial attempts that achieve the attacker's goal. The primary metric for red teaming and for measuring guardrail effectiveness.
- Benchmark
- A public, fixed test set used to compare models on a shared task. Useful for model selection; insufficient for deciding whether a specific system is safe to ship.
- Calibration (grader)
- Measuring and adjusting a model-based grader so that its verdicts agree with expert human labels for a given category.
- CI gate
- An automated check in a continuous-integration pipeline that blocks a change from merging or releasing if eval scores fall below thresholds or regress significantly.
- Cohen's kappa (κ)
- A statistic for agreement between two raters that corrects for agreement expected by chance. Values above roughly 0.8 indicate strong agreement.
- Constitutional / policy-based training
- Post-training approaches that use a written set of principles or policies to generate or judge training data.
- Crescendo attack
- A multi-turn jailbreak that escalates gradually from innocuous requests towards a prohibited goal, exploiting conversational momentum.
- Data exfiltration
- Unauthorised extraction of data from a system. In LLM apps it often happens through outputs, rendered URLs, markdown images or tool arguments.
- DPO (Direct Preference Optimization)
- A post-training method that optimises a model directly on preference pairs without training a separate reward model.
- Eval suite
- A versioned collection of test cases, graders and thresholds used to measure an AI system's behaviour. See LLM evaluation guide.
- Excessive agency
- A vulnerability class in which an AI system has more permissions, autonomy or functionality than its task requires, so manipulation leads to harmful actions. OWASP LLM06.
- False-positive rate (guardrails)
- The share of benign requests a guardrail wrongly blocks or modifies. Must be measured alongside attack success rate.
- Gold set
- A sample labelled by experts that serves as ground truth for calibrating graders and auditing annotation quality.
- Grounding / faithfulness
- Whether a model's claims are supported by the sources it was given, typically retrieved documents.
- Hallucination
- Model output that is fluent but unsupported or false, such as fabricated facts or citations.
- Harmful compliance
- When a model fulfils a request it should have declined under policy. The counterpart of over-refusal.
- Human-in-the-loop
- Routing certain decisions, such as high-impact tool calls or low-confidence grades, to a person for approval or review.
- Indirect prompt injection
- Prompt injection delivered through content the system reads, such as web pages, emails, documents or tool output, rather than typed by the user. See prompt injection guide.
- Inter-annotator agreement
- How consistently different annotators label the same items. Low agreement usually means the policy or task is unclear.
- Jailbreak
- An input crafted to make a model bypass its safety training or content policy.
- Krippendorff's alpha (α)
- An agreement statistic that works for any number of raters and missing data. Commonly used to report annotation quality.
- LLM-as-a-judge
- Using a language model to grade another model's outputs against a rubric. Reliable only per category and after calibration against human labels.
- MCP (Model Context Protocol)
- An open protocol for connecting AI applications to tools and data sources. MCP servers expand what agents can do, and what guardrails must cover.
- Many-shot jailbreak
- An attack that fills a long context window with fabricated examples of the model complying, to induce compliance with a final harmful request.
- MITRE ATLAS
- A knowledge base of adversary tactics and techniques against AI systems, modelled on MITRE ATT&CK.
- Online evaluation
- Scoring a sample of live production traffic with the same graders used offline, to detect drift and new failure modes.
- Over-refusal
- When a model declines a legitimate request because it superficially resembles a harmful one.
- OWASP Top 10 for LLM Applications
- A widely used list of the most critical security risks for LLM applications, including prompt injection (LLM01) and excessive agency (LLM06).
- Policy as code
- Defining AI guardrail rules in versioned, reviewable configuration files rather than in a vendor console.
- Preference data
- Pairs or rankings of model responses judged by humans, used to train reward models or for preference optimisation.
- Prompt injection
- An attack in which supplied text causes a model to ignore or override its developer's instructions. OWASP LLM01.
- Provenance tracking
- Labelling every piece of content in a model's context with its source (user, system, retrieval, tool) so policies can treat instructions from untrusted sources differently.
- Regression
- A drop in an eval score between versions that is larger than sampling noise.
- Reward model
- A model trained on preference data to score responses, used to guide reinforcement learning from human feedback (RLHF).
- RLHF
- Reinforcement learning from human feedback: post-training a model to maximise a reward model learned from human preferences.
- Safety training data
- Labelled data used to teach models safe behaviour, including red-team corpora, preference pairs and refusal calibration sets. See buyer's guide.
- System prompt leakage
- Disclosure of a system's hidden instructions, which can reveal business logic, tool names or secrets. OWASP LLM07.
- Tool-call policy
- Rules enforced outside the model on the name and arguments of each tool call, such as spend limits, recipient allow-lists and approval requirements.
- Zero retention
- A deployment mode in which request and response payloads are not stored after processing; only decisions and metadata are logged.
Next step
Find out what your model does before your users do.
A 45-minute briefing with our evaluation team. Bring a use case; leave with a threat model, a draft eval plan and a sense of where you stand.