AI safety
AI safety is the engineering and research discipline concerned with making AI systems do what their developers and users intend, without causing harm along the way. It spans questions as concrete as "will this support agent issue a refund it shouldn't?" and as open as "how do we supervise a system that is better than us at the task?". The common thread is measurement: a safety claim is only as good as the evidence behind it.
Start with the overview, then read alignment to see why intended and specified goals come apart. Specification gaming shows what that gap looks like in practice, with documented examples and Goodhart's law. Safe reinforcement learning covers constrained optimisation, where an agent must maximise reward while keeping costs below a limit. Scalable oversight asks how humans can keep checking work they can no longer verify directly. The last three pages move to deployed models: how jailbreaks bypass safety training, how dangerous capability evaluations are designed, and what the public safety benchmarks do and do not measure.
The guides on red teaming, prompt injection and guardrails linked below cover the security side in more depth.
What is AI safety? Risks, research areas and how evaluation fits
What AI safety is: misuse, accident and structural risks, the technical sub-areas from alignment to oversight, and how evaluation turns them into evidence.
Suggested reading order
- 1What is AI safety? Risks, research areas and how evaluation fits
What AI safety is: misuse, accident and structural risks, the technical sub-areas from alignment to oversight, and how evaluation turns them into evidence.
- 2AI alignment explained: outer vs inner alignment, methods and open problems
What AI alignment means, outer vs inner alignment with a goal misgeneralisation toy you can run, the methods used today and the problems still open.
- 3Specification gaming and Goodhart's law in AI
What specification gaming is, Goodhart's four variants, a formal toy model of when proxies collapse under optimisation, and published examples.
- 4Safe reinforcement learning: constrained MDPs, Lagrangian methods and shielding
Safe RL explained: constrained MDPs, Lagrangian primal-dual training and why it oscillates, constraints during training, shielding and safe exploration.
- 5Scalable oversight: debate, amplification, weak-to-strong generalisation and sandwiching
How to supervise AI on tasks the supervisor cannot check: debate, amplification, recursive reward modelling, critique, weak-to-strong and sandwiching.
- 6LLM jailbreaks explained: why they work, attack families and how success is measured
What an LLM jailbreak is, why safety training fails, the main attack families at a conceptual level, and why attack success rate needs a stated budget.
- 7Dangerous capability evaluations: what they measure, elicitation, thresholds and uncertainty
What dangerous capability evaluations measure, why elicitation and sandbagging dominate the error, how thresholds work, and why more tasks cannot fix bias.
- 8AI safety benchmarks: what they measure, over-refusal, validity and how to choose
What the main public AI safety benchmarks measure, why refusal needs an over-refusal pair, how capability leaks into safety scores, and how to choose.
Guides and explainers
AI red teaming: a field methodology
Pillar guide.
Prompt injection: detection and defense
Pillar guide.
AI guardrails for agents: an architecture guide
Pillar guide.
Indirect prompt injection, explained
How instructions hidden in documents, web pages and tool output hijack agents, with real research and the controls that hold.
OWASP Top 10 for LLM Applications (2025), explained for engineers
All ten risks, what each looks like in an agent, and the test and control that addresses it.
AI compliance testing: EU AI Act, NIST AI RMF and ISO/IEC 42001
What each framework actually asks you to test, mapped to concrete evaluations and evidence.
Research notes on this topic
Evaluating agents against adaptive, multi-turn attacks
Why static attack suites overstate defences, how success grows over turns, why attack budgets must be reported, and a protocol that avoids overfitting.
A 2% attack success rate is not low: retry economics of prompt injection
Cumulative compromise probability, and why correlated defence layers deliver 3–30× less protection than the independence assumption predicts.
Guardrail thresholds under low base rates
With a 0.1% attack rate, a classifier with AUC 0.96 tuned to 95% recall has 0.5% precision. Cost-optimal thresholds, worked through.
The latency budget of AI guardrails: sequential, parallel and hedged
Order statistics of parallel checks. Hedging at the single-check p95 cuts eight-check p99 from 49 ms to 35 ms for 5% extra load.
Guardrail classifiers under distribution shift
How label, covariate and attack drift move guardrail precision and recall, when EM prior correction works, why small audits miss recall, and what to monitor.
Is the chain of thought faithful? Tests, monitors and false negatives
How CoT faithfulness is tested, why reveal rates need a drift correction, how faithfulness sets a CoT monitor's miss rate, and what training against it does.
Over-refusal and under-refusal are one trade-off: measuring and calibrating both
How to measure over-refusal and harmful compliance together: borderline prompt sets, sample sizes, label noise, contrastive synthetic data and the frontier.
Attack success rate is a curve, not a number: power laws for Best-of-N jailbreaks
Why Best-of-N jailbreak success follows a power law, how to extrapolate ASR to large budgets, compare defences and bound rare successes. Seeded simulations.
Know the number before you ship.
A 45-minute working session with an evaluation engineer. Bring one use case; leave with a threat model, a draft eval plan with sample sizes, and an honest read on where you stand.