AI safety is the field concerned with preventing AI systems from causing harm, whether through mistakes, misuse, adversarial attack or pursuing goals their developers did not intend. It spans research on how models learn their objectives, engineering of guardrails and monitoring, and evaluation that measures how often things go wrong.
What the field covers
Hendrycks et al. (2021) group open problems into four areas, a useful map:
- Robustness: holding up under distribution shift and deliberate attack; see adversarial robustness and jailbreaks.
- Monitoring: detecting anomalies, hidden capabilities and misuse in deployment, including dangerous capability evaluations.
- Alignment: making systems pursue intended goals; see AI alignment.
- Systemic safety: how AI interacts with the institutions and infrastructure around it.
Earlier, Amodei et al. (2016) framed five concrete technical problems for learning systems: avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration and robustness to distributional shift.
Worked example: why rare failures are hard to rule out
Illustrative numbers: a deployed assistant handles one million requests a day. If it produces a seriously harmful output on 1 in 10,000 requests, that is about 100 harmful outputs a day, even though a 500-item test set would most likely show zero failures. To claim with 95% confidence that the rate is below after observing no failures, the "rule of three" says you need about
clean samples, so about 30,000 for . This is why safety work leans on targeted red teaming, large adversarial test sets and live online evaluation, not just a benchmark score.
Safety in practice
For a team shipping a model or agent, safety usually means: a written policy of what counts as harm, an eval suite that measures both harmful compliance and over-refusal, red teaming before release, guardrails and tool-call limits at runtime, and monitoring afterwards. Each layer catches failures the others miss.
Why it matters
AI systems are now given tools, data access and autonomy, so a failure is no longer only a bad sentence but a wrong transaction, a leaked record or an unsafe instruction followed. Treating safety as a measurable engineering property, with error bars, is what lets organisations decide whether a system is ready and track whether it stays that way.
Related terms
- AI alignment
- AI red teaming
- AI guardrails
- Dangerous capability evaluation
- Safe exploration
- Reward hacking