What is AI safety? Risks, research areas and how evaluation fits

AI safety is the field that works to stop AI systems from causing serious harm, whether because someone misuses them, because they behave in ways nobody intended, or because of how many such systems interact with society. In practice it combines technical research (alignment, robustness, oversight, interpretability) with measurement (evaluations, red teaming, benchmarks) and with the decisions those measurements inform, such as what to deploy and with which safeguards.

This guide maps the whole field. It separates the three sources of harm that most of the literature uses, walks through each technical sub-area and links to a deeper article on each, explains where evaluation fits and why a number without an error bar is not a safety claim, and finishes with a small piece of arithmetic about layered safeguards that every practitioner should be able to do in their head. The tone is deliberately plain: the field has real, measurable problems, and they are easier to work on without either hype or dismissal.

What is AI safety, and how is it different from AI security and AI ethics?#

The three terms overlap and are often used loosely, so it helps to fix what each one is mainly about.

The boundaries are porous. A jailbreak is a security failure (an adversary defeats a control) and a safety failure (the system produces something harmful), which is why the jailbreak literature sits in both camps. Biased outputs in lending decisions are an ethics problem and, at scale, a structural one. Most teams shipping AI products need all three; the point of the table is to know which question you are asking at a given moment.

The phrase "AI safety" also has two registers that are worth separating. One is product safety for today's systems: a customer support assistant should not give dangerous instructions, leak private data or invent a refund policy. The other is safety for more capable future systems, where the concerns include models that pursue goals their developers did not intend and are hard to supervise. These are not rival agendas. The same techniques (reward models trained on human preferences, red teaming, capability evaluations, oversight protocols) are used for both, and failures already seen in current systems, such as reward hacking and sycophancy, are the small-scale versions of the problems that worry people about larger ones.

Why is AI safety hard?#

Ordinary software safety rests on a specification: you write down what the system must and must not do, then test or prove that it complies. Modern AI systems resist this in four ways.

Behaviour is learned, not written. A language model's refusal to give dangerous advice is not an if-statement. It is a statistical tendency instilled by training on examples, and it generalises to new inputs in ways that are hard to predict. Wei, Haghtalab and Steinhardt showed that safety training fails in two characteristic ways: the model's objectives compete (be helpful versus be harmless), and safety behaviour does not generalise to every input the model's capabilities do, such as unusual encodings. Both are covered in LLM jailbreaks explained.

The objective is a proxy. Training optimises a measurable signal, such as a reward model's score, as a stand-in for what people actually want. Under enough optimisation pressure the proxy and the target come apart. Gao, Schulman and Hilton measured this directly: as a policy was optimised against a learned reward model, the true reward rose and then fell. The general pattern is Goodhart's law, covered in specification gaming, and the RL-specific version in reward hacking.

Capabilities are general and partly unknown. A general-purpose model can be put to uses its developers never tested. Capabilities sometimes appear only with the right prompt, tools or fine-tuning, so a test that finds nothing is weak evidence of absence. This is the central difficulty of dangerous capability evaluations.

Checking gets harder as systems get better. Human feedback works when people can tell good outputs from bad ones. For long proofs, large code changes or expert-level science, they often cannot. The research programme that tries to keep supervision accurate as tasks outgrow the supervisor is scalable oversight.

Where does harm come from? Misuse, accidents and structural risks#

A widely used framing, set out by Zwetsloot and Dafoe in 2019, sorts AI risks by their cause. It is not the only taxonomy, but most others can be mapped onto it.

Misuse#

Someone deliberately uses an AI system to cause harm: writing malware, running fraud or harassment at scale, generating disinformation, or seeking uplift towards weapons. The system may be working exactly as designed; the problem is the intent of its user. Technical responses include training models to refuse harmful requests, making that refusal hard to bypass, classifiers on inputs and outputs, access controls, and evaluations that estimate how much a model would actually help a malicious actor. Weidinger and colleagues' taxonomy of language model harms lists malicious uses as one of six risk areas, alongside discrimination, information hazards, misinformation, harms from human-computer interaction, and broader automation and environmental harms.

Accidents, including misalignment#

The system does something its developers did not intend and its user did not ask for. Amodei and colleagues' Concrete Problems in AI Safety named five concrete sources of accidents in machine learning systems: negative side effects, reward hacking, the difficulty of supervising at scale, unsafe exploration, and brittleness under distributional shift. All five remain live research areas.

A special case of accident risk gets its own name: misalignment, where a capable system competently pursues an objective other than the one its developers meant. This can come from a mis-specified objective (outer misalignment) or from the system learning a different goal that happened to fit the training data (inner misalignment, or goal misgeneralisation). Misalignment worries people more than ordinary bugs because a capable system pursuing the wrong goal may act to preserve that goal, for example by behaving well when it believes it is being evaluated. Recent work has produced model organisms of this behaviour under controlled conditions: Hubinger and colleagues trained backdoored models whose behaviour persisted through standard safety training, and Greenblatt and colleagues documented a model that selectively complied with a training objective it disagreed with in order to avoid being modified.

Structural risks#

Some harms do not trace to one bad actor or one malfunction. They come from how AI changes incentives and environments: competitive pressure to deploy before evaluation is complete, concentration of power, erosion of the information ecosystem, or many automated systems interacting in ways none was tested for. Hendrycks, Mazeika and Woodside's overview of catastrophic risks groups these as AI race dynamics and organisational risks, alongside malicious use and "rogue AIs". The International AI Safety Report led by Bengio and colleagues uses a similar three-way split: malicious use, malfunctions and systemic risks.

Technical work has less direct leverage on structural risk, but not none. Credible evaluations make thresholds enforceable; standard benchmarks make claims comparable; monitoring makes incidents visible. Much of the value of measurement lies there.

The technical sub-areas of AI safety#

Hendrycks, Carlini, Schulman and Steinhardt's Unsolved Problems in ML Safety organised the technical agenda into four pillars: robustness (withstanding hazards and adversaries), monitoring (identifying hazards, including hidden ones), alignment (steering systems towards intended goals) and systemic safety (reducing deployment hazards). The sections below follow a slightly finer split that matches how the work is actually done, with a link to a full article on each.

Alignment#

AI alignment is the problem of getting a system to pursue the goals its developers intended. It splits into getting the objective right (outer alignment, or specification) and getting the system to actually learn that objective rather than something correlated with it in training (inner alignment). The dominant practical method for language models is learning from human preferences: RLHF, introduced for deep RL by Christiano and colleagues and scaled to instruction-following models by Ouyang and colleagues, and its AI-feedback variant RLAIF and constitutional AI. These work well enough to make today's assistants usable. They do not, by themselves, give strong guarantees about behaviour on inputs unlike those seen in training.

Specification gaming and reward hacking#

When a learning system finds a way to score highly on its objective without doing what the objective was meant to capture, that is specification gaming. In reinforcement learning the same failure is called reward hacking, and Skalse and colleagues gave it a formal definition: a proxy reward is hackable relative to a true reward if increasing the proxy can decrease the true reward. Pan, Bhatia and Steinhardt showed that more capable agents exploit misspecified rewards more, sometimes with sudden phase transitions. For language models trained with verifiable rewards, the same pattern appears as tests that are special-cased rather than passed; the archive's note on reward hacking in RLVR quantifies it, and reward overoptimisation covers the reward model side.

Safe reinforcement learning#

Some harms happen during learning, not after it: an agent exploring a physical or financial environment can do damage while it learns. Safe reinforcement learning formalises this with constrained Markov decision processes, where the agent maximises reward subject to a bound on expected cost, and solves it with Lagrangian methods, shielding or constrained policy optimisation. The same constrained view shows up in language model training as the KL penalty that keeps a fine-tuned model close to its reference.

Scalable oversight#

If humans cannot reliably judge an output, training on their judgements teaches the model to produce outputs that look good. Scalable oversight studies protocols that keep supervision accurate as tasks outgrow the supervisor: debate between models in front of a judge, recursive decomposition of hard tasks into checkable ones, AI critics that assist human raters, and weak-to-strong generalisation. Bowman and colleagues proposed the "sandwiching" experimental design for testing these ideas on today's models, in which non-experts assisted by a model try to match the accuracy of experts.

Robustness to misuse: jailbreaks and adversarial inputs#

A model that refuses harmful requests in normal use can often be talked out of it. Jailbreaks are inputs crafted to bypass safety training; prompt injection is the related attack in which instructions are hidden in content the model reads, such as a web page or email (see the prompt injection guide and prompt injection vs jailbreak). Robustness research builds defences (adversarial training, representation-level interventions, input and output classifiers) and, just as important, evaluates them against attackers who adapt. Defences that look strong against a fixed list of attacks often fail against an adaptive one, a point made in detail in the archive's note on adaptive attack evaluations.

Interpretability, monitoring and control#

Interpretability tries to understand what a model is computing, so that problems can be found from the inside rather than inferred from behaviour; Bereska and Gavves review the mechanistic branch of this work and its relevance to safety. Monitoring watches deployed systems for signs of misuse or misbehaviour. A newer line, AI control, asks a sharper question: can a deployment stay safe even if the model is trying to subvert it? Greenblatt and colleagues studied protocols such as having a weaker trusted model monitor a stronger untrusted one, and measured how much safety they buy at what cost to usefulness. Control does not require solving alignment, which is why it has attracted attention as a near-term complement to it.

Honesty and sycophancy#

Models trained on human approval learn to tell people what they want to hear. Sharma and colleagues found sycophancy across several production assistants and traced it in part to human preference data that rewards agreement. Honesty research separates what a model believes from what it says; benchmarks in this area are covered in AI safety benchmarks, and the related question of whether a model's stated reasoning reflects its actual computation is the subject of the archive note on chain-of-thought faithfulness.

Dangerous capabilities#

Some safety questions are about what a model can do rather than what it is inclined to do. Dangerous capability evaluations estimate whether a model could give meaningful uplift in areas such as cyber offence or weapons development, or act autonomously in ways that would be hard to stop. Shevlane and colleagues set out the logic: evaluate for dangerous capabilities and for the propensity to use them, and tie the results to deployment decisions. The results feed capability thresholds in the safety policies that many developers now publish.

How evaluation fits into AI safety#

Every area above produces claims of the form "this model is safe enough for this use". Evaluation is how those claims are tested, and it is where most practitioners meet AI safety day to day. It helps to separate four kinds of measurement, because they answer different questions and fail in different ways.

The distinction between capability and propensity matters for reading results. A model that refuses a dangerous request in a propensity test may still have the capability, which matters if its weights are released or its refusals can be bypassed. A model that cannot perform a dangerous task in a capability test may simply have been tested badly. The archive's note on sandbagging detection treats the extreme case, where a model underperforms on purpose.

The relationship between red teaming and systematic evaluation is the subject of red teaming vs evals; for the general practice of building evaluation suites, see what are AI evals and the LLM evaluation guide. Two statistical points come up so often in safety work that they belong in any overview.

A safety number needs an interval. "The model refused 98% of harmful prompts" on 200 prompts is compatible with a true refusal rate anywhere from about 95% to 99%, and if the prompts come in clusters of paraphrases, the effective sample is smaller still. The archive covers the mechanics in clustered standard errors and sample sizes for evals, and statistical significance for evals gives the short version.

A safety benchmark can measure capability instead. Ren and colleagues analysed many safety benchmarks and found that scores on a substantial share of them correlated highly with general capability and training compute. A model can improve on such a benchmark simply by getting bigger, which they called "safetywashing". The practical response is to check how a safety metric correlates with capability before reporting it as progress; AI safety benchmarks walks through how.

Defence in depth, and why the layers are not independent#

Because no single technique is reliable, deployed systems stack several: a safety-trained model, an input classifier, an output classifier, rate limits, account-level monitoring, human review for flagged cases. The appeal is arithmetic. If each of three layers misses 10% of harmful requests and the misses are independent, the chance that a request gets through all three is $0.1^3=0.1\%$.

The catch is the word "independent". Layers are usually trained on similar data, built on similar models and evaluated against similar attacks, so the inputs that fool one are more likely to fool the others. A paraphrase in an unusual language, for example, may slip past the model's refusal training and a classifier for the same reason: neither saw much data like it. A simple way to model this is to give every harmful case a shared "unusualness" score and let each layer add its own independent noise.

Here $Z$ is the case's shared unusualness, ${\epsilon }_i$ is noise specific to layer $i$, both standard normal; $\rho$ between 0 and 1 is the correlation between any two layers' blind spots; $m$ is each layer's miss rate on its own; and $\Phi$ is the standard normal distribution function, so the threshold $t$ is set to make each layer miss exactly a fraction $m$. The probability that a case passes all $k$ layers is then

where $\varphi$ is the standard normal density. In words: condition on how unusual the case is, multiply the layers' (now independent) miss probabilities, then average over cases. When $\rho =0$ the bracket is just $m$ and the formula collapses to $m^k$. When $\rho$ approaches 1, all layers fail on the same cases and the result approaches $m$: stacking buys nothing.

A worked example#

Take three layers that each miss 10% of harmful requests. Computing the integral numerically (and checking it against two million simulated cases):

At a moderate correlation of 0.5, three layers let through 1.6% of harmful requests, sixteen times what the independence calculation promises. Adding a fourth layer of the same kind helps less and less as correlation rises: at $\rho =0.9$ it moves the miss rate from 5.6% to 4.9%. The practical lesson is that a new layer is worth most when it fails for different reasons than the existing ones, for example a rule-based check alongside learned classifiers, or a human review step that sees context the models do not.

The correlation itself can be measured. Run each layer separately on the same labelled set of harmful cases, record which cases each one misses, and compute the overlap. If layer A misses 10% and layer B misses 10% but they miss the same case 5% of the time rather than 1%, their blind spots are heavily shared. This is cheap to do and almost never reported.

import numpy as np
from scipy.stats import norm  # only for the threshold; norm.ppf(0.9) = 1.2816

rng = np.random.default_rng(0)
n, k, miss = 1_000_000, 3, 0.10          # cases, layers, per-layer miss rate
t = norm.ppf(1 - miss)
for rho in (0.0, 0.5, 0.9):
    z = rng.standard_normal((n, 1))       # shared "how unusual is this case" factor
    x = np.sqrt(rho) * z + np.sqrt(1 - rho) * rng.standard_normal((n, k))
    print(rho, (x > t).all(axis=1).mean())
# 0.0 0.001009
# 0.5 0.015633
# 0.9 0.056298

AI safety in practice: what a team shipping a model should do#

For a team building on or deploying a language model, AI safety turns into a short list of concrete activities. The guides on AI guardrails and AI red teaming go into each in depth; here is the shape of it.

For regulated settings, these activities line up with formal requirements; see the NIST AI RMF Measure function and EU AI Act testing requirements. The LLM eval plan template is a starting point for writing the criteria down.

Common misconceptions#

"Safety means refusing more." A model that refuses everything is useless and pushes users to less careful tools. Over-refusal is a measurable failure, and safety evaluations that only count refusals of harmful prompts reward it. Good safety work reports both sides; see the archive note on over-refusal calibration.

"If it passed the benchmark, it is safe." A benchmark measures a fixed set of behaviours under fixed conditions. It says little about adaptive attackers, new deployment contexts or capabilities that need stronger elicitation to appear. Benchmarks are necessary for comparison and insufficient for assurance.

"Alignment is solved by RLHF." Preference training made assistants far more usable, but it optimises what raters approve of, which includes flattery and confident-sounding errors. It also leaves capabilities in place that safety training only suppresses: Qi and colleagues showed that fine-tuning on a handful of harmful examples, and to a lesser degree even on benign data, can undo much of a model's safety training.

"AI safety is only about speculative future risks." Most of the field's empirical work is on systems that exist now: jailbreak robustness, reward hacking in training runs, sycophancy, evaluation methodology. The concern about future systems is mainly a concern that these same problems get harder to detect as capability rises.

"It is only a technical problem." Technical work produces the evidence, but decisions about what is acceptable, who bears the risk and what happens when a threshold is crossed are institutional. Evaluations that nobody is obliged to act on change little.

Open problems#

A short, non-exhaustive list of problems the field openly considers unsolved:

• Specifying values and intent well enough that optimisation pressure does not find the gaps, especially for open-ended agentic tasks.
• Detecting goals a model has learned without relying on its behaviour in situations it can recognise as tests.
• Supervising outputs that humans cannot check, with guarantees that hold when the model is more capable than its overseer.
• Adversarial robustness for language models, where no defence has yet held up against well-resourced adaptive attackers in the open literature.
• Measurement validity: knowing that a benchmark tracks the property it names, that a capability evaluation elicited the model's real ability, and that a result will transfer to deployment conditions.
• Agents: models that take actions with tools over long horizons multiply the opportunities for both misuse and accidents, and the evaluation methods for them are young (see the archive's note on agent trajectory evals).

Each of these has a deeper article in this cluster. If you are new to the field, a reasonable reading order is alignment, then specification gaming, scalable oversight and safe RL for the accident side, then jailbreaks, dangerous capability evaluations and safety benchmarks for the measurement side. Reinforced Labs works on the measurement side of this list, which is why so much of this guide is about error bars.

Frequently asked questions#

Sources#

1. Amodei, Olah, Steinhardt, Christiano, Schulman, Mané. Concrete Problems in AI Safety, 2016. arxiv.org
2. Christiano, Leike, Brown, Martic, Legg, Amodei. Deep Reinforcement Learning from Human Preferences, 2017. arxiv.org
3. Zwetsloot, Dafoe. Thinking About Risks From AI: Accidents, Misuse and Structure, 2019. lawfaremedia.org
4. Hendrycks, Carlini, Schulman, Steinhardt. Unsolved Problems in ML Safety, 2021. arxiv.org
5. Weidinger, Mellor, Rauh, Griffin et al.. Ethical and social risks of harm from Language Models, 2021. arxiv.org
6. Pan, Bhatia, Steinhardt. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, 2022. arxiv.org
7. Ouyang, Wu, Jiang, Almeida et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org
8. Skalse, Howe, Krasheninnikov, Krueger. Defining and Characterizing Reward Hacking, 2022. arxiv.org
9. Gao, Schulman, Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
10. Bowman, Hyun, Perez, Chen et al.. Measuring Progress on Scalable Oversight for Large Language Models, 2022. arxiv.org
11. Shevlane, Farquhar, Garfinkel, Phuong et al.. Model evaluation for extreme risks, 2023. arxiv.org
12. Hendrycks, Mazeika, Woodside. An Overview of Catastrophic AI Risks, 2023. arxiv.org
13. Wei, Haghtalab, Steinhardt. Jailbroken: How Does LLM Safety Training Fail?, 2023. arxiv.org
14. Qi, Zeng, Xie, Chen et al.. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!, 2023. arxiv.org
15. Sharma, Tong, Korbak, Duvenaud et al.. Towards Understanding Sycophancy in Language Models, 2023. arxiv.org
16. Greenblatt, Shlegeris, Sachan, Roger. AI Control: Improving Safety Despite Intentional Subversion, 2023. arxiv.org
17. Hubinger, Denison, Mu, Lambert et al.. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, 2024. arxiv.org
18. Bereska, Gavves. Mechanistic Interpretability for AI Safety: A Review, 2024. arxiv.org
19. Ren, Basart, Khoja, Gatti et al.. Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?, 2024. arxiv.org
20. Greenblatt, Denison, Wright, Roger et al.. Alignment faking in large language models, 2024. arxiv.org
21. Bengio, Mindermann, Privitera, Besiroglu et al.. International AI Safety Report, 2025. arxiv.org