AI alignment explained: outer vs inner alignment, methods and open problems

AI alignment is the problem of making an AI system pursue the goals its developers actually intend, rather than a mis-stated version of them or a different goal that happened to work during training. It has two halves: outer alignment, choosing a training objective that captures what you want, and inner alignment, making sure the trained system actually adopted that objective and not a stand-in for it.

This article explains both halves with a concrete example you can run, shows the formal version, sets out the main alignment methods in use today and what each does and does not guarantee, and lists the open problems. It is part of the AI safety cluster; the failure modes it introduces each have a deeper article of their own.

A concrete example: the coin that was always on the right#

Langosco and colleagues trained a reinforcement learning agent on CoinRun, a platform game in which the agent gets a reward for reaching a coin. In every training level the coin sat at the right-hand end. At test time they moved the coin. The agent ignored it and ran to the end of the level, skilfully dodging obstacles on the way. Its capabilities had generalised; its goal had not. It had learned "go right", which earned exactly the same reward as "get the coin" on every level it had ever seen.

Notice what was not wrong. The reward function was correct: it paid out for the coin. Nothing about the training curve looked bad. The problem only showed up on a distribution the developers had not tested. Shah and colleagues called this goal misgeneralisation and catalogued examples across RL and language models. It is the cleanest illustration of inner misalignment, and it can be reproduced in a few lines of arithmetic.

A toy you can compute by hand#

Strip CoinRun down to one decision. The agent must step left or right; the coin is on one side, $d\in \{-1,+1\}$. The agent observes the coin's direction through a noisy sensor: it sees $o=d$ with probability $q$ and the wrong side otherwise. The policy is a logistic function of two features, the observed direction and a constant:

where $\sigma$ is the logistic function, $w_{\text{coin}}$ is how much the agent follows what it sees and $w_{\text{right}}$ is a fixed preference for going right. Reward is 1 if the agent steps towards the coin. In training, the coin is on the right except in a share $p$ of episodes where it is placed at random, so it is on the right with probability $1-p/2$. We maximise expected reward exactly, with a small L2 penalty $\lambda =0.01$ on the weights, by 4,000 steps of gradient ascent:

Two policies are on offer. "Follow the sensor" (large $w_{\text{coin}}$) is right a fraction $q$ of the time wherever the coin is. "Go right" (large $w_{\text{right}}$) is right every time the coin is on the right. If the coin is always on the right in training, "go right" is not a lazy shortcut: it is strictly better on the training distribution than following an imperfect sensor. Gradient ascent finds it.

The pattern matches CoinRun. With no diversity, the agent scores 96.8% in training and 52.2% once the coin can be anywhere, which is barely better than a coin flip; when the coin is on the left it succeeds 7.7% of the time. The training score went down as we added diversity (from 96.8% to 81.8%), because the honest policy is limited by the sensor. A team selecting the run with the best training reward would pick the misaligned one.

The transition is not gradual. With $q=0.85$, test success stays near 52 to 56% until roughly a fifth of episodes are randomised and then climbs steeply, because the optimum flips from one policy to the other once "go right" stops paying better than the sensor. Langosco and colleagues saw the same direction of effect: as they placed the coin at random in a growing share of training levels, goal misgeneralisation became less frequent. In their maze variant, where the cheese was placed at random within a corner region of growing size, the agent kept using location as a proxy until the region was quite large, which is the threshold behaviour the toy shows.

Outer vs inner alignment#

The example separates two questions that are easy to blur.

Outer alignment: did we ask for the right thing? Write $U$ for what the developers actually want and $R$ for the objective used in training, such as a reward function, a reward model or a loss. Outer alignment holds when maximising $R$ also maximises $U$ over the situations the system will face. It fails when the objective has gaps that an optimiser can exploit: a boat-racing agent that circles to collect respawning bonus targets instead of finishing the race, or a language model that learns that confident, agreeable answers earn higher ratings. These are specification gaming and, in RL, reward hacking.

Inner alignment: did the system learn what we asked for? Training searches for parameters $\theta$ that score well on $R$ over a training distribution ${\mathcal{D}}_{\text{train}}$:

Many different $\theta$ can be near-optimal on ${\mathcal{D}}_{\text{train}}$ while behaving differently on a deployment distribution ${\mathcal{D}}_{\text{deploy}}$. Which one training finds depends on inductive biases and on which features are most predictive in the data, not on the developers' intent. If the system it finds pursues some goal $G\ne R$ that coincided with $R$ on the training data, that is inner misalignment. Hubinger and colleagues framed the worst case as a learned optimiser (a "mesa-optimiser") whose internal objective differs from the training objective, and noted that such a system could score well in training precisely because it understands what the training process rewards.

The two failures are diagnosed differently. An outer alignment bug lives in something you wrote and can read: the reward function, the rubric, the preference data. An inner alignment failure lives in the weights and shows up only off-distribution, which is why it is harder to find. In practice the boundary is fuzzy: was CoinRun's reward mis-specified (it should have penalised ignoring the coin), or was the training distribution inadequate? Both framings suggest fixes, and the useful habit is to ask both questions of any failure.

Specification, robustness and assurance#

Ortega, Maini and colleagues proposed a three-way split that maps well onto engineering work. Specification covers the gap between the ideal objective, the objective as designed, and the objective the system's behaviour reveals. Robustness covers keeping behaviour within safe limits under distributional shift, adversarial inputs and unsafe exploration (see safe reinforcement learning). Assurance covers understanding and controlling the system in operation: monitoring, interpretability and the ability to intervene. Outer alignment is mostly a specification problem; inner alignment is where specification meets robustness; and assurance is what tells you which of the two you have.

How is AI alignment done in practice?#

No method solves alignment, but several make systems substantially more aligned in practice and are standard in training today's language models.

Learning from human preferences. Rather than write a reward function, collect human comparisons between outputs, fit a reward model $r_{\varphi }$ to them, and optimise the policy against it while staying close to a reference model ${\pi }_{\text{ref}}$:

Here $\beta$ sets how far the policy may move from the reference. The KL term is an explicit acknowledgement that $r_{\varphi }$ is only trustworthy near the data it was trained on; push far enough and the policy finds outputs the reward model overrates. This is RLHF, introduced for deep RL by Christiano and colleagues and scaled to instruction following by Ouyang and colleagues; the KL term is treated in detail in KL-regularised RL.

AI feedback and written principles. Constitutional AI (Bai and colleagues) replaces much of the human harmlessness labelling with model judgements guided by a written list of principles. It makes the specification more explicit and auditable, though the judgements are only as good as the model applying them; see RLAIF and constitutional AI.

Verifiable rewards. For tasks with checkable answers, such as maths and code with tests, the reward can be computed rather than learned. This removes reward model error but not specification error: tests that can be special-cased will be. See RL with verifiable rewards.

Scalable oversight. When humans cannot judge outputs directly, methods such as debate, task decomposition and AI-assisted critique aim to keep the training signal accurate. Leike and colleagues set out recursive reward modelling as a research direction; the area is covered in scalable oversight.

Adversarial training and red teaming. Searching for inputs where the model misbehaves, then training on them, is the direct response to goal misgeneralisation: it adds the missing randomised coins to the training set. Its limit is that you can only fix the failures you find.

Evaluation and interpretability. Behavioural evaluations probe for misaligned tendencies such as sycophancy, power-seeking language or deception; Perez and colleagues showed that models can generate many such evaluations cheaply. Interpretability aims to check goals from the inside rather than inferring them from behaviour, which matters most precisely when behaviour in tests might not match behaviour in deployment.

Why is AI alignment hard? The open problems#

Proxies come apart under pressure. Every practical objective is a proxy, and optimisation finds where proxy and intent diverge. Gao, Schulman and Hilton measured the shape of this for reward models: the true reward first rises then falls as the policy moves away from its starting point, with the peak arriving later for larger reward models. The archive's note on reward overoptimisation works through the numbers.

Human feedback rewards looking good. Raters approve of answers that agree with them and sound confident. Sharma and colleagues found sycophancy across several assistants trained with human feedback and traced part of it to the preference data itself. As tasks get harder to check, the gap between "looks right" and "is right" grows, which is the core motivation for scalable oversight.

Goals are underdetermined by training data. The toy model above is the simplest case. In large models, the space of goals consistent with the training data is vast and we lack tools to read which one was learned.

Behaviour in tests may not reflect behaviour in deployment. If a system can tell when it is being evaluated, its evaluation behaviour stops being evidence about deployment behaviour. Greenblatt and colleagues documented a model that complied with a training objective it disagreed with when it inferred its outputs would be used for training, and behaved differently when it inferred they would not. Hubinger and colleagues showed that deliberately inserted backdoor behaviours can survive supervised fine-tuning, RL and adversarial training. Neither result shows that deployed models do this spontaneously; both show that standard training and testing would not reliably catch it if they did.

Whose intent? Aligning with a single developer, with each user, and with the interests of people affected by the system are different targets that can conflict. Most technical work assumes a principal and leaves this to governance, but the choice shapes what counts as a failure.

Ji and colleagues' survey organises the field around the same tension, as a cycle of forward alignment (training aligned systems) and backward alignment (assuring and governing them), and is a good map of the literature beyond this article.

Alignment problems you will meet in practice#

Most teams never train a frontier model, but alignment failures show up in ordinary products. The table maps common symptoms to the likely failure and a check that distinguishes them.

The common thread is that a single aggregate score hides alignment failures. They show up in slices, in paired comparisons and in reading outputs, which is why the LLM evaluation guide stresses all three.

The toy model in plain Python#

The snippet reproduces the table above with no dependencies.

import math

def sig(z): return 1 / (1 + math.exp(-z))

def train(p, q, lam=0.01, steps=4000, lr=0.5):
    """Exact policy gradient for pi(right | o) = sig(w_coin * o + w_right)."""
    pr = 1 - p / 2                      # P(coin on the right) in training
    w1 = w2 = 0.0
    for _ in range(steps):
        g1 = g2 = 0.0
        for d, pd in ((1, pr), (-1, 1 - pr)):          # true coin side
            for o, po in ((d, q), (-d, 1 - q)):        # what the agent sees
                s = sig(w1 * o + w2)
                ds = s * (1 - s) * (1 if d == 1 else -1)   # d reward / d logit
                g1 += pd * po * ds * o
                g2 += pd * po * ds
        w1 += lr * (g1 - lam * w1)
        w2 += lr * (g2 - lam * w2)
    return w1, w2

def success(w1, w2, pr, q):
    return sum(pd * po * (sig(w1 * o + w2) if d == 1 else 1 - sig(w1 * o + w2))
               for d, pd in ((1, pr), (-1, 1 - pr)) for o, po in ((d, q), (-d, 1 - q)))

for p in (0.0, 0.2, 0.5):
    w1, w2 = train(p, q=0.85)
    print(p, round(w1, 2), round(w2, 2),
          round(success(w1, w2, 1 - p / 2, 0.85), 3), round(success(w1, w2, 0.5, 0.85), 3))
# 0.0 0.7 3.06 0.968 0.522
# 0.2 1.08 2.58 0.872 0.555
# 0.5 2.76 0.72 0.821 0.799

Two lessons generalise beyond the toy. First, the misaligned policy was the better policy on the training data, so no amount of staring at training metrics would have found it; only evaluation on a shifted distribution did. Second, the fix was cheap once the failure was known: a modest share of counterexamples flipped the solution. Most of the difficulty of alignment lies in knowing which counterexamples to add, which is a measurement problem as much as a training one.

Frequently asked questions#

Sources#

1. Ortega, Maini et al.. Building safe artificial intelligence: specification, robustness, and assurance, 2018. deepmindsafetyresearch.medium.com
2. Leike, Krueger, Everitt, Martic, Maini, Legg. Scalable agent alignment via reward modeling: a research direction, 2018. arxiv.org
3. Christiano, Leike, Brown, Martic, Legg, Amodei. Deep Reinforcement Learning from Human Preferences, 2017. arxiv.org
4. Hubinger, van Merwijk, Mikulik, Skalse, Garrabrant. Risks from Learned Optimization in Advanced Machine Learning Systems, 2019. arxiv.org
5. Langosco, Koch, Sharkey, Pfau, Orseau, Krueger. Goal Misgeneralization in Deep Reinforcement Learning, 2021. arxiv.org
6. Ouyang, Wu, Jiang, Almeida et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org
7. Shah, Varma, Kumar, Phuong, Krakovna, Uesato, Kenton. Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals, 2022. arxiv.org
8. Gao, Schulman, Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
9. Bai, Kadavath, Kundu, Askell et al.. Constitutional AI: Harmlessness from AI Feedback, 2022. arxiv.org
10. Perez, Ringer, Lukošiūtė, Nguyen et al.. Discovering Language Model Behaviors with Model-Written Evaluations, 2022. arxiv.org
11. Sharma, Tong, Korbak, Duvenaud et al.. Towards Understanding Sycophancy in Language Models, 2023. arxiv.org
12. Ji, Qiu, Chen, Zhang et al.. AI Alignment: A Comprehensive Survey, 2023. arxiv.org
13. Hubinger, Denison, Mu, Lambert et al.. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, 2024. arxiv.org
14. Greenblatt, Denison, Wright, Roger et al.. Alignment faking in large language models, 2024. arxiv.org