Evaluation lab · San Francisco · London · Zürich
Reinforced Labs Request a briefing
Glossary

Outer and inner alignment

Also called: outer alignment, inner alignment, mesa-optimization, goal misgeneralization

Outer alignment asks whether the training objective captures what we want; inner alignment asks whether the trained model actually pursues that objective.

Outer alignment is the problem of choosing a training objective, such as a reward function, that truly captures what the designer wants. Inner alignment is the problem of making sure the model that training produces actually pursues that objective, rather than some other goal that happened to score well on the training data.

Where the terms come from

Hubinger et al. (2019) introduced the distinction while analysing mesa-optimization: the possibility that a trained model is itself an optimiser with its own internal objective (the mesa-objective), distinct from the base objective used by the training process. Outer alignment is about the gap between intent and base objective; inner alignment is about the gap between base objective and mesa-objective. The broader, less mechanistic term for an inner failure is goal misgeneralization: the model is capable on new inputs but competently pursues the wrong goal.

Worked example: the coin that moved

Langosco et al. (2021) trained agents on CoinRun, a platform game where the reward is for reaching a coin that always sits at the right-hand end of the level. The reward was correct in training (outer alignment holds: you get reward for the coin). At test time they moved the coin elsewhere. Agents frequently ignored it and ran to the right-hand end anyway. In training, "go to the coin" and "go right" were indistinguishable; the agent learned the second. That is an inner alignment failure: the specification was right, the learned goal was not. Shah et al. (2022) collected further examples, including in language models, under the title "why correct specifications aren't enough".

Telling them apart

  • Outer failure: the policy does exactly what the reward asked, and that turns out to be bad. Example: specification gaming and reward hacking.
  • Inner failure: the reward was fine on the training distribution, but the policy's learned goal diverges off-distribution. Detected mainly by testing under distribution shift.

The split is a conceptual tool, not a clean partition. Some researchers argue it obscures the fact that any objective is only defined on the data you train on, so the two problems blur together in practice.

Why it matters

The two gaps call for different remedies. Outer alignment is attacked with better reward design, preference data and scalable oversight. Inner alignment needs diverse training environments, evaluations that deliberately break spurious correlations, and interpretability tools that look at what the model actually represents. A model that passes every in-distribution eval tells you little about the inner problem; the CoinRun agent would have passed all of them.

Sources

  1. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, Scott Garrabrant. Risks from Learned Optimization in Advanced Machine Learning Systems, 2019. arxiv.org
  2. Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau, David Krueger. Goal Misgeneralization in Deep Reinforcement Learning, 2021. arxiv.org
  3. Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, Zac Kenton. Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals, 2022. arxiv.org

Know the number before you ship.

A 45-minute working session with an evaluation engineer. Bring one use case; leave with a threat model, a draft eval plan with sample sizes, and an honest read on where you stand.

Request a briefing →Read our methods