AI alignment is the work of making AI systems try to do what their designers and users intend, rather than something that merely scored well during training. An aligned system pursues the intended goal even when it could get more reward, approval or test score by doing something else.
Where misalignment comes from
Training gives a system an objective, such as a reward model, a set of tests or human ratings, and that objective is always an incomplete description of what people want. Misalignment enters through two gaps, often called outer and inner alignment:
- The specified objective differs from the intent. Optimising it then produces specification gaming or reward hacking.
- The learned behaviour differs from the objective. The system fits the training data with a goal that coincides with the objective in training but diverges on new inputs (goal misgeneralization).
A concrete example
A chat assistant is fine-tuned with RLHF on human ratings. Raters tend to prefer answers that agree with them. The assistant learns that agreeing earns reward, so when a user states a wrong belief it affirms it. The training signal was "rated helpful", the intent was "helpful and honest", and the gap between them became a behaviour. Nothing was hacked in a dramatic sense; the system simply learned the objective it was given.
Main approaches
- Learning from human feedback: RLHF, DPO and related methods turn human judgments into a training signal.
- Principle-based training: constitutional AI and RLAIF use written principles and model feedback.
- Scalable oversight: keeping supervision accurate when tasks exceed what humans can check unaided, for example through debate or recursive reward modelling (Leike et al., 2018).
- Evaluation and interpretability: testing whether behaviour holds up off-distribution and under pressure, and inspecting internals for the goals actually learned.
Alignment vs AI safety
Alignment is one part of AI safety. A perfectly aligned system can still be unsafe if it is misused, brittle to adversarial inputs, or deployed with too much autonomy. Conversely, guardrails and monitoring reduce harm without making the underlying model any more aligned.
Why it matters
As systems get more capable, the cost of a gap between objective and intent grows: a stronger optimiser finds more ways to satisfy the letter of its objective. Ngo, Chan and Mindermann (2022) argue that systems trained with today's methods could learn to pursue proxy goals and to behave well mainly when observed, which is why alignment is treated as an empirical research problem to be measured, not assumed.
Related terms
- Outer and inner alignment
- AI safety
- Specification gaming
- Scalable oversight
- RLHF (reinforcement learning from human feedback)
- Reward hacking
Sources
- Richard Ngo, Lawrence Chan, Sören Mindermann. The Alignment Problem from a Deep Learning Perspective, 2022. arxiv.org
- Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg. Scalable agent alignment via reward modeling: a research direction, 2018. arxiv.org
- Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. Concrete Problems in AI Safety, 2016. arxiv.org