Constitutional AI

Constitutional AI is a post-training approach in which a short written set of principles, the "constitution", replaces most human harmlessness labels. A model critiques and revises its own responses against those principles, and then a model judges pairs of responses against them to produce preference data for reinforcement learning.

The two stages

• Supervised critique and revision. Sample a response to a (often red-team) prompt, ask the model to critique it against a randomly drawn principle, then ask it to rewrite the response to fix the problems. Repeat a few times and fine-tune on the final revisions.
• RL from AI feedback. Generate pairs of responses from the fine-tuned model, ask a feedback model which response better satisfies a sampled principle, convert its answer into a soft preference label, train a preference model and run RLHF-style optimisation against it. This step is an instance of RLAIF.

The soft label for a pair is the feedback model's normalised probability of choosing each option, often averaged over $K$ sampled principles:

where $c_k$ is the $k$-th sampled principle and $p_{\mathrm{FM}}$ is the feedback model's probability of answering "A".

Worked example

Illustrative numbers: four principles are drawn for one pair: "avoid assisting with violence", "be honest about uncertainty", "do not be preachy" and "respect privacy". The feedback model prefers response A with probabilities 0.8, 0.6, 0.7 and 0.9. The training target is their mean, 0.75, rather than a hard "A wins". The soft target keeps genuine trade-offs visible: a response can be safer on one principle and more condescending on another.

Policy-based training more broadly

The same idea generalises beyond a constitution: any written content policy can drive data generation, grading and filtering. Bai et al. (2022) reported models that were both less harmful and less evasive than a model trained on human harmlessness labels, partly because principles can ask the model to explain objections rather than refuse flatly.

Why it matters

Writing behaviour down as principles makes training targets inspectable and editable, which is useful for audits and for fixing over-refusal. The risks are that principles are only as good as their wording, that the feedback model can share the policy model's blind spots, and that it can be gamed like any reward model. Constitutional training should therefore be checked with independent human-labelled evals and red teaming.

Related terms

• RLAIF (reinforcement learning from AI feedback)
• RLHF (reinforcement learning from human feedback)
• Reward model
• Over-refusal
• Scalable oversight
• Safety training data

Sources

1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al.. Constitutional AI: Harmlessness from AI Feedback, 2022. arxiv.org