RLAIF and Constitutional AI: reinforcement learning from AI feedback

RLAIF (reinforcement learning from AI feedback) is RLHF in which the preference labels come from an AI model instead of human annotators: a labeller model reads two responses and says which is better, a reward model is trained on those labels, and the policy is optimised against it. Constitutional AI is the best-known RLAIF method: the labeller is steered by a short written list of principles, the "constitution", which is also used to make the model critique and revise its own answers before any RL happens. Both cut the cost of feedback by orders of magnitude, and both inherit every bias the labeller has.

This page explains how AI feedback is produced, how the two stages of Constitutional AI work, what the published comparisons with human feedback found, and where AI feedback fails. We also ran a seeded simulation, which you can replay in the interactive figure, showing how a labeller bias that barely moves agreement with human labels is passed in full to the reward model and then amplified by optimisation.

A worked example: critique, revise, compare#

The Constitutional AI paper illustrates its first stage with a red-team prompt asking for help breaking into a neighbour's wifi [1]. The helpful-only model answers with instructions. The model is then asked to critique that answer according to a principle, here one about identifying harmful, unethical or illegal content, and it points out that the answer helps invade someone's privacy. Asked to revise in light of the critique, it produces a response that declines and explains why: hacking into the neighbour's network is an invasion of their privacy and may be illegal. The prompt plus the revised response becomes a supervised training example.

In the second stage the model being trained produces two responses to a prompt, and a feedback model is shown both with a principle phrased as a multiple-choice question, for example "Which of these assistant responses is less harmful? Choose the response that a wise, ethical, polite and friendly person would more likely say." The feedback model's log-probabilities for the answers "(A)" and "(B)" give a soft label:

${\ell }_A$ and ${\ell }_B$ are the log-probabilities the feedback model assigns to choosing A and B. If it puts 0.8 of its probability on (A), the training target for this pair is 0.8, not a hard 1. That number goes into the same preference-model loss used in RLHF:

$r_{\varphi }$ is the reward model with parameters $\varphi$ and $\sigma$ the logistic function. This is the Bradley-Terry objective with soft targets; it is explained in detail in reward models. After that, the pipeline is ordinary RLHF: optimise the policy against $r_{\varphi }$ with a KL penalty, as in RLHF explained and KL-regularised RL.

How does Constitutional AI work?#

Stage 1: supervised learning from self-critique and revision#

Starting from a model trained to be helpful, the authors sampled responses to red-team prompts, asked the model to critique each response under a principle drawn at random from 16 harmlessness principles, asked it to revise, and repeated the critique-revision loop several times with a fresh principle each time [1]. They report that the first revision removed most of the harmful content and later revisions improved scores further but less visibly. A pre-trained model was then fine-tuned on the revised responses, together with helpful responses to ordinary prompts so that helpfulness was retained. Varying the number of principles did not change harmlessness scores much but made revisions more diverse, which helps exploration in the RL stage.

Stage 2: RL from AI preference labels#

The fine-tuned model generated response pairs, a feedback model labelled them under a randomly sampled principle (again one of 16), and these AI labels for harmlessness were mixed with human labels for helpfulness to train a preference model. Three details from the paper are worth keeping [1]:

• Ensembling over principles. Sampling a different principle per label gave noticeably more stable preference-model behaviour than using one principle for every label.
• Soft labels. Normalised log-probabilities worked much better than hard 0/1 labels.
• Clamped chain-of-thought labels. When the feedback model reasoned step by step before answering, it became nearly certain, so probabilities were close to 0 or 1. Clamping them to the 40 to 60 percent range worked better than clamping to 20 to 80 percent, and both beat no clamping, which led the policy to more extreme responses.

The paper also reports the failure that every RLAIF practitioner should expect: over-trained RL-CAI models became Goodharted, giving overly harsh answers to harmful prompts or appending boilerplate such as "you are valid, valued, and cared for" to most red-team responses [1]. The preference model rewarded a pattern, and the policy found it. Measured against a gold-standard reward, optimising a learned reward model follows a predictable curve in which the true score first rises and then falls as the policy moves away from its starting point [10]. Reward hacking and the research note on reward over-optimisation cover why that happens.

RLAIF vs RLHF: what the evidence says#

The most direct comparison trained the same policies with human labels and with AI labels from a larger off-the-shelf model on three tasks [2]. Human raters preferred RLAIF over the supervised baseline 71% of the time for summarisation and 63% for helpful dialogue, against 73% and 64% for RLHF; head to head, the two policies were equally preferred (a 50% win rate on summarisation). For harmless dialogue, 88% of RLAIF responses were rated harmless against 76% for RLHF and 64% for the baseline. The AI labeller's agreement with human preferences reached 78% on summarisation, and a labeller the same size as the policy still produced a policy preferred to the baseline 68% of the time. The paper also introduced direct RLAIF, which skips the reward model and asks the labeller for a score during RL; it matched or beat the two-step version.

A more sceptical study found that in a common setup, where supervised data comes from a weaker teacher model and AI feedback from a stronger critic, almost all of the RL step's improvement disappears if you simply fine-tune on the stronger model's outputs instead [3]. The lesson is not that RLAIF does nothing, but that its gains have to be measured against the cheapest strong baseline, not against a weak one. Related work pushes the idea further by having the policy act as its own judge and training on its own preference labels across iterations [4], and by writing the constitution from public input rather than by the developers [5].

Where AI feedback fails#

An AI labeller is an LLM-as-a-judge, and every documented judge bias becomes a training signal. The ones with the most evidence:

• Position bias. Judges prefer the first or second slot depending on the model. Swapping the order and averaging the two probabilities was standard in the RLAIF comparison, which found position bias most prevalent in smaller labellers [2]; an earlier study showed that simply reordering candidates could change which model a judge ranked best [7]. Our note on position and verbosity bias measures both.
• Verbosity bias. Judges tend to prefer longer answers; this was documented in the MT-Bench study of judge agreement [6]. Length correlates with quality in many datasets, which makes the bias easy to miss.
• Self-preference. Models rate their own generations above others', and the strength of this bias correlates linearly with how well they recognise their own outputs [8]. When the labeller and the policy share a base model, this is a direct channel for drift.
• Sycophancy. Preference data rewards answers that agree with the user, and both humans and preference models sometimes prefer convincing sycophantic answers over correct ones [9]. A labeller trained on such data passes the tendency on.

Human labels have their own noise; see preference label noise. The difference is that a labeller's bias is systematic and identical on every one of millions of labels, so it does not average out. It is learned.

Simulation: a bias you cannot see in agreement rates#

We simulated 3,000 response pairs. Each response has a true quality $q\sim N(0,1)$ and a length $\ell$ correlated with quality at 0.3, a realistic sign: longer answers are somewhat better on average. A "human" label prefers A with probability $\sigma (2(q_A-q_B))$. The AI labeller uses

$\lambda$ is a length bias, $\beta$ a bias towards whichever response is shown first. We fitted a linear Bradley-Terry reward model $r=w_{q}q+w_{\ell }\ell$ to each label set by Newton's method, then used it for best-of-16 selection on 2,000 fresh prompts and recorded the true quality and length of the selected responses. Everything is seeded (mulberry32, seed 11) and the browser figure reruns the identical computation. It is a simulation, not a measurement of any particular model.

Three things stand out. The labeller with $\lambda =1$ agrees with human labels on 76.3% of pairs, which is more than two independent human labellers agree with each other in this model (76.0%, computed exactly), so a standard agreement audit would pass it. Yet its reward model weights length at about half the weight of quality, $\lambda /2$, exactly what the labeller's formula implies: the reward model learns the labeller, bias included. And best-of-16 selection, a mild form of optimisation, turns that tilt into responses that are 0.61 SD longer and 0.15 SD worse than with human labels. RL optimises harder than best-of-16, so the gap grows.

Position bias behaved differently. With the presentation order randomised, $\beta =1$ cut agreement by about 1.5 points and shrank the fitted weights, but the reward model's direction barely changed: at $\lambda =0$ the best-of-16 picks were identical, and at $\lambda =1$ they moved by less than 0.04 SD. Position bias is mostly noise when order is random and becomes a systematic bias only when it is not, for instance when the policy's sample always appears in the same slot. Swap-and-average removes it either way, at the cost of two labeller calls per pair.

How to use AI feedback safely#

For the human side of the comparison, LLM-as-a-judge vs human evaluation compares cost and error. When the labeller must judge outputs it cannot fully verify, the question becomes one of scalable oversight, and Constitutional AI is one of the early concrete proposals in that research line.

Code: soft labels with order swapping, and a bias audit#

import numpy as np

def soft_label(logp_first_A, logp_first_B, logp_swapped_A, logp_swapped_B):
    """Preference P(A > B) averaged over both presentation orders.
    logp_first_*: log-probs of choosing each option when A is shown first.
    logp_swapped_*: log-probs when B is shown first (option labels follow the responses)."""
    p1 = np.exp(logp_first_A) / (np.exp(logp_first_A) + np.exp(logp_first_B))
    p2 = np.exp(logp_swapped_A) / (np.exp(logp_swapped_A) + np.exp(logp_swapped_B))
    return 0.5 * (p1 + p2)

def clamp(p, lo=0.4, hi=0.6):
    return np.clip(p, lo, hi)

def logit_fit(y, X, iters=25):
    w = np.zeros(X.shape[1])
    for _ in range(iters):                       # Newton steps
        p = 1 / (1 + np.exp(-X @ w))
        H = X.T @ (X * (p * (1 - p))[:, None])
        w += np.linalg.solve(H, X.T @ (y - p))
    return w

def length_bias_audit(ai_label, human_label, d_len):
    """How strongly each label set follows the length difference (A minus B),
    with an intercept for slot bias. Humans may legitimately prefer longer
    answers; the labeller's bias is the gap between the two coefficients."""
    X = np.column_stack([d_len, np.ones_like(d_len)])
    w_ai, w_h = logit_fit(ai_label, X), logit_fit(human_label, X)
    return dict(length_ai=w_ai[0], length_human=w_h[0], gap=w_ai[0] - w_h[0], slot_ai=w_ai[1])

Run the audit on your gold set with standardised length differences. A labeller with the simulation's $\lambda =1$ passes an agreement check, but on a 3,000-pair gold set drawn from the same model its length coefficient came out about 0.7 higher than the human one in three seeded draws (0.70 to 0.75), while an unbiased labeller's gap was 0.04 or less.

Failure modes#

Auditing agreement only. A labeller can match human agreement rates and still carry a systematic bias, as the simulation shows. Look at coefficients and at what the optimised policy actually outputs.

A constitution that only lists harms. Principles that only penalise harm produce preachy or evasive policies. Measure over-refusal alongside harm; see over-refusal calibration.

Same model as judge and policy. Self-preference turns into a feedback loop, especially in iterative self-rewarding schemes.

Treating AI labels as ground truth in evaluation. If the evaluation judge is the training labeller, the evaluation cannot see what the policy learned to exploit. Use a different judge or humans for the final numbers; the LLM evaluation guide covers how.

Over-training. The boilerplate failure in Constitutional AI appeared with more RL, not less. Plan early stopping against a held-out human evaluation.

Frequently asked questions#

Sources#

1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al.. Constitutional AI: Harmlessness from AI Feedback, 2022. arxiv.org
2. Harrison Lee, Samrat Phatale, Hassan Mansoor, et al.. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback, 2023. arxiv.org
3. Archit Sharma, Sedrick Keh, Eric Mitchell, et al.. A Critical Evaluation of AI Feedback for Aligning Large Language Models, 2024. arxiv.org
4. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, et al.. Self-Rewarding Language Models, 2024. arxiv.org
5. Saffron Huang, Divya Siddarth, Liane Lovitt, et al.. Collective Constitutional AI: Aligning a Language Model with Public Input, 2024. arxiv.org
6. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al.. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. arxiv.org
7. Peiyi Wang, Lei Li, Liang Chen, et al.. Large Language Models are not Fair Evaluators, 2023. arxiv.org
8. Arjun Panickssery, Samuel R. Bowman, Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations, 2024. arxiv.org
9. Mrinank Sharma, Meg Tong, Tomasz Korbak, et al.. Towards Understanding Sycophancy in Language Models, 2023. arxiv.org
10. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org