What Is RLAIF (reinforcement learning from AI feedback)?

Reinforcement learning from AI feedback (RLAIF) is RLHF with the human labeller swapped for a language model. An off-the-shelf or specially prompted model compares pairs of responses, its judgements become preference data, and the rest of the pipeline (reward model, KL-regularised RL) is unchanged.

How AI labels are produced

The labeller sees the prompt, two candidate responses and instructions describing what "better" means. Rather than taking its text answer, implementations usually read the probabilities it assigns to the tokens "1" and "2" and normalise them into a soft label:

p (y_{1} ≻ y_{2}) = \frac{e ^{ℓ_{1}}}{e ^{ℓ_{1}} + e ^{ℓ_{2}}} = σ (ℓ_{1} - ℓ_{2})

where $ℓ_{1}$ and $ℓ_{2}$ are the labeller's log-probabilities for the two answer tokens. Lee et al. (2023) also tested "direct RLAIF", which skips the reward model and asks the labeller for a score during RL.

Worked example: correcting position bias

Illustrative numbers: with response A shown first, the labeller's log-probabilities are $- 0.3$ for "A" and $- 1.5$ for "B", giving $σ (1.2) \approx 0.77$ . With the order swapped, it prefers A with probability only 0.61, because it slightly favours whichever response comes first. Averaging both orders gives a label of 0.69. Running each comparison in both orders is cheap and removes most of this bias.

RLAIF vs constitutional AI

RLAIF names the general technique: AI-generated preferences feeding RL. Constitutional AI is one specific recipe that uses RLAIF in its second stage, with principles drawn from a written constitution and an earlier critique-and-revision stage. RLAIF can just as well use a plain rubric or a task-specific grading prompt.

Limits

The labeller's biases (length, style, self-preference, position) transfer to the reward model.
A labeller cannot reliably judge tasks it cannot do itself, which is the core scalable oversight problem.
AI labels should be audited against a human-labelled sample, exactly as for LLM-as-a-judge graders.

Why it matters

Lee et al. (2023) found that human raters preferred RLAIF and RLHF summaries over the supervised baseline at similar rates. That makes AI feedback a practical way to iterate quickly, but it moves the trust question from the labellers to the labelling model and its prompt, which then need their own calibration and evals.

Sources

Harrison Lee, Samrat Phatale, Hassan Mansoor, et al.. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback, 2023. arxiv.org
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al.. Constitutional AI: Harmlessness from AI Feedback, 2022. arxiv.org