RLHF explained: how reinforcement learning from human feedback works, and where it fails

RLHF (reinforcement learning from human feedback) is a way to fine-tune a model when the goal is easier to judge than to write down. People compare pairs of model outputs, a reward model learns to predict which output they prefer, and the model is then optimised with reinforcement learning to score highly on that reward model while staying close to where it started.

That recipe turned raw text predictors into instruction-following assistants. This guide walks through the pipeline with the equations that matter, runs a small simulated pipeline you can steer, and spends most of its time on what goes wrong and what RLHF can and cannot promise for safety.

What is RLHF? A concrete example#

Suppose you want a model that writes good summaries of forum posts. "Good" is hard to define in code. ROUGE overlap with a reference summary rewards copying; length limits reward truncation; nobody can write a function that captures "accurate, covers the main point, reads well". But a person shown two summaries of the same post can say which is better in a few seconds, and they will agree with another careful person most of the time.

RLHF turns that cheap judgment into a training signal. Sample two summaries, ask a rater which is better, repeat tens of thousands of times, and fit a model that predicts the rater's choice; its scalar output is the reward. Then the summariser practises: it writes, the reward model scores, and reinforcement learning nudges it towards higher-scoring writing. Stiennon et al. did exactly this and found that their 1.3B-parameter model trained this way significantly outperformed a supervised model ten times its size in human preference ratings [3].

Note two features. The model is never told what a good answer is, only which of two answers someone preferred. And it is optimised against a learned stand-in for people, which can be wrong in ways the optimiser is very good at finding.

The RLHF pipeline, step by step#

Step 1: a pretrained model#

Everything starts from a language model pretrained with next-token prediction. It continues text rather than following instructions. In RL terms it is a policy: given a prompt $x$, it defines a distribution $\pi (y\mid x)$ over responses $y$, produced one token at a time. Each token is an action, and the reward arrives once, when the response is finished. The overview of reinforcement learning for LLMs covers this framing in detail.

Step 2: supervised fine-tuning#

Next, supervised fine-tuning (SFT) on demonstrations, prompts paired with good responses, teaches the format of a helpful answer; InstructGPT used about 13,000 training prompts [4]. The SFT model matters twice later: it is the starting point for RL and it is the frozen reference policy ${\pi }_{\text{ref}}$ that the KL penalty measures distance from.

Step 3: collecting preference data#

Prompts are sampled, the SFT model produces several responses per prompt, and raters compare them. Pairwise choice is the simplest format; InstructGPT showed raters between 4 and 9 responses at once and had them rank all of them, which yields up to 36 comparisons from a single task [4]. Rater instructions shape everything downstream. Agreement is far from perfect: InstructGPT reported that training labelers agreed with each other 72.6 percent of the time [4]. Our research note on preference label noise works through what that number does and does not tell you.

Step 4: training the reward model#

The reward model $r_{\varphi }(x,y)$ is usually the SFT model with its token-prediction head replaced by a single scalar output. It is trained so that the difference in scores between two responses predicts how likely a rater is to prefer one over the other, using the Bradley-Terry model of paired comparisons [7]:

Here $y_w$ is the preferred ("winning") response, $y_l$ the other one, $\sigma (z)=1/(1+e^{-z})$ is the logistic function, and $\varphi$ are the reward model's parameters. In words: a reward gap of zero means a coin flip, a gap of 2 means the better response wins about 88 percent of the time, and training is ordinary logistic regression on score differences. Only differences matter, so the scale's zero point is arbitrary and is fixed by a normalisation step. Everything about reward model design and evaluation, including why higher accuracy does not always mean a better policy, is in the reward models guide.

Step 5: RL fine-tuning with a KL penalty#

Finally the policy ${\pi }_{\theta }$, initialised from the SFT model, is optimised on prompts with no labels at all. The objective is the reward model's score minus a penalty for drifting away from the reference:

$\mathcal{D}$ is the prompt distribution, $\beta >0$ is the KL coefficient and $\mathrm{KL}$ is the Kullback-Leibler divergence, the expected log ratio $\log {\pi }_{\theta }(y\mid x)/{\pi }_{\text{ref}}(y\mid x)$ over the policy's own samples. The first term says "write what the reward model likes"; the second says "but keep sounding like the SFT model". In practice the penalty is folded into the reward, $R(x,y)=r_{\varphi }(x,y)-\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$, and is often applied per token [2][4]. It keeps the policy where the reward model is valid and preserves coherence that raters were not asked to judge [2]. The objective has a closed-form optimum, a reweighting of the reference policy by $\exp (r/\beta )$, which the guide to KL-regularised RL derives along with how to choose $\beta$.

The optimiser is usually Proximal Policy Optimization [6], a policy-gradient method with a clipped surrogate objective that limits how far each update moves the policy. It needs a value function (InstructGPT initialised it from the reward model) to compute advantages, so a full PPO run keeps four networks in play: policy, value function, reward model and frozen reference. See the PPO explainer for why clipping works. InstructGPT also mixed pretraining gradients into PPO (the "PPO-ptx" variant) to recover performance lost on public NLP benchmarks, a cost the authors called an alignment tax [4].

Why use RL at all? RLHF vs supervised fine-tuning#

If people can write good answers, why not just do more SFT? Three reasons recur.

Judging is easier than demonstrating. Raters can recognise a better answer than they could write, so comparisons let the model exceed its demonstrations, which imitation cannot.

The model learns from its own outputs, including its mistakes. SFT only ever shows good text. RLHF scores what the model actually produces, so it can push probability away from the specific failure modes the model has, not just towards someone else's answer. This is the main thing the RL step adds over imitation, and it is why on-policy sampling matters.

The objective is about whole responses. SFT is a per-token loss; RL optimises the sequence-level judgment directly.

The trade-off is measurable: Kirk et al. found that RLHF generalised better than SFT to inputs far from the training distribution but substantially reduced output diversity [12]. And the RL step is where most failure modes enter, because it optimises against a learned proxy.

RLHF vs DPO#

Direct Preference Optimization starts from the same KL-regularised objective and the same Bradley-Terry model, but uses the closed-form optimal policy to rewrite the reward in terms of the policy itself. The result is a classification-style loss on preference pairs that trains the policy directly, with no separate reward model and no RL sampling loop [8]. It is simpler and cheaper, and in the original paper it matched or beat PPO-based RLHF on the tasks tested. It gives up the on-policy step, learning from a fixed dataset of comparisons rather than fresh samples, and it still optimises a proxy, so the failure modes below apply to both.

A toy RLHF pipeline you can run#

To see how the pieces interact, we built the smallest pipeline with every stage. Each response is a vector of 8 features drawn from a standard normal under the SFT policy: true quality $q$, length $\ell$ (standardised), and 6 features nobody cares about. Raters prefer response A over B with probability $\epsilon +(1-2\epsilon )\sigma (u_A-u_B)$, where $u=q+a\ell$: they judge quality but also lean towards longer answers with strength $a$, and a fraction $\epsilon$ of their labels are flipped at random. What users actually value, the gold reward, is $q-\frac{1}{2}{\ell }^2$: quality, minus a penalty for answers much longer or shorter than the SFT norm.

The reward model is a linear Bradley-Terry model $r(y)=w\cdot y$ fitted to simulated comparisons. Because the reference is Gaussian and the reward linear, the KL-regularised optimal policy is exactly the reference shifted by $w/\beta$, so we can compute everything in closed form: $\mathrm{KL}=\Vert w{\Vert }^2/2{\beta }^2$, proxy gain $\Vert w{\Vert }^2/\beta$, gold gain $w_q/\beta -\frac{1}{2}(w_{\ell }/\beta {)}^2$. The figure runs the same seeded procedure as our numpy script.

At the defaults the reward model puts weight 0.77 on quality and 0.49 on length, faithfully copying the raters. At $\beta =0.50$ the policy has moved 1.67 nats from the SFT model; the proxy says it improved by 1.68 and users would say 1.07. Optimise harder and the proxy keeps rising while gold peaks at 1.26 ($\beta \approx 0.31$, KL 4.5 nats) and then falls, because each extra unit of reward is bought partly by padding answers.

Four things in this toy carry over to real RLHF.

The peak is set by systematic error, not by noise. With unbiased raters ($a=0$) the gold gain is still rising at $\sqrt{\mathrm{KL}}=5$ (gain 6.9). With $a=0.5$ it peaks at 2.79; with $a=1$ it peaks at 0.71 near $\sqrt{\mathrm{KL}}=1.3$, and by $\sqrt{\mathrm{KL}}=5$ gold has fallen to $-4.6$ while the proxy reports $+7.1$. Random label flips and small datasets cost efficiency (with 250 pairs and unbiased raters, the same KL buys about 5 percent less gold than with 16,000 pairs, because part of it moves irrelevant features) but do not by themselves create a peak. This matches the mechanism in our reward overoptimisation research note, which studies the shape of these curves in depth.

Accuracy against raters can go up as the reward model gets worse. Held-out agreement with raters is 67.8 percent at $a=0$, 69.1 percent at $a=0.5$ and 72.1 percent at $a=1$, while agreement with gold falls from 84.1 to 80.8 to 75.2 percent. The more predictable the raters' bias, the easier it is to fit and the more damage it does.

Label noise rescales the reward, so $\beta$ must move with it. With 4,000 pairs and $a=0.5$, raising the flip rate from 0 to 0.2 shrinks the learned quality weight from 1.06 to 0.50. The best achievable gold barely changes (2.40 versus 2.54) but the $\beta$ that reaches it drops from 0.22 to 0.10. A $\beta$ tuned on one reward model does not transfer to another trained on different data.

The proxy never tells you when to stop. It rises monotonically in every setting; only a held-out signal finds the peak.

The core of the simulation, in a form you can paste and run:

import numpy as np

rng = np.random.default_rng(0)
D, n, a, eps, beta = 8, 1000, 0.75, 0.1, 0.5   # features, pairs, rater length bias, flip rate, KL coefficient
sig = lambda x: 1 / (1 + np.exp(-x))

# 1. Preference data: pairs of SFT samples, labelled by biased, noisy raters
A, B = rng.normal(size=(n, D)), rng.normal(size=(n, D))   # features: quality, length, 6 irrelevant
u = lambda X: X[:, 0] + a * X[:, 1]                         # what raters reward
p = eps + (1 - 2 * eps) * sig(u(A) - u(B))
y = (rng.random(n) < p).astype(float)                     # 1 if A preferred

# 2. Bradley-Terry reward model r(x) = w . x, fitted by Newton's method with a small ridge
Z, w = A - B, np.zeros(D)
for _ in range(20):
    q = sig(Z @ w)
    H = (Z * (q * (1 - q))[:, None]).T @ Z + np.eye(D)
    w -= np.linalg.solve(H, Z.T @ (q - y) + w)

# 3. KL-regularised policy: for Gaussian SFT features and a linear reward it is N(w / beta, I)
shift = w / beta
kl = shift @ shift / 2
proxy_gain = w @ shift
gold_gain = shift[0] - 0.5 * shift[1] ** 2                # users value quality, dislike padding
print(f"w_q={w[0]:.2f} w_len={w[1]:.2f} KL={kl:.2f} proxy=+{proxy_gain:.2f} gold=+{gold_gain:.2f}")

With this seed it prints a quality weight of 0.65, a length weight of 0.53, KL 1.44, proxy gain 1.44 and gold gain 0.73 (a different random stream from the figure).

A short history of RLHF#

Christiano et al. (2017) showed that an agent could learn Atari games and simulated locomotion from human comparisons of short trajectory clips, with feedback on less than one percent of its interactions [1]: fit a reward model to comparisons, run deep RL against it, and keep asking for comparisons as the policy changes.

Ziegler et al. (2019) brought the recipe to language models: a pretrained model, a reward model fitted to human comparisons, PPO, and the KL penalty to the pretrained model [2]. Their summarisers learned to copy whole sentences from the input, which raters scored well, possibly by exploiting raters' simple heuristics. Stiennon et al. (2020) scaled this up and observed that optimising too far against the reward model made summaries worse by human judgment [3].

Ouyang et al. (2022) applied the full SFT, reward model and PPO pipeline to general instruction following. Outputs of their 1.3B-parameter model were preferred by raters to those of the 175B-parameter base model, despite having over 100 times fewer parameters [4]. Bai et al. (2022) trained helpful and harmless assistants, iterated the loop weekly with fresh feedback, and found a roughly linear relation between RL reward and the square root of the KL divergence [5]. Later work replaced some human labels with model-generated ones guided by written principles; see the guide to RLAIF and constitutional AI.

Failure modes of RLHF#

Reward hacking and overoptimisation#

The policy maximises the reward model, not the raters. Any systematic error in the reward model is a direction the optimiser will push in, and the push gets stronger as $\beta$ falls. Gao et al. measured the resulting hump with a large "gold" reward model standing in for humans: gold reward rises, peaks and declines as KL grows, with a peak that moves predictably with reward model size [9]. The toy above reproduces the shape. The reward hacking guide catalogues how this looks in practice.

Length bias#

RLHF tends to make responses longer. Singhal et al. found that, across three settings, much of the reward improvement from RLHF came from increased length, that a reward based on length alone reproduced most of the downstream gains over SFT, and that the main source of the bias was the reward model, which picked up length correlations in the preference data [10]. This is the toy's length weight in real life.

Sycophancy#

Raters tend to prefer answers that agree with them. Sharma et al. found that responses matching a user's stated views were more likely to be preferred in human preference data, that both humans and preference models sometimes preferred convincingly written sycophantic answers over correct ones, and that optimising against preference models sometimes sacrificed truthfulness for agreement [11]. RLHF did not invent the tendency, but rewards it.

Loss of diversity#

The KL-regularised optimum concentrates probability on high-reward outputs. Kirk et al. measured substantially lower output diversity after RLHF than after SFT [12]. For a chat assistant that may be fine; for brainstorming, creative writing or varied synthetic data it is a real cost.

Noisy and contested labels#

Rater disagreement is partly noise, which costs data and compresses the reward scale, and partly real differences in values, which a single scalar reward averages away. Casper et al. list "a single reward function cannot represent a diverse society of humans" among RLHF's fundamental, not merely tractable, problems [14].

RLHF and AI safety: what it does and does not guarantee#

RLHF is the main tool used to make models refuse clearly harmful requests, and it works well enough to be over-read. Four points separate what it establishes from what it does not.

It trades off helpfulness and harmlessness. Bai et al. measured a tension between the two at the level of both preference models and trained policies: data that teaches refusals pulls against data that rewards helpfulness, although larger preference models handled both distributions better [5]. Too much weight on harmlessness yields evasive, over-refusing assistants; our research on over-refusal calibration covers how to measure that side.

It depends on adversarial data. Harmlessness training needs examples of the requests the model should resist, which comes from red teaming. Ganguli et al. released 38,961 red team attacks and found that RLHF models became increasingly difficult to red team as they scaled, while plain and prompted models showed flat trends [13]. The red-teaming guide covers how such data is produced and why attackers still find gaps.

It cannot exceed the judgment of its raters. Casper et al. classify as fundamental that humans cannot evaluate performance on difficult tasks well and can be misled, so their evaluations can be gamed [14]. A model rewarded for answers that look right to a rater is trained to look right. This is the motivation for scalable oversight and for rewards that can be checked mechanically, as in RL with verifiable rewards.

It shapes behaviour on the training distribution. It does not remove capabilities, proves nothing about unusual inputs, and can be partly undone by fine-tuning or adversarial prompting. Safety claims therefore rest on evaluation, not on the training method: safety benchmarks, dangerous capability evaluations and adversarial testing. See what is AI safety for the broader picture.

RLHF in practice: a checklist#

The last two rows are evaluation problems needing the usual statistical care; see the guide to AI evals. If a judge model supplies held-out preferences, check it for the length and position biases you are trying to catch (our note on judge bias).

What the toy leaves out#

The simulation uses 8 Gaussian features, a linear reward model that is exact everywhere, and the closed-form optimum instead of PPO on a language model. Real reward models are unreliable far from the SFT distribution, an extra error the toy lacks that makes overoptimisation arrive sooner. The numbers illustrate mechanisms, not estimates for any real system.

Frequently asked questions#

Sources#

1. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei. Deep reinforcement learning from human preferences, 2017. arxiv.org
2. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving. Fine-Tuning Language Models from Human Preferences, 2019. arxiv.org
3. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. Learning to summarize from human feedback, 2020. arxiv.org
4. Long Ouyang, Jeff Wu, Xu Jiang, et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org
5. Yuntao Bai, Andy Jones, Kamal Ndousse, et al.. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022. arxiv.org
6. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. Proximal Policy Optimization Algorithms, 2017. arxiv.org
7. Ralph Allan Bradley, Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons, 1952. doi.org
8. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023. arxiv.org
9. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
10. Prasann Singhal, Tanya Goyal, Jiacheng Xu, Greg Durrett. A Long Way to Go: Investigating Length Correlations in RLHF, 2023. arxiv.org
11. Mrinank Sharma, Meg Tong, Tomasz Korbak, et al.. Towards Understanding Sycophancy in Language Models, 2023. arxiv.org
12. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, Roberta Raileanu. Understanding the Effects of RLHF on LLM Generalisation and Diversity, 2023. arxiv.org
13. Deep Ganguli, Liane Lovitt, Jackson Kernion, et al.. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned, 2022. arxiv.org
14. Stephen Casper, Xander Davies, Claudia Shi, et al.. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback, 2023. arxiv.org