RLHF (reinforcement learning from human feedback)

Reinforcement learning from human feedback (RLHF) is a post-training method in which people compare pairs of model outputs, a reward model is trained to predict those preferences, and the model is then optimised with reinforcement learning to score highly under that reward model. A KL penalty keeps the tuned model close to its starting point.

The three stages

• Supervised fine-tuning (SFT). Train a pretrained model on demonstrations of good behaviour. This model becomes both the starting policy and the reference policy.
• Reward modelling. Collect preference data: for each prompt, a person picks the better of two responses. Fit a reward model $r_{\varphi }(x,y)$ with the Bradley-Terry loss.
• RL fine-tuning. Optimise the policy ${\pi }_{\theta }$, usually with PPO, on the KL-regularised objective below.

Here $x$ is a prompt from the training distribution $\mathcal{D}$, $y$ is a sampled response, ${\pi }_{\mathrm{ref}}$ is the SFT model and $\beta$ sets how strongly the policy is held near it. In the RL step the generated response is the action sequence and the reward arrives once, at the end.

Worked example

A response scores $r_{\varphi }=2.0$. The policy assigns it a log-probability 3.0 nats higher than the reference does, so its sampled KL contribution is $\log {\pi }_{\theta }(y\mid x)-\log {\pi }_{\mathrm{ref}}(y\mid x)=3.0$. With $\beta =0.1$ the reward the RL algorithm actually sees is $2.0-0.1\times 3.0=1.7$. A second response with reward 1.9 and log-ratio 0.5 gets $1.9-0.05=1.85$, so the penalty flips the ranking in favour of the response the reference model already found plausible.

Known limits

The reward model is a learned proxy, so pushing hard against it produces reward overoptimisation and reward hacking: longer, more flattering or more confident answers that raters liked on average but that are not better. Human labels are noisy and expensive, and labellers can be misled by fluent wrong answers, which is one motivation for RLAIF, DPO and scalable oversight research.

Why it matters

RLHF is the method that turned pretrained language models into instruction-following assistants: in Ouyang et al. (2022), labellers preferred outputs of a 1.3B-parameter RLHF model over a 175B model trained only by pretraining. It is also where many safety behaviours, including refusals and their failure mode over-refusal, are instilled, so its reward model and preference data deserve the same scrutiny as any eval set.

Related terms

• Reward model
• Preference data
• KL penalty
• PPO (Proximal Policy Optimization)
• RLAIF (reinforcement learning from AI feedback)
• Reward overoptimization

Sources

1. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei. Deep reinforcement learning from human preferences, 2017. arxiv.org
2. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. Learning to summarize from human feedback, 2020. arxiv.org
3. Long Ouyang, Jeff Wu, Xu Jiang, et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org