RL for language models
Language models are pretrained to predict text, then shaped by reinforcement learning to be helpful, follow instructions, refuse harmful requests and solve problems with checkable answers. That second stage is where most behaviour that users notice is decided, and it is also where a model learns whatever the reward actually pays for, intended or not.
This topic starts with the overview and then follows the pipeline in the order teams build it. RLHF covers collecting human preferences and turning them into a training signal. The reward model page explains how preferences become a scalar score and why that score is only a proxy. The KL penalty keeps the policy close to its starting point, and the page on it shows the trade-off between reward and drift. Verifiable rewards replace learned judges with programmatic checks for maths and code. RLAIF and constitutional methods replace some human labels with model feedback. Agent environments extend all of this to multi-step tool use. The final page collects the ways optimisation finds loopholes, and how to detect them before deployment.
Where a specific post-training algorithm appears, we explain it only as far as the topic needs.
Reinforcement learning for LLMs: how RL trains language models, from RLHF to RLVR
How RL trains LLMs: the model as a policy, tokens as actions, what RL adds over SFT, RLHF to RLVR, and why reward hacking is a safety problem.
Suggested reading order
- 1Reinforcement learning for LLMs: how RL trains language models, from RLHF to RLVR
How RL trains LLMs: the model as a policy, tokens as actions, what RL adds over SFT, RLHF to RLVR, and why reward hacking is a safety problem.
- 2RLHF explained: how reinforcement learning from human feedback works, and where it fails
How RLHF works end to end: SFT, preference data, Bradley-Terry reward models, PPO with a KL penalty, a toy pipeline, failure modes and safety limits.
- 3Reward models: how they are trained, how to evaluate them, and why accuracy is not enough
What a reward model is, how Bradley-Terry training works, how to evaluate reward models, and why pairwise accuracy can miss what matters downstream.
- 4KL penalty in RLHF: KL-regularised RL, the optimal policy and how to choose beta
Why RLHF penalises KL to a reference model, the closed-form optimal policy, how beta trades reward for drift, k1/k2/k3 estimators and adaptive KL control.
- 5Reinforcement learning with verifiable rewards (RLVR), explained
How RLVR trains models with programmatic verifiers: group-relative advantages, why pass rate controls the learning signal, and what verifier errors do.
- 6RLAIF and Constitutional AI: reinforcement learning from AI feedback
How RLAIF and Constitutional AI replace human labels with model judgments: critique and revision, AI preference labels, evidence vs RLHF, failures.
- 7RL environments for LLM agents: state, tools, reward and sandboxing
How to design RL environments for LLM agents: tasks, state, tool calls, rewards, sandboxing, reward leakage, and why eval and training environments differ.
- 8Reward hacking: definition, taxonomy, examples and mitigation
What reward hacking is, a taxonomy of how rewards get exploited, documented examples from RL and LLM training, and how to detect and prevent it.
Guides and explainers
Preference label noise: annotator agreement, Bradley-Terry reward models and how many labels per pair
What annotator agreement does and does not tell you, how flip noise affects Bradley-Terry reward models and DPO, and how many annotators per pair to pay for.
Reward model overoptimization: why gold reward peaks while the proxy keeps climbing
Why gold reward rises then falls as you optimise a reward model, how best-of-n and RL differ in KL, and what KL penalties and ensembles really fix.
Reward hacking with verifiable rewards: measuring and hardening the verifier
How policies exploit RLVR verifiers, why a small false-positive rate lifts training reward while true accuracy falls, and how to audit and harden them.
What expertise buys in training and evaluation data
Label error by annotator tier, why majority vote misses systematic errors, expert adjudication, routing and the cost-quality frontier for reward-model data.
Research notes on this topic
Know the number before you ship.
A 45-minute working session with an evaluation engineer. Bring one use case; leave with a threat model, a draft eval plan with sample sizes, and an honest read on where you stand.