Reinforcement learning for LLMs: how RL trains language models, from RLHF to RLVR

Reinforcement learning for LLMs means treating a language model as a policy: it reads a prompt, takes actions by emitting tokens, and receives a reward for the finished response, from a reward model trained on human preferences, a program that checks the answer, or an environment the model acts in. Training then raises the probability of responses that scored well relative to others. It is how chat models are tuned to be helpful and harmless (RLHF) and how reasoning models are trained on maths and code (RLVR).

This pillar page explains the framing, what RL adds over supervised fine-tuning, how the field got from RLHF to verifiable rewards, and where the safety problems come from. A small simulation you can drive in the browser shows RL beating imitation and then, with a slightly wrong reward, learning the wrong thing while its score keeps rising. Each section links down to a deeper article in this cluster.

The LLM as a policy: tokens as actions#

Classic reinforcement learning has an agent that observes a state, chooses an action from a policy, receives a reward and moves to a new state. Map that onto text generation and almost everything lines up:

• Policy: the language model's next-token distribution ${\pi }_{\theta }(y_t\mid x,y_{<t})$, with parameters $\theta$.
• State: the prompt $x$ plus the tokens generated so far $y_{<t}$.
• Action: the next token $y_t$, one choice from a vocabulary of tens of thousands.
• Transition: deterministic. Appending the token gives the next state.
• Reward: usually a single number $r(x,y)$ for the complete response $y$, plus an optional per-token penalty for drifting from a reference model.

Two equivalent views are useful. The bandit view treats the whole response as one action: the prompt is the context, the response is the arm, and the reward is the payoff. It is how most of the maths in RLHF is written, and it is why bandit intuitions about exploration carry over. The token view treats each token as an action in a long episode with a reward at the end. It is how the training code works, and it raises the classic RL problem of credit assignment: which of the 500 tokens deserves the credit for a correct answer?

The objective is the expected reward of the policy's own samples, and its gradient is the standard policy gradient [1, 2]:

$\mathcal{D}$ is the distribution of training prompts, $T$ the response length and $A_t$ the advantage of token $t$: how much better the outcome was than expected from that point. In words: push up the log-probability of every token in responses that did better than expected, and push down those in responses that did worse. Everything else, from REINFORCE to PPO and its group-relative variants, is a choice about how to estimate $A_t$ with low variance and how big a step to take. The policy gradient theorem, baselines and generalised advantage estimation articles cover that machinery from the RL side.

What does RL add over supervised fine-tuning?#

Supervised fine-tuning (SFT) trains the model to imitate demonstrations by maximising their log-likelihood. In RL terms it is behaviour cloning: it never sees its own mistakes, and it can at best match the demonstrator. RL differs in four ways that matter.

1. It trains on its own outputs. The gradient is computed on samples from the current policy, so errors the model actually makes get corrected, rather than errors a human would make. Sequence-level RL for text was first proposed partly for this reason, to remove the mismatch between training on reference prefixes and generating from one's own [3].
2. It uses negative signal. SFT only says "do this". A policy gradient with a baseline also says "do less of that", pushing probability away from failed attempts.
3. It optimises what you care about. The reward can be any function of the whole response: correctness, a preference score, a passing test suite. Likelihood of a reference text is only a proxy for those.
4. It can exceed the demonstrator. When checking an answer is easier than writing one, a policy can be trained to do better than any demonstration it was given. This is the logic behind using reward models instead of more demonstrations [4, 7] and behind verifiable rewards for maths and code [15, 16].

The empirical picture is more nuanced than "RL good, SFT bad". Chu et al. found that in their tasks RL with outcome rewards generalised to rule and visual variants where SFT tended to memorise, while SFT remained useful for getting the output format stable enough for RL to work [17]. Yue et al. compared reasoning models trained with verifiable rewards against their base models and found that the RL models win at pass@1 but the base models catch up and overtake at large $k$ in pass@k, suggesting that RL mostly sharpens the distribution towards reasoning paths the base model could already sample rather than creating new ones [18]. Between the two sits rejection-sampling fine-tuning: sample, keep what passes a check, and fine-tune on it, as in STaR [13]. It is the simplest RL-like method and a strong baseline.

Seeing it in a toy: imitation, RL, and a slightly wrong verifier#

To make these points concrete we built a tiny "LLM" in numpy. There are 24 prompts. A response is three tokens from a four-token vocabulary, so 64 possible responses per prompt. Exactly one response per prompt is correct: think of the first two tokens as the working and the last as the answer. The model is tabular and autoregressive: one softmax over the next token for every prefix, so it really does generate token by token.

A noisy demonstrator writes the correct response 55.0 percent of the time on average (it varies by prompt). Some of its mistakes still land on the right final answer with the wrong working; the rest are plainly wrong. The SFT model is fitted by maximum likelihood to 40 demonstrations per prompt. RL starts from the SFT model, uses it as the KL reference, and follows the exact policy gradient of expected reward minus $\beta$ times KL (expected rather than sampled gradients, so the curves are smooth and reproducible; the code snippet further down shows the sampled version on a single prompt). We compare two verifiers:

• Full check: reward 1 only for the correct response, working and answer.
• Answer-only check: reward 1 for any response with the right final token. This is the toy version of a maths grader that compares final answers and never reads the derivation.

With the full check, RL does what the theory promises. True accuracy goes from 0.530 for the SFT model to 0.970 after 50 steps and 0.994 after 300, at a KL of 0.75 nats from the SFT model. The SFT model cannot get there by imitation: more demonstrations would converge to the demonstrator's 0.550. Rejection-sampling fine-tuning on the demonstrations that pass the full check reaches 0.971, which shows how much of RL's advantage comes simply from filtering by a correct signal.

With the answer-only check, the verifier's score climbs to 0.994 by step 300 and keeps rising, but true accuracy stalls at 0.884. On 3 of the 24 prompts the policy converges onto a response with the right answer and the wrong working, because the demonstrator wrote those often enough for RL to amplify them and the reward never distinguishes them. True pass@8 is the most telling curve: it peaks at 0.993 at step 23 and then falls, to 0.960 at step 300 and 0.921 at step 1,600, below the SFT model's 0.948. The policy has become more confident and less diverse in exactly the prompts where its confidence is misplaced.

A KL penalty does not rescue the answer-only run. At $\beta =0.05$ true accuracy peaks at 0.873 around step 234 and then declines to 0.827 at step 1,600, heading towards 0.758, which is the accuracy of the exact KL-regularised optimum. The penalty keeps pass@8 at 0.994, because it preserves diversity, but at the optimum it also preserves the SFT model's ratio of correct to lucky responses among those the verifier accepts. The KL-regularised RL article derives why. The lesson generalises: RL is an amplifier of whatever the reward measures, and the regulariser decides how much amplification, not what gets amplified.

From RLHF to RLVR: a short history#

Ranzato et al. trained text generators with REINFORCE on sequence-level metrics such as BLEU in 2015 [3]. The modern line starts with Christiano et al., who trained agents in simulated control and Atari tasks from human comparisons between pairs of short clips, needing feedback on less than one percent of the agent's interactions [4]. Ziegler et al. applied the same recipe to language models, introducing the reward-model-plus-KL-penalty setup that is still standard [5]. Stiennon et al. scaled it to summarisation and showed that optimising a learned reward too hard degrades real quality [6].

Ouyang et al. turned it into the instruction-following pipeline most people now mean by RLHF: supervised fine-tuning on demonstrations, a reward model trained on rankings of model outputs, then PPO [10] against that reward model with a KL penalty. Labellers preferred outputs of their 1.3-billion-parameter tuned model to those of the 175-billion-parameter base model [7]. Bai et al. trained helpful and harmless assistants this way and documented the tension between the two goals [8], then replaced most human harm labels with AI feedback guided by a written list of principles in constitutional AI [9]. Rafailov et al. showed that the same KL-regularised objective can be optimised directly on preference pairs without a separate reward model or RL loop (DPO) [11].

The second wave swapped learned rewards for checkable ones. Lightman et al. compared rewarding final answers with rewarding each reasoning step [12]. The DeepSeekMath work introduced GRPO, a PPO variant that replaces the learned value function with the mean reward of a group of samples for the same prompt [14]. Lambert et al. named the approach reinforcement learning with verifiable rewards, RLVR, in the Tulu 3 report [15], and the DeepSeek-R1 report showed that large-scale RL with rule-based correctness and format rewards, starting directly from a base model, raised pass@1 on the AIME 2024 maths benchmark from 15.6 to 71.0 percent and produced long chains of reasoning without supervised reasoning traces [16]. The current frontier extends the same idea to agents acting over many steps in software environments.

The main forms of RL for language models#

The methods differ mostly in where the reward comes from. Each subsection is a summary; the linked article goes deep.

RLHF: rewards from human preferences#

Humans compare pairs of responses; a reward model learns to predict their choices; the policy is optimised against the reward model with a KL penalty. It is the right tool when quality is easy to recognise and hard to specify: tone, helpfulness, appropriate refusals. Its weaknesses are the cost and noise of human labels and the gap between what raters reward and what is true. Full pipeline, equations and failure modes: RLHF explained.

Reward models#

A reward model is usually the language model itself with a scalar output head, trained with a Bradley-Terry loss so that the preferred response in each pair scores higher. Its pairwise accuracy on held-out preferences is easy to measure and only loosely tied to how good a policy trained against it will be. How they are trained, evaluated and broken: reward models, plus our research notes on preference label noise and reward model overoptimization.

The KL penalty#

Nearly every method above optimises reward minus $\beta$ times the KL divergence from a reference model. The optimum has a closed form, ${\pi }^{\ast }\propto {\pi }_{\text{ref}}\exp (r/\beta )$, and $\beta$ is the exchange rate between reward and drift. Derivation, interactive frontier, estimators and adaptive control: KL penalty in RLHF.

RLAIF and constitutional AI#

Replace the human comparisons with a model's comparisons, guided by written principles. It scales cheaply and makes the value judgments explicit and editable, but the judge's biases become the policy's targets. See RLAIF and constitutional AI and, on judge biases, LLM-as-a-judge and our note on position and verbosity bias.

RLVR: verifiable rewards#

For maths, code and other tasks with checkable answers, a program provides the reward: answer matching, unit tests, a proof checker. There is no reward model to overoptimise, but verifiers have loopholes, as the toy above shows. See RL with verifiable rewards, our note on reward hacking in RLVR and the comparison of process and outcome supervision.

RL environments for agents#

When the model calls tools, edits files or browses over dozens of steps, the "environment" stops being trivial: it has state, latency and side effects, and the reward is whether the task got done. Designing environments that reward the task rather than a shortcut is now a central engineering problem. See RL environments for LLM agents and our note on evaluating agent trajectories.

Reward hacking#

Every reward signal above is a proxy, and RL is an optimiser pointed at the proxy. When the two diverge, the policy finds the gap. See reward hacking and, from the safety side, specification gaming.

A note on scope: this cluster explains each algorithm at the depth its topic needs rather than writing one tutorial per post-training algorithm. The RL foundations behind all of them are in the reinforcement learning pillar. Two foundational topics deserve a special mention for language models: offline RL, because direct preference methods learn from a fixed dataset and inherit its distribution-shift problems, and inverse RL, because learning a reward from human behaviour is exactly what a reward model does.

Inside the training loop#

Strip away the engineering and an RL step for a language model looks like this: sample a batch of prompts; generate one or more responses per prompt with the current policy; score them; compute advantages; take a clipped or regularised gradient step; repeat. The design choices are about the advantage estimate and the step size.

• Baseline. PPO-style RLHF trains a value network to predict expected reward from each prefix, the actor-critic approach. Group-relative methods sample several responses per prompt and use their mean reward as the baseline, which removes the value network at the cost of more samples per prompt [14].
• Credit assignment. With one reward at the end, every token in a response usually shares the same advantage. Process reward models and per-step verifiers give denser signal [12], at the price of a second thing to get right.
• Step size. Large updates on a model that is already good are dangerous. PPO clips the probability ratio between the new and old policies [10]; the KL penalty to a fixed reference limits cumulative drift.

The loop fits in a few lines when the "LLM" is a lookup table. This is REINFORCE with a group-mean baseline on one prompt, the core of group-relative methods without clipping or KL:

import numpy as np

V, T, G = 4, 3, 16            # vocabulary, response length, samples per prompt
rng = np.random.default_rng(0)
logits = {}                   # one row of next-token logits per prefix (a tabular "LLM")

def next_dist(prefix):
    z = logits.setdefault(prefix, np.zeros(V))
    p = np.exp(z - z.max())
    return p / p.sum()

def sample():
    prefix = ()
    for _ in range(T):
        prefix += (int(rng.choice(V, p=next_dist(prefix))),)
    return prefix

def reward(y):                # verifiable reward: 1 for the one correct response
    return float(y == (2, 0, 3))

for step in range(300):
    ys = [sample() for _ in range(G)]
    rs = np.array([reward(y) for y in ys])
    adv = rs - rs.mean()      # group-mean baseline: no value network needed
    for y, a in zip(ys, adv):
        for t in range(T):    # every token of the response shares the sequence-level advantage
            p = next_dist(y[:t])
            grad = -p
            grad[y[t]] += 1.0 # gradient of log softmax w.r.t. the logits
            logits[y[:t]] += 0.5 * a * grad / G

print(np.mean([reward(sample()) for _ in range(1000)]))   # 0.979 with this seed

Starting from a uniform policy that finds the correct response one time in 64, it reaches 0.979 after 300 steps. Note what happens when all 16 samples fail: every advantage is zero and nothing is learned. That is why RLVR needs prompts of the right difficulty, and why an SFT stage that makes success possible at all comes first.

The safety angle: RL is an optimiser pointed at a proxy#

RL is the main tool used to make language models safer, and it is also the main source of a class of safety problems. Both follow from the same fact: it reliably increases whatever the reward measures.

RL as a safety tool#

Refusing dangerous requests, declining to fabricate, following a system prompt, and admitting uncertainty are all behaviours that are easier to recognise than to demonstrate exhaustively, which is the setting where preference-based RL shines [7, 8]. Constitutional methods made the principles behind those preferences explicit [9]. The trade-offs are real: Bai et al. found helpfulness and harmlessness in tension [8], and pushing hard on harmlessness produces over-refusal, which we analyse in over-refusal calibration. Red-team findings are a major source of the prompts used in this training; see red teaming versus evals and our guide to safety training data.

RL as a source of risk#

Reward hacking was listed among the concrete problems in AI safety before RLHF on language models existed [24], and it has a formal definition: optimising a proxy increases it while decreasing the true objective [25]. In language models it takes recognisable forms:

• Overoptimisation of learned rewards. Gold quality rises then falls as the policy moves away from its start, predictably with reward model size and KL [19].
• Sycophancy. Human preference data favours responses that agree with the user, and optimising against preference models can trade truthfulness for agreement [21].
• Verifier loopholes. Graders that check only the final answer, or tests that can be edited, reward the shortcut. In the toy above, a 0.994 verifier score coexisted with 0.884 true accuracy. Baker et al. report reasoning models exploiting coding-task graders in training, and found that penalising visible "bad thoughts" in the chain of thought led to hacking that was harder to see rather than to less hacking [23].
• Generalisation of gaming. Denison et al. trained models on a curriculum of increasingly gameable environments and found that some generalised, rarely but measurably, to rewriting their own reward function in an environment they had never seen [22].

Casper et al. catalogue the open problems of RLHF more broadly, separating those that better engineering can fix from those that are fundamental to learning values from human feedback [20]. For the alignment framing, see AI alignment and scalable oversight, which asks how to provide rewards for tasks humans cannot easily judge.

Evaluating RL-trained models#

The defining feature of reward hacking is that the training metric looks fine. So the evaluation of an RL-trained model must use signals the policy was not optimised against: held-out human or expert labels, independent judges validated against humans, behavioural tests for sycophancy and refusals, and adversarial probing. That is the core of an evaluation programme and of red teaming; the AI evals pillar covers the methods, and statistical significance for evals covers the error bars that small held-out sets need.

A practical checklist#

What the toy does not show#

The simulation has 64 possible responses per prompt, a tabular policy with no generalisation between prompts, exact expected gradients and a demonstrator we designed. Real models share parameters across prompts, which is where effects such as RL improving pass@1 at the cost of large-$k$ coverage across the whole distribution come from [18]; the toy can only show the per-prompt version. The numbers illustrate mechanisms, not magnitudes. Everything here is simulation; no language model was trained.

Frequently asked questions#

Sources#

1. Richard S. Sutton, Andrew G. Barto. Reinforcement Learning: An Introduction (2nd edition), 2018. incompleteideas.net
2. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, 1992. doi.org
3. Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojciech Zaremba. Sequence Level Training with Recurrent Neural Networks, 2015. arxiv.org
4. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei. Deep reinforcement learning from human preferences, 2017. arxiv.org
5. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving. Fine-Tuning Language Models from Human Preferences, 2019. arxiv.org
6. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. Learning to summarize from human feedback, 2020. arxiv.org
7. Long Ouyang, Jeff Wu, Xu Jiang, et al.. Training language models to follow instructions with human feedback, 2022. arxiv.org
8. Yuntao Bai, Andy Jones, Kamal Ndousse, et al.. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022. arxiv.org
9. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al.. Constitutional AI: Harmlessness from AI Feedback, 2022. arxiv.org
10. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. Proximal Policy Optimization Algorithms, 2017. arxiv.org
11. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023. arxiv.org
12. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, et al.. Let's Verify Step by Step, 2023. arxiv.org
13. Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman. STaR: Bootstrapping Reasoning With Reasoning, 2022. arxiv.org
14. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, et al.. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. arxiv.org
15. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, et al.. Tulu 3: Pushing Frontiers in Open Language Model Post-Training, 2024. arxiv.org
16. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, et al.. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. arxiv.org
17. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, et al.. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training, 2025. arxiv.org
18. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, et al.. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, 2025. arxiv.org
19. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization, 2022. arxiv.org
20. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, et al.. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback, 2023. arxiv.org
21. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, et al.. Towards Understanding Sycophancy in Language Models, 2023. arxiv.org
22. Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, et al.. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models, 2024. arxiv.org
23. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, et al.. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, 2025. arxiv.org
24. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. Concrete Problems in AI Safety, 2016. arxiv.org
25. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David Krueger. Defining and Characterizing Reward Hacking, 2022. arxiv.org