DPO (Direct Preference Optimization)

Direct Preference Optimization (DPO) is a post-training method that fits a language model to preference pairs with a single supervised loss, without training a separate reward model or running reinforcement learning. It solves the same KL-regularised objective as RLHF, in closed form.

Where the loss comes from

The KL-regularised RLHF objective has a known optimal policy, ${\pi }^{\ast }(y\mid x)\propto {\pi }_{\mathrm{ref}}(y\mid x)\exp (r(x,y)/\beta )$. Rearranging gives the reward implied by any policy: $r(x,y)=\beta \log \frac{\pi (y\mid x)}{{\pi }_{\mathrm{ref}}(y\mid x)}+\text{const}$. Substituting that into the Bradley-Terry likelihood, the constant cancels and leaves

where $y_w$ and $y_l$ are the chosen and rejected responses, ${\pi }_{\theta }$ is the model being trained, ${\pi }_{\mathrm{ref}}$ is a frozen copy of the starting model and $\beta$ controls how far the model may move. Intuitively, DPO raises the likelihood of the chosen response relative to the reference and lowers that of the rejected one, weighted most on pairs it currently gets wrong.

Worked example

Take $\beta =0.1$. The policy gives the chosen response log-probability $-10$ against $-12$ under the reference (log-ratio $+2$) and the rejected response $-15$ against $-14$ (log-ratio $-1$). The margin is $0.1\times (2-(-1))=0.3$, so the loss is $-\log \sigma (0.3)\approx 0.55$. As the margin grows the loss falls towards zero.

DPO vs RLHF

• Simpler: two models in memory (policy and reference), no sampling during training, no value network.
• Offline: it learns only from fixed pairs, so it cannot discover and correct its own new failure modes the way on-policy RL can. Iterated or online variants regenerate pairs from the current model.
• Still exploitable: the implicit reward can overfit, for example by lowering the probability of both responses or drifting towards longer outputs.

Why it matters

DPO made preference tuning cheap enough for small teams, and it is now a common default for safety and style tuning. For evaluation, the lesson is that a lower DPO loss is not the same as a better model: it measures fit to the preference set, so held-out behavioural evals and refusal-calibration checks still decide whether the tuned model is safe to ship.

Related terms

• RLHF (reinforcement learning from human feedback)
• Preference data
• Bradley-Terry model
• KL penalty
• Reward model

Sources

1. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023. arxiv.org