DPO: Where Did the Reward Model Go?

Sat, 26 Sep 2026 00:00:00 +0000

Classical RLHF resembles a three-stage production line: collect pairwise preferences, train a scalar reward model, then use PPO to make the language model pursue that score while a KL penalty keeps it tethered. DPO says the second and third stages can be removed, leaving a loss that looks like binary classification.

That claim is often compressed into the misleading phrase “DPO needs no reward model.” More precisely, DPO does not train a separate, callable reward network, but it still assumes that preferences arise from a latent reward. It rewrites that reward as a change in policy log-probability relative to a reference model. The reward model moves from an explicit artifact into the language model’s coordinate system.

Understanding why that substitution works, what a batch actually computes, and where the simplification moves the remaining risk explains most of DPO.

It Removes Training Stages, Not the Preference Assumption

For prompt $x$, an annotator prefers response $y_w$ over $y_l$. A conventional reward model often uses the Bradley–Terry assumption:

$$ p(y_w \succ y_l\mid x)=\sigma\big(r(x,y_w)-r(x,y_l)\big). $$

After fitting that reward network, RLHF solves a KL-regularized policy objective:

$$ \max_\pi\;\mathbb E_{y\sim\pi}[r(x,y)]- \beta D_{\mathrm{KL}}\!\left(\pi(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x)\right). $$

The first term favors high-reward responses. The second prevents the policy from moving too far from its reference. This is not decorative: a reward model is reliable only near the finite distribution on which it was trained, and a policy can learn to exploit it outside that region.

Figure 1: conventional RLHF fits a reward model and then runs PPO; DPO updates the policy directly from preference pairs

DPO’s key move is to observe that the optimal policy for this objective has a closed form:

$$ \pi_r(y\mid x)=\frac{1}{Z(x)}\pi_{\mathrm{ref}}(y\mid x) \exp\!\left(\frac{r(x,y)}{\beta}\right). $$

Rearranging expresses reward through the policy:

$$ r(x,y)=\beta\log\frac{\pi_r(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta\log Z(x). $$

The inconvenient partition function $Z(x)$ depends only on the shared prompt. When the expression is substituted into Bradley–Terry, the two response rewards are subtracted and $\beta\log Z(x)$ cancels. We can fit which answer a person would choose without recovering an absolute reward first. This is a change of variables, not an approximation.

One Pair Becomes Four Log-Probabilities

For each $(x,y_w,y_l)$, DPO computes sequence log-probabilities for the chosen and rejected responses under both the current policy and the reference. Define

$$ a_w=\log\pi_\theta(y_w\mid x)-\log\pi_{\mathrm{ref}}(y_w\mid x), $$$$ a_l=\log\pi_\theta(y_l\mid x)-\log\pi_{\mathrm{ref}}(y_l\mid x). $$

Each $a$ measures how much more support the current policy gives a response than its starting point did. DPO minimizes

$$ \mathcal L_{\mathrm{DPO}}=-\log\sigma\big(\beta(a_w-a_l)\big). $$

Figure 2: DPO constructs one relative advantage gap from policy/reference × chosen/rejected log-probabilities

Three details matter.

First, the isolated rule is not “the chosen probability must rise.” The chosen response’s gain over the reference must exceed the rejected response’s gain. Because parameters are shared, both absolute probabilities can even fall while the relevant margin improves.

Second, the reference is not merely an old checkpoint label. It sets the zero point of the implicit reward and acts as the KL anchor. The paper initializes both policy and reference from an SFT model when one is available. If a matching SFT model is unavailable, it first maximizes likelihood on preferred completions to reduce distribution mismatch.

Third, $\beta$ controls conservativeness relative to the reference, but conventions differ across implementations; its numeric value should not be copied blindly between libraries. In the authors’ reference code, the core really is short: compute the policy’s chosen/rejected log-ratio, subtract the reference log-ratio, and pass the result through logsigmoid. A short loss does not mean one model-sized forward pass, however. A straightforward implementation still evaluates the trainable policy and frozen reference, so memory, parallelism, and truncation remain engineering concerns.

Its Simplicity Comes From Being Offline—and So Does Its Limit

DPO does not generate fresh responses inside the training loop, score them with a reward model, or maintain PPO actor, critic, and rollout machinery. A fixed dataset of $(x,y_w,y_l)$ triples can be replayed like supervised learning. That lowers system complexity and makes experiments easier to reproduce.

The original paper evaluated controlled sentiment generation, Reddit TL;DR summarization, and single-turn dialogue from Anthropic Helpful and Harmless, using models up to 6B parameters. In its GPT-4-evaluated summarization setup, DPO reached a reported win rate of roughly 61%, versus about 57% for PPO, and was more robust to sampling temperature. In a direct human comparison, DPO samples were preferred over PPO samples 58% of the time.

Those results support “DPO achieves competitive preference alignment with simpler optimization in these settings.” They do not establish that DPO beats online RL at every scale and on every preference distribution. A fixed dataset cannot observe new failures produced by the changing policy. If training moves the policy beyond the data’s coverage, the supervision does not follow automatically. Who generated the response pairs, which model produced them, and how consistently they were labeled all become part of the algorithm.

Figure 3: DPO removes the explicit reward network and online PPO loop, but retains reference computation and concentrates risk in preference data and distribution shift

It is therefore more useful to distinguish regimes than to say one algorithm replaced another. DPO directly fits a policy on fixed preference data. Online RL is more complex, but can keep exploring the current policy and gathering feedback about its new mistakes.

An Implicit Reward Can Still Be Exploited

Removing the standalone reward network does not remove reward hacking. If the preference data contains a shortcut, DPO can amplify whatever feature predicts a win.

Length is a clear example. Follow-up work by Rafailov and colleagues analyzed Anthropic HH and TL;DR. Preferred responses were only slightly longer in the datasets, yet unregularized DPO produced answers roughly twice as long on average and outside the length distribution of the feedback. The authors interpreted this as out-of-distribution extrapolation of a length-correlated preference and introduced a length regularizer, reporting gains of up to 20% in length-controlled win rate.

The conclusion is not that long answers are bad. It is that a model may turn “more complete answers often win” into “more tokens cause winning,” while automated judges may have their own verbosity preference. Raw win rate can then mix content quality, response length, and judge bias.

Binary preferences impose a deeper compression as well. DPO knows only that $y_w$ beat $y_l$—not by how much, nor whether both were poor. Annotation noise, prompt shift, and entanglement between style and factuality survive the elegant objective. Real systems still need independent evaluation: length-controlled comparisons, human audits, factuality and safety tests, and monitoring of drift from the reference.

The Breakthrough Was Choosing the Right Coordinates

DPO’s durable contribution is not the slogan “one loss replaces RLHF.” It is the coordinate change: under KL-regularized preference optimization, reward differences can be represented by differences in policy-to-reference log-ratios. Bradley–Terry depends only on differences, so the unknown normalization disappears.

That view shortens an expensive pipeline and reveals a broader design lesson. An intermediate model can be only one parameterization of the final object. If its constraint can be written directly into that object, stages may disappear without changing the underlying assumptions.

But removing a module does not remove every problem the module carried. Preferences can remain biased. The reference still defines what “not too far” means. Offline data still bounds what can be learned, and evaluation can still reward surface features. DPO makes the optimization more direct; it does not make the question “what do people actually prefer, and why?” any simpler.

References

Rafailov et al., Direct Preference Optimization: Your Language Model Is Secretly a Reward Model, NeurIPS 2023.
Rafailov et al., DPO paper and appendices, 2023.
Mitchell et al., DPO official reference implementation.
Park et al., Disentangling Length from Quality in Direct Preference Optimization, Findings of ACL 2024.

Preference Optimization on Lee