Gen AI

DPO

DPO trains a model straight on pairs of preferred and rejected answers, skipping the reward model and reinforcement learning that RLHF needs entirely.

RLHF trains a reward model and then runs PPO against it; DPO replaces both stages with one loss computed directly on the policy's probabilities for the chosen and rejected response

Why Does This Exist?

Say you've got a general-purpose chat assistant, already through supervised fine-tuning, and a pile of preference data: pairs of responses to the same prompt, each with a human-marked winner and loser. RLHF turns that data into an aligned model in two stages — train a reward model to imitate the human's judgment, then run PPO to push the policy toward whatever that reward model scores highly. It works. It's also a lot of machinery for what is, underneath, a fairly simple signal: make chosen more likely than rejected, for the same prompt.

DPO exists because someone noticed that machinery is optional. Make one reasonable assumption about how preferences relate to reward, and you can write a loss that trains the policy directly on the preference pairs — no reward model, no PPO, no RL at all. One training stage instead of three, same data, same goal.

Think of It Like This

Grading a cooking contest without hiring a judge

Imagine a cooking contest with pairs of dishes, each pair marked with which one the tasters preferred. The RLHF way: hire a judge, train them for weeks on those notes until they can score any dish the way the tasters would, then send contestants back to the kitchen to cook against that judge's feedback, over and over, hoping the scores track real taste.

The DPO way: skip the judge. Read the preference notes directly and tell each contestant, "make your winning dish more like itself and your losing dish less like itself, relative to how you were already cooking." You adjust behavior straight from the comparisons, no scorer standing between the data and the change you want.

How It Actually Works

What RLHF actually needs, and what DPO removes

Strip RLHF down to its purpose: the preference data only exists to teach a reward model what "good" looks like, and the reward model only exists so PPO has something to run reinforcement learning against. It's an intermediate step, not the goal — nobody wants a reward model for its own sake, they want an aligned policy, and the reward model is just the tool that makes running RL possible.

DPO's insight is that this intermediate step is mathematically removable. Under a mild assumption about how preferences relate to an underlying reward function, the optimal policy for the RLHF objective has a closed form: expressible directly in terms of the policy's own probabilities, with the reward model's terms canceling out entirely. Once it's gone algebraically, you're left with a loss over the policy alone — comparing how likely it makes the chosen response against how likely it makes the rejected one, same prompt. No separate model ever gets trained.

The DPO loss, intuitively

For each pair, DPO looks at two probabilities under the policy: how likely it is to generate the chosen response, and how likely it is to generate the rejected one, both for the same prompt. The loss pushes those apart — up for chosen, down for rejected — using a comparative signal in the same spirit as a classification loss: is the policy correctly more confident in chosen than rejected, and by how much.

That "by how much" is measured against a fixed reference copy of the original SFT model, frozen before training starts. The policy's chosen-versus-rejected gap gets compared to the reference model's gap on the same pair, and the loss only rewards the policy for pulling further ahead of where the reference sat. That reference term plays the exact role the KL penalty played in RLHF: it anchors the policy near the model it started as, so it doesn't collapse into repeating chosen text while breaking everything SFT had already gotten right.

What you gain, and what you give up

DPO's case is simplicity and stability. No reward model to train, tune, or watch for overfitting; no reward-hacking failure mode where the policy exploits quirks in a scorer instead of genuinely improving; no PPO instability, no rollout loop, no extra RL hyperparameters to babysit. It's also one fewer trained artifact than RLHF needs, a real saving in engineering time, since a whole stage and its model just disappear.

The tradeoff is flexibility. RLHF's reward model is reusable — once trained, you can rerank candidates, filter generations, or score outputs from a different model entirely. DPO gives you none of that; it never produces a standalone scorer, since the "reward" stays implicit, folded into the policy's own probabilities. It also inherits whatever biases sit in the preference dataset with nowhere left to catch them. RLHF at least lets you interrogate the reward model on held-out examples before it touches the policy. DPO skips straight from raw preferences to policy weights, so a biased dataset shows up directly in the final model, with no checkpoint that could have caught it.

Show Me the Code

A toy comparative loss, showing the structure DPO actually optimizes: how much the policy's gap between chosen and rejected exceeds the reference model's gap on the same pair.

1import numpy as np2
3
4def dpo_loss_signal(5    chosen_logprob: float,6    rejected_logprob: float,7    ref_chosen_logprob: float,8    ref_rejected_logprob: float,9    beta: float = 0.1,10) -> float:11    """Illustrative only -- shows the comparative shape, not a full trainer."""12    policy_gap = chosen_logprob - rejected_logprob      # policy's own preference13    ref_gap = ref_chosen_logprob - ref_rejected_logprob  # same gap, frozen reference14    margin = beta * (policy_gap - ref_gap)               # progress beyond the reference15    prob_correct = 1 / (1 + np.exp(-margin))             # confidence chosen > rejected16    return float(-np.log(prob_correct))17
18
19loss = dpo_loss_signal(-1.2, -2.4, -1.3, -1.8)20print(round(loss, 4))  # -> 0.6588

Watch Out For

Treating the dataset as solved because the reward model is gone

DPO removes the reward model, not the need for good preference data. A noisy or biased set of chosen/rejected labels flows straight into the policy with nothing in between to inspect or catch it. The dataset quality problem doesn't disappear — it just has nowhere left to hide.

Dropping the reference-model term to simplify the loss

The comparison against a frozen SFT reference isn't a minor regularizer you can skip for memory. Without it, nothing anchors the policy, and it can drift arbitrarily far while still technically satisfying "chosen more likely than rejected" — often by degenerating in ways unrelated to what you wanted it to learn.

The Quick Version

RLHF needs a reward model purely as a stepping stone to running RL; DPO shows that stepping stone can be removed mathematically under a mild assumption.
The DPO loss compares the policy's chosen-versus-rejected probability gap against the same gap under a frozen reference model, and rewards the policy for widening it.
The reference model plays the same role the KL penalty played in RLHF: it stops the policy from drifting arbitrarily far from the original SFT model.
DPO is simpler and more stable to train — no reward model to overfit, no RL loop — and needs one fewer trained artifact than RLHF.
The tradeoff is a lost standalone reward model for reranking or filtering, and no separate checkpoint to catch bias baked into the preference data.

DPO