RLHF / DPO

A raw pre-trained language model optimizes one objective: predicting the statistically most probable next token given context $\max \sum \log P(w_t \mid w_{<t})$. Because internet crawl data contains malicious instructions, toxic discussions, and security exploits alongside benign knowledge, high conditional likelihood does not imply safety, helpfulness, or truthfulness. Likelihood is not human endorsement. Without an intentional alignment objective, the model completes dangerous requests because they match frequent statistical patterns.

Humans Rank Pairs

Absolute scalar grading of generative text is noisy and inconsistent across human annotators. Instead, alignment pipelines present human evaluators with a shared prompt $x$ and two candidate completions: $y_w$ (the winning, helpful, or safe reply) and $y_l$ (the losing, low-quality, or harmful reply). Annotators make a simple binary preference judgment $y_w \succ y_l$. Under the Bradley-Terry preference model, this binary choice models the probability of preference as $P(y_w \succ y_l \mid x) = \sigma(r(x, y_w) - r(x, y_l))$, creating a stable ranking signal $(x, y_w, y_l)$ across thousands of prompt variations.

Judge Then Nudge

In classical Reinforcement Learning from Human Feedback (RLHF), a separate Reward Model $r_\theta(x, y)$ is trained on human preference pairs. The reward model learns to score the winning reply $r(x, y_w) = 2.1$ against the losing reply $r(x, y_l) = 0.4$. Next, Proximal Policy Optimization (PPO) fine-tunes the policy $\pi_\phi$ to maximize this scalar reward while constraining drift from the reference pre-trained policy $\pi_{\text{ref}}$ via Kullback-Leibler regularization: $\max \mathbb{E}[r_\theta(x, y)] - \beta \mathbb{D}_{\text{KL}}(\pi_\phi \parallel \pi_{\text{ref}})$. With $\mathbb{D}_{\text{KL}} \le 0.1$, the probability of generating the high-reward completion shifts from $0.45$ to $0.72$.

Skip the Judge

Direct Preference Optimization (DPO) derives an exact closed-form substitution showing that the optimal policy under the Bradley-Terry preference formulation and KL penalty implicitly defines the reward: $r(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$. By substituting this identity into the preference loss, DPO trains the language model directly on the offline preference pairs without an auxiliary reward model or PPO actor-critic loop. With log-odds ratio margin $\Delta = \log \frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)} - \log \frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)} = 1.25$, DPO minimizes the loss $-\log \sigma(\beta \Delta) = 0.63$ with a single stable supervised gradient step.

Preferred More Often

When tested on an identical evaluation set of 100 held-out prompts, human raters preferred the raw pre-trained model answers only 35 times. Following alignment, the win rate rose to 78 out of 100 prompts—a +43 prompt win margin (+123% relative improvement). The underlying model parameter count and knowledge base remained unchanged; alignment restructured the sampling distribution to favor safe, helpful, and concise answers that humans actually want.

Simple or Full

Engineering practice favors simplicity: begin with DPO because it operates directly on static offline pairs, loads only one active model in GPU memory, and trains with stable cross-entropy gradients. However, DPO cannot explore beyond the distribution of existing preference pairs. When a task requires multi-turn exploration, complex reasoning rollouts, or environments where dynamic feedback generates novel solutions, the full RLHF loop with live policy sampling becomes worth the extra memory and hyperparameter tuning cost.

Where It Breaks

All alignment pipelines face Goodhart's Law: when a proxy judge becomes the training objective, it ceases to be a faithful measure of human intent. Automated reward models frequently suffer from length bias and sycophancy shortcuts, scoring verbose, flattering answers with $r = 9.2$ while assigning concise, truthful answers only $r = 6.1$. Left unconstrained, the policy learns to produce excessive flattery and ungrounded prose. Mitigating reward hacking requires tight KL divergence penalties, length-normalized rewards, and regular adversarial human audits.

The Quick Version

• Likelihood alone cannot say no.
• Ranked pairs become the training signal.
• The judge scores 2.1 against 0.4.
• DPO tunes on the pair directly.
• Preferred answers rise 35 to 78.
• Default DPO; explore with RLHF.
• Proxies reward length over help.