Gen AI

Preference Optimization Variants

DPO proved we don't need a Reward Model to train preferences. Newer variants like ORPO, IPO, and SimPO strip the process down further, attempting to remove the massive VRAM penalty of needing a reference model in memory.

Different optimization variants aim to reduce the memory footprint by eliminating the reference model or changing the data requirements.

Why Does This Exist?

Direct Preference Optimization (DPO) was a breakthrough because it completely eliminated the Reward Model from RLHF. However, it did not solve the memory problem.

To train DPO, you must load the model being trained (the Policy Model) and a frozen copy of the model as it was before training started (the Reference Model). If you are tuning a 70B model, keeping a second 70B model frozen in memory just to compute KL-divergence penalties is an enormous waste of expensive VRAM.

Researchers immediately began searching for mathematically sound ways to align models without the reference model, or without needing strictly paired data (where Answer A is explicitly better than Answer B). This resulted in an alphabet soup of variants: ORPO, IPO, KTO, and SimPO.

The Variants Explained

1. ORPO (Odds Ratio Preference Optimization)

The breakthrough: It eliminates the reference model entirely. How it works: ORPO fundamentally alters the initial Supervised Fine-Tuning (SFT) phase. Instead of doing SFT first and preference optimization second, ORPO combines them. It trains the model to maximize the likelihood of the chosen response (standard SFT), while simultaneously adding a penalty term that pushes down the odds of the rejected response. Why it matters: Because there is no reference model, ORPO cuts VRAM requirements in half compared to DPO, making it a favorite for teams fine-tuning large models on limited hardware.

2. KTO (Kahneman-Tversky Optimization)

The breakthrough: It eliminates the need for pairwise preference data. How it works: Getting humans to meticulously rank "A is better than B" is expensive. KTO works on unary data—a simple thumbs up or thumbs down. The math is based on Prospect Theory (from psychology): it heavily penalizes the model for generating a "thumbs down" response when it could have done better, but offers diminishing returns for generating a "thumbs up" response when it was already likely to do well. Why it matters: You can use cheap, massive datasets of upvotes and downvotes (like Reddit data or customer feedback logs) without having to format them into strict A/B pairs.

3. IPO (Identity Preference Optimization)

The breakthrough: It fixes DPO's tendency to overfit and degrade in quality. How it works: Standard DPO pushes the probability of the winning answer up and the losing answer down to infinity. Over time, the model becomes too confident and breaks (it starts generating repetitive nonsense). IPO changes the loss function to bound this optimization, acting as a mathematical regularizer. Why it matters: It is used when DPO training runs become unstable over many epochs.

4. SimPO (Simple Preference Optimization)

The breakthrough: Like ORPO, it eliminates the reference model. It does this by using a length-normalized reward formulation directly inside the policy model. How it works: DPO implicitly prefers longer answers (a known flaw). SimPO explicitly normalizes the probabilities by the length of the generated sequence, preventing the model from hacking its own loss function by just generating more tokens. Why it matters: SimPO is rapidly gaining traction as a highly stable, memory-efficient alternative to DPO that natively resists the "long responses are better" bias.

Watch Out For

Treating them as silver bullets

None of these variants strictly outperform standard PPO (RLHF) at massive scale. OpenAI and Anthropic still use PPO for their frontier models. ORPO, DPO, and SimPO are primarily tools for democratizing alignment, allowing teams with 4 GPUs to do what used to require 16 GPUs.

The Quick Version

DPO: Needs paired data and a frozen reference model in VRAM.
ORPO & SimPO: Need paired data but eliminate the reference model, cutting memory costs in half.
KTO: Eliminates the need for paired data, allowing alignment using simple thumbs up / thumbs down signals.
IPO: Fixes DPO's tendency to over-optimize and break.