Classifier-Free Guidance (CFG)

The Problem

Early diffusion models could generate unconditional images (random faces) or class-conditioned images (a "dog"), but they struggled with complex text prompts. To force the model to follow a text prompt, researchers originally used "classifier guidance": they trained a separate, noisy image classifier to push the diffusion model in the right direction. But training a classifier on noisy images was extremely difficult, expensive, and didn't scale well to open-ended text.

The Idea

Google researchers proposed Classifier-Free Guidance (CFG). Instead of using a separate classifier, they realized they could use the diffusion model itself as the classifier. By occasionally dropping the text prompt during training (e.g., 10% of the time replacing the text with an empty string), the model learns both a conditioned distribution (images matching text) and an unconditional distribution (generic images). During generation, the model predicts the noise twice: once with the text, once without. It then extrapolates away from the unconditional prediction towards the conditioned prediction, forcing the image to strongly adhere to the text.

How It Works

At generation step $t$, the model makes two predictions:

1. $\epsilon_{cond}$: The noise predicted given the text prompt.
2. $\epsilon_{uncond}$: The noise predicted given an empty prompt.

The final noise used to update the image is calculated using the guidance scale $w$: $\epsilon_{final} = \epsilon_{uncond} + w \cdot (\epsilon_{cond} - \epsilon_{uncond})$

If $w = 1$, it is standard generation. If $w = 7.5$ (a common default), the model takes the vector pointing from the generic image toward the prompted image, and multiplies its strength by 7.5. This violently forces the image to manifest the concepts in the text prompt, at the cost of some image diversity.

Why It Mattered

CFG is the magic ingredient that makes text-to-image models actually follow the prompt. Without it, models like Stable Diffusion or Midjourney would mostly ignore complex prompts and generate generic, safe images. It provided a simple "knob" (the CFG scale) for users to control how strongly the model should obey the text.

What Came After

CFG became universally adopted across all diffusion models (image, video, and audio). While effective, it requires running the massive neural network twice per step (doubling the compute cost). Subsequent research focused on "distilling" CFG into a single step or finding alternative guidance mechanisms to regain that lost speed.