Classifier Guidance Steers Diffusion

Classifier guidance steers a diffusion sampler with gradients from a noisy image classifier, pushing each step toward the requested class.

A noisy classifier gradient nudges each denoising mean toward the target class, and the scale knob sets the push strength

Why Does This Exist?

Unconditional diffusion draws from the whole data distribution at once. Asking it for a golden retriever means hoping one appears, and retraining a conditional model per control is prohibitively expensive. Classifier guidance bolts steering onto a frozen model: one big unconditional sampler plus a small classifier per control. This page covers the gradient-steered original; the successor that drops the extra classifier lives in classifier-free guidance, and the sampler loop it plugs into in diffusion samplers.

The one idea you need first

A reverse diffusion step cleans a noisy sample $x_t$ slightly, predicting a mean $\mu(x_t)$ for the cleaner sample. Run that loop from pure noise and an image emerges. Guidance intervenes inside each single step: before sampling, shift the mean a little toward whatever scores higher under a classifier for the target class. Small shifts compound over dozens of steps into a different image.

Think of It Like This

A bloodhound leading a painter

A painter works half-blind, sketching whatever emerges. A bloodhound beside them sniffs each stroke and tugs the brush toward the scent of retriever. A gentle leash lets the painter compose freely with a nudge; a yanked leash drags every stroke onto the same canonical dog pose.

The guidance scale is the leash tension, and the analogy stops at training: a real bloodhound arrives pre-trained, while this one must learn every noise level or its tugs point nowhere.

How It Actually Works

At reverse step $t$ with current sample $x_t$ , the unconditional model predicts mean $\mu(x_t)$ . A noise-aware classifier $p_\phi(y \mid x_t)$ scores the target class $y$ . The guided step shifts the mean along the classifier gradient:

\hat{\mu} = \mu(x_t) + s \cdot \Sigma \, \nabla_{x_t} \log p_\phi(y \mid x_t)

$s$ is the guidance scale, $\Sigma$ the step covariance. At $s = 0$ sampling is unconditional; $s = 1$ follows the classifier's natural slope; production values often land near $s = 5$ to $10$ for crisp class features at visible diversity cost. The classifier must train on the same noise schedule, because a clean-image classifier's gradients point meaninglessly on heavy noise.

Worked example

Mid-sampling, the classifier gives target probability $0.7$ with gradient $0.4$ in the class direction, and $\Sigma$ scales steps by $0.02$ . At $s = 1$ the nudge is $1 \times 0.02 \times 0.4 = 0.008$ per step, gentle pressure that compounds over fifty steps. At $s = 8$ the nudge is $8 \times 0.02 \times 0.4 = 0.064$ per step, enough to drag marginal samples onto canonical class features within a dozen steps, and enough to collapse variety if every sample takes the same ride.

Code

Guided mean update, the arithmetic inside one reverse step:

1def guided_mean(mu: float, grad: float, sigma: float = 0.02, s: float = 1.0) -> float:2    """Shift the predicted mean along the classifier gradient."""3    return mu + s * sigma * grad4
5
6print(round(guided_mean(0.5, 0.4, s=1.0), 3))  # -> 0.508 gentle pressure7print(round(guided_mean(0.5, 0.4, s=8.0), 3))  # -> 0.564 firm steering

Watch Out For

Guiding with a clean-image classifier

A classifier trained on pristine photos has never seen noise, so its gradients on $x_t$ at high noise are arbitrary directions that shred composition. Samples degenerate into adversarial textures that maximise classifier score while looking like static. Always train or fine-tune the classifier across the diffusion noise schedule.

Cranking scale until diversity dies

High $s$ prints the classifier's favourite exemplar repeatedly. Benchmarks scored at $s = 10$ flatter fidelity metrics while coverage collapses. Sweep $s$ and plot fidelity against diversity, then ship the knee of the curve rather than the fidelity maximum.

The Quick Version

Classifier guidance adds $s \cdot \Sigma \nabla \log p(y \mid x_t)$ to each denoising mean.
The classifier must train on the same noise schedule as the diffusion model.
A worked step moves 0.508 at $s = 1$ versus 0.564 at $s = 8$ .
The scale knob trades class crispness against sample variety.
Clean-image classifiers and maxed-out scales are the two standard failures.