Gen AI

Classifier-Free Guidance

At each denoising step, the diffusion model predicts the image both with and without the text prompt, then pushes toward the prompt version by a set amount.

Why Does This Exist?

Diffusion models generate images by starting from noise and repeatedly asking a trained network what the clean picture underneath should look like. Hand it a text prompt, and each step gets nudged toward whatever the prompt describes. The problem: plain conditional generation — "here's the prompt, denoise toward it," no tricks — tends to come out timid. The prompt says "red bicycle," and the model has heard it, but hasn't really committed.

Classifier-free guidance fixes that timidity without retraining anything. It's a generation-time trick: run the model, then deliberately push its output further in the direction the prompt was already nudging it. Every text-to-image tool with a "guidance scale" or "CFG scale" slider is exposing exactly this knob.

Take "a red bicycle leaning against a blue wall," run at three scales. Very low, and you get something bicycle-shaped, wrong colors, a generic wall — the prompt barely registered. Moderate, and you get a clean, recognizable red bicycle against a blue wall. Push it much higher and the image turns aggressively red-and-blue, oversaturated, geometry starting to buckle. Same prompt, same model, three different images — one slider is the entire reason.

Think of It Like This

Two dinner opinions and a friend who overcorrects on the difference

Ask two friends what to order. One has no idea what you're in the mood for and gives a bland default: "get something." The other actually caught your prompt — "I'm craving something red and Italian tonight" — and says "get the marinara." The gap between those two answers is real information. It's specifically what your stated preference added to a generic guess.

Now imagine a third friend who takes that gap and travels past it, in the same direction: not just marinara, but the reddest, most tomato-drenched dish on the menu, on the theory that more of the same direction must mean more correct. A little of that overshoot and you get a livelier recommendation. A lot of it, and you stop getting Italian food you'd actually enjoy — you get a caricature of "red and Italian" instead.

How It Actually Works

Two predictions per step

At every denoising step, the same network runs twice, on the same noisy image. Once conditioned on the text prompt — "a red bicycle leaning against a blue wall" — and once with no conditioning at all, usually an empty string standing in for "no prompt." Same weights, same noisy input, two different noise predictions out — one guessing what to remove if it's honoring the prompt, the other guessing as if no prompt existed at all.

The extrapolation

The guided prediction that classifier-free guidance actually uses isn't the conditional one on its own. It's the unconditional prediction, plus the gap between conditional and unconditional, scaled by the guidance scale: guided = uncond + scale × (cond − uncond). At a scale of exactly 1, that formula collapses to the conditional prediction with nothing added — plain conditional generation, no extrapolation at all. Push the scale above 1 and the model deliberately overshoots: it moves further in the direction the prompt was already pulling it, past the point its own conditional prediction had already reached.

Why overshooting has a cost

That "further in the same direction" step is the whole mechanism, and also exactly where the artifacts come from. The conditional prediction already represents the model's honest, trained guess at "following the prompt." Extrapolate past that and you land somewhere the model was never confident about — nothing in training told it what a prediction several times past its own best guess should look like. A high guidance scale doesn't read as "trying harder." It reads as oversaturated color, blown-out contrast, and warped geometry: the signature of a prediction pushed past what the model actually learned, not evidence of it working better.

Show Me the Code

The same extrapolation formula, run at three guidance scales on one toy noise value, so the "past the conditional prediction" behavior is explicit.

1def cfg_extrapolate(cond: float, uncond: float, scale: float) -> float:2    """Move from the unconditional prediction toward, and past, the conditional one."""3    return uncond + scale * (cond - uncond)4
5
6cond = 0.8       # noise prediction with the prompt7uncond = 0.1      # noise prediction with no prompt8
9print(round(cfg_extrapolate(cond, uncond, 1.0), 2))   # -> 0.8 -- exactly the conditional prediction10print(round(cfg_extrapolate(cond, uncond, 3.0), 2))   # -> 2.2 -- past it, same direction11print(round(cfg_extrapolate(cond, uncond, 7.0), 2))   # -> 5.0 -- far past it, where artifacts start

At scale 1, the guided value is just cond. Scales above 1 land further past it, in the same direction the conditional prediction was already pointing — which is exactly the overshoot that produces artifacts once it goes too far.

Watch Out For

Cranking the guidance scale up assuming higher always means more faithful

Past a certain point, a higher guidance scale stops improving prompt adherence and starts introducing oversaturated colors, harsh contrast, and warped geometry instead — the red bicycle's frame bends, the blue wall turns nearly fluorescent. The model isn't obeying the prompt more at that point; it's being extrapolated into territory it was never trained to produce. Raise the scale to fix weak prompt adherence, but check the actual image past a moderate value instead of trusting the number alone.

Treating a guidance scale of 1 as 'no guidance'

A scale of 1 isn't "off" — it's plain conditional generation, with zero extrapolation, since 1 × (cond − uncond) added to uncond just gives back cond. Actual "ignore the prompt" behavior lives near 0 or below, where the guided prediction moves toward or past the unconditional one instead. Confusing these two settings when debugging a pipeline leads to the wrong conclusion about what's causing weak or strong prompt adherence.

The Quick Version

Diffusion models run twice per denoising step: once conditioned on the text prompt, once with no conditioning at all.
Classifier-free guidance computes the gap between those two predictions and extrapolates past the conditional one, scaled by the guidance scale.
A scale of 1 gives plain conditional generation with no extrapolation; scales above 1 deliberately overshoot in the prompt's direction.
Overshooting too far pushes the prediction into territory the model wasn't trained on, which is what produces oversaturated colors and warped geometry.
The right scale is a balance point between a prompt the model barely followed and one it followed so hard the image degrades.