Stable Diffusion

Stable Diffusion generates detailed images on a single GPU by denoising inside a tiny compressed latent space steered by frozen text embeddings.

Stable Diffusion compresses a 512 pixel image to a 64 by 64 latent once, denoises that small grid for fifty text-guided steps, then decodes to pixels once.

Why Does This Exist?

Pixel-space diffusion produced gorgeous images on hardware almost nobody owned. Dozens of full-resolution network passes per image meant data-center GPUs and minute-long waits, a research toy rather than a tool. Open, fast, cheap generation needed the cost structure broken, not the quality.

Stable Diffusion broke it with two moves: run diffusion in a frozen autoencoder's latent space, and steer with frozen CLIP text embeddings through cross-attention. The $48$ -fold per-step saving put $512$ -pixel generation on consumer cards, and open weights put it everywhere. The general denoising idea lives in diffusion models, the broader field in image generation, and prompt steering in classifier guidance. This page covers the compress-denoise-decode loop, the exact cell arithmetic, and the two failure modes that waste the most debugging time.

Think of It Like This

A mural painted on a postage stamp, then projected

An artist must paint a huge mural but rents a tiny studio. The fix: paint the whole composition on a postage stamp where every brushstroke is cheap, following written instructions slipped under the door, then project the finished stamp onto the wall with a perfect enlarger exactly once.

The latent grid is the stamp, the text prompt is the instruction slip, and the decoder is the projector. Quality survives because the stamp holds the composition faithfully.

The analogy stops at the enlarger. A real projector adds no detail of its own, while the VAE decoder reconstructs texture from its training, so fine detail the stamp never stored comes back blurry no matter how long denoising runs.

How It Actually Works

The three pieces

Three components cooperate. The VAE encoder compresses $512 \times 512 \times 3$ pixels to a $64 \times 64 \times 4$ latent exactly once. A CLIP text encoder converts the prompt to $77$ token embeddings exactly once. The trained UNet then denoises the latent for about $50$ steps, reading text through cross-attention at every resolution block, and the VAE decoder expands the finished latent to pixels exactly once. Compress once, denoise small fifty times, decode once. Model lineage runs v1 at $512$ pixels, v2 at $768$ with a stronger text encoder, and SDXL at $1024$ with two text encoders and a larger UNet.

Worked example: pricing one generation

Count processed cells, where a cell is one value in the grid:

\text{cells}_{\text{pixel}} = 512^2 \times 3 = 786{,}432, \qquad \text{cells}_{\text{latent}} = 64^2 \times 4 = 16{,}384

The ratio is exactly $48$ . Over $50$ DDIM steps the latent grid touches $50 \times 16{,}384 = 819{,}200$ cell-visits while pixel space would touch $50 \times 786{,}432 = 39{,}321{,}600$ . Same step count, same architecture family, $48$ times less arithmetic, which is the gap between a desktop card finishing in seconds and a cluster job. Cross-attention adds the prompt cost, $77$ text tokens attending into each spatial block, small beside the denoising itself.

Code

1def generation_cells(px=512, latent=64, steps=50):2    """Total cell visits for pixel-space versus latent-space generation."""3    pixel = px * px * 3 * steps4    lat = latent * latent * 4 * steps5    return pixel, lat6
7
8print(generation_cells())9# -> (39321600, 819200)

Watch Out For

Blaming the diffusion steps for VAE blur

Mushy fine detail, garbled small text, and smeared fingers often come from the frozen autoencoder's $8$ -fold spatial compression, not from too few denoising steps. Raising steps from $50$ to $200$ cannot recover detail the latent never stored. Diagnose by encoding and decoding a real photo with no diffusion at all: if the round trip already loses the detail, no sampler setting brings it back.

Prompt words the text encoder never learned

The CLIP encoder freezes old vocabulary and old associations. Rare names, new slang, and precise counts fall through to nearest-known tokens, and cranking guidance only amplifies the wrong interpretation. Test the encoder's reading with the bare noun first: if the base concept is wrong, rephrase into words the encoder knows rather than raising guidance strength.

The Quick Version

Stable Diffusion denoises a $64 \times 64 \times 4$ latent, not $512 \times 512 \times 3$ pixels.
Per-step arithmetic drops $48$ -fold: $819{,}200$ versus $39{,}321{,}600$ cell-visits over $50$ steps.
Frozen CLIP embeddings steer every UNet block through cross-attention over $77$ tokens.
Lineage v1, v2, SDXL climbs $512$ to $768$ to $1024$ pixels with stronger text encoders.
VAE compression, not step count, usually causes blur, and unknown prompt words need rephrasing.