Pre-Norm vs Post-Norm

Why Does This Exist?

The transformer block wraps both its sub-layers in a residual connection and a normalization step, but that description leaves out one detail that turns out to matter enormously: where exactly does the normalization go relative to the addition? The original transformer paper placed it after the residual addition — normalize the sum of the input and the sub-layer's output. That ordering, now called post-norm, is what the paper actually shipped, and for years it was simply "how transformers are normalized."

Then researchers training deeper transformer stacks kept hitting a specific wall: past a certain depth, post-norm transformers became difficult or impossible to train without a very careful, slow learning-rate warmup, and sometimes not even then. The fix wasn't a new normalization formula — it was moving the existing normalization to a different position in the block, which changed how gradients flow through the stack in a way the formula alone doesn't reveal.

Think of It Like This

How It Actually Works

Two orderings, one formula each

Post-norm, as shown on the left of the diagram, normalizes after the residual addition:

Pre-norm, on the right, normalizes only the sub-layer's input, leaving the addition itself untouched:

The difference looks small — a normalization step moved from one side of an addition to the other — but it changes something structural: in pre-norm, the raw input $x$ skips straight through to the final addition, completely unmodified by any normalization. In post-norm, the entire running sum — the thing every earlier layer contributed to — gets renormalized at every single layer.

Why the raw residual stream matters

Recall from residual connections why the identity path was the whole point: it gives gradients a route back to early layers that doesn't have to pass through any nonlinearity or transformation, however deep the stack gets. Post-norm's placement of the normalization after the addition inserts a normalization operation directly into that identity path, at every layer — the "untouched" shortcut residual connections were supposed to provide isn't actually untouched anymore, because the sum it feeds into gets renormalized before continuing to the next layer.

Pre-norm avoids this precisely: the term that skips straight through to the next addition is the raw, unnormalized $x$, exactly as residual connections intended. Normalization still happens, but only on the copy that feeds into the sub-layer computation, never on the running sum that carries gradient information down the stack. That preserved, untouched identity path is why pre-norm transformers train reliably at far greater depth, and with far less sensitivity to learning-rate warmup, than post-norm ones.

The cost, stated honestly

Pre-norm's stability isn't free. Because the residual stream in a pre-norm model can grow in magnitude across many layers — each layer adds its correction to an ever-growing sum without ever renormalizing that sum directly — very deep pre-norm stacks can eventually face their own numerical issues from that unbounded growth, and some architectures address this with an additional final normalization layer at the very end of the whole stack. Pre-norm trades post-norm's training instability for a different, generally more manageable problem.

QK-normalization, a related fix at the same layer

A related technique, QK-normalization, normalizes the query and key vectors themselves — before the $QK^\top$ dot product inside attention — rather than normalizing the block's input or output. It targets a different failure mode: attention logits growing unstably large as models scale up, independent of which pre-norm or post-norm convention the surrounding block uses. It's worth knowing as a sibling fix living at the same architectural layer, not a replacement for the pre-norm choice.

Show Me the Code

Both orderings, computed explicitly, to see exactly which term is normalized in each.

1import numpy as np2
3
4def layer_norm(x: np.ndarray, eps: float = 1e-5) -> np.ndarray:5    mean, var = x.mean(axis=-1, keepdims=True), x.var(axis=-1, keepdims=True)6    return (x - mean) / np.sqrt(var + eps)7
8
9def post_norm_block(x: np.ndarray, sublayer_fn) -> np.ndarray:10    return layer_norm(x + sublayer_fn(x))          # the WHOLE sum gets normalized11
12
13def pre_norm_block(x: np.ndarray, sublayer_fn) -> np.ndarray:14    return x + sublayer_fn(layer_norm(x))           # only the sub-layer's INPUT is normalized15
16
17rng = np.random.default_rng(0)18x = rng.normal(size=(4, 8))19identity_sublayer = lambda z: 0.1 * z                # stand-in for attention or FFN20
21post = post_norm_block(x, identity_sublayer)22pre = pre_norm_block(x, identity_sublayer)23print(np.allclose(pre - x, 0.1 * layer_norm(x)))     # -> True — x survives untouched in pre-norm24print(np.allclose(post, x))                          # -> False — x itself is gone after post-norm

pre - x isolates exactly what got added to the original input, and it's a function purely of the sub-layer's output — x itself is never inside a layer_norm call in the pre-norm path, which post_norm_block cannot say.

Watch Out For

The Quick Version

• Post-norm normalizes after the residual addition, so the entire running sum gets renormalized at every layer.
• Pre-norm normalizes only the sub-layer's input, leaving the raw residual stream to pass through untouched.
• Post-norm's placement disrupts the clean identity path residual connections are meant to provide; pre-norm preserves it.
• Pre-norm trains far more reliably at depth and needs less careful learning-rate warmup, which is why it became the modern default.
• Pre-norm trades that stability for potentially unbounded residual-stream growth at extreme depth, a separate, generally more manageable issue.

What to Read Next

• Transformer Architecture is where this normalization choice is applied, around both sub-layers of every block.
• Feed-Forward Networks is one of the two sub-layers this page's normalization wraps.
• Self-Attention is the other sub-layer, and the one QK-normalization operates inside of.
• Residual Connections is the identity-path mechanism this page's entire argument depends on preserving.