Batch Norm vs Layer Norm
Both stabilize training by normalizing activations — they just disagree about which axis to normalize over.
Batch NormvsLayer Norm
Verdict: Layer norm for sequences and transformers; batch norm for CNNs with large batches.
The core difference
Batch normalization normalizes each feature across the batch dimension, so its statistics depend on the other examples in the batch. Layer normalization normalizes across the feature dimension within a single example, so it is independent of batch size.
Side by side
| Dimension | Batch Norm | Layer Norm |
|---|---|---|
| Normalizes over | Batch | Features |
| Batch-size sensitive | Yes | No |
| Train vs eval behavior | Differs (running stats) | Identical |
| Great for | CNNs | RNNs, transformers |
| Small batches | Struggles | Fine |
When to choose which
- Choose batch norm for convolutional networks trained with reasonably large batches.
- Choose layer norm for variable-length sequences, transformers, or tiny batches where batch statistics are noisy.
Bottom line
Pick the normalization whose "population" is stable for your setup. Transformers use layer norm precisely because per-example feature statistics don't depend on how you batched.