Skip to content
AI360Xpert
Core ML
Visual explainer

Batch Normalization

How standardizing layer inputs stabilizes deep networks. See the mechanism, the train/eval asymmetry, and why it breaks on small batches.

Internal covariate shift: the distribution of inputs to deep layers drifts continuously as earlier weights update.
Internal covariate shift: the distribution of inputs to deep layers drifts continuously as earlier weights update.

As a network trains, the weights in the first layer change. This alters the distribution of outputs fed into the second layer, which changes the inputs to the third layer. This constant drifting, called internal covariate shift, makes deep networks incredibly hard to stabilize.

Scale and Shift

Batch Norm normalizes each feature across the batch, then applies a learned scale and shift.
Batch Norm normalizes each feature across the batch, then applies a learned scale and shift.

Batch Normalization intercepts this drift. It forces the inputs of a layer to have a mean of zero and a variance of one across the current batch. It then applies a learned scale (gamma) and shift (beta), allowing the network to recover the exact distribution it actually needs.

The Train/Eval Asymmetry

Training relies on the current batch's statistics, but evaluation uses running averages.
Training relies on the current batch's statistics, but evaluation uses running averages.

During training, the mean and variance are calculated from the live batch. During evaluation, there might only be one example, so the network uses a running global average collected during training. This asymmetry is the source of many deployment bugs.

Where It Breaks

BatchNorm relies on batch statistics, so small batches produce noisy estimates, and batch size 1 breaks it completely.
BatchNorm relies on batch statistics, so small batches produce noisy estimates, and batch size 1 breaks it completely.

Because it normalizes across the batch dimension, the batch must be large enough to provide stable statistics. If the batch size is 2, the variance is pure noise. If the batch size is 1, the variance is mathematically undefined, and training crashes.

The Fix: Layer Norm

Layer Norm normalizes across features instead of the batch, making it independent of batch size.
Layer Norm normalizes across features instead of the batch, making it independent of batch size.

To fix the batch size dependency, alternatives were created. Layer Normalization calculates the mean and variance across the features of a single example, completely ignoring the rest of the batch. This is why Layer Norm is the standard for Transformers and RNNs.

The Quick Version

  • Deep networks suffer from internal covariate shift as layer input distributions drift.
  • Batch Norm standardizes inputs to mean zero, variance one, then applies a learned scale and shift.
  • Training uses live batch statistics; evaluation uses running global averages.
  • Batch Norm breaks down on small batches and crashes on a batch size of 1.
  • Layer Norm fixes this by normalizing across features instead of the batch.

What to Read Next