Batch Normalization
Introduced Batch Normalization, a technique that normalized activations across a mini-batch, making networks faster and dramatically more stable to train.
Paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Authors: Sergey Ioffe, Christian Szegedy · 2015
Read the paper
The Problem
Training deep neural networks was notoriously fragile. As the parameters of the early layers changed during training, the distribution of the inputs to the later layers kept shifting. The authors called this 'Internal Covariate Shift'. Because later layers constantly had to adapt to new input distributions, learning rates had to be kept painfully small, and careful weight initialization was required. This made training painfully slow.
The Idea
If standardizing the inputs to a neural network (e.g., zero mean, unit variance) helps the first layer learn faster, why not do that for every layer? The idea was to normalize the activations of intermediate layers so their distribution remained stable throughout training.
How It Works
Batch Normalization (BatchNorm) is inserted immediately after a fully connected or convolutional layer, and usually before the non-linearity (like ReLU).
For a given mini-batch of data during training, BatchNorm calculates the mean and variance of the activations. It then normalizes the activations by subtracting the mean and dividing by the standard deviation.
However, simply normalizing the data might restrict what the layer can represent. To fix this, BatchNorm introduces two learnable parameters: gamma (scale) and beta (shift). This allows the network to learn to undo the normalization if that's what's optimal for the task. During inference, since there are no mini-batches, it uses a running average of the mean and variance computed during training.
Why It Mattered
BatchNorm was a magic bullet for optimization. It made networks remarkably robust to bad initialization. It allowed for much higher learning rates, which meant networks trained an order of magnitude faster. It also provided a slight regularization effect, reducing the need for Dropout. It was so effective that it became an immediate, unquestioned default in almost every architecture built after 2015.
What Came After
BatchNorm dominated for years, particularly in Convolutional Neural Networks (CNNs). However, it had flaws: it performed poorly when batch sizes were very small, and it didn't work well for Recurrent Neural Networks (RNNs) or sequence models. This led to the development of alternative normalization techniques like Layer Normalization (which became the standard for Transformers) and Group Normalization.