Batch Normalization

In deep feedforward networks, parameter updates in early layers cause cascading shifts in the activation distributions fed to downstream layers. This phenomenon, termed internal covariate shift, forces deeper neurons to continuously adapt to an unstable, drifting input distribution. As downstream weights chase moving statistical targets, optimization slows and training becomes vulnerable to vanishing or exploding activations.

Center, Then Even Out

Batch Normalization stabilizes intermediate representations by standardizing activations across the mini-batch dimension. For a mini-batch $\mathcal{B} = \{0, 2, 4, 6\}$, the batch mean is $\mu_\mathcal{B} = 3.0$ and sample variance is $\sigma_\mathcal{B}^2 = 5.0$ ($\sigma_\mathcal{B} \approx 2.24$). Centering by subtracting $\mu_\mathcal{B}$ and scaling by $\sqrt{\sigma_\mathcal{B}^2 + \epsilon}$ transforms raw inputs into standardized coordinates $\{-1.34, -0.45, +0.45, +1.34\}$, enforcing zero mean and unit variance.

Reshape on Purpose

Constraining activations strictly to standard normal distributions risks trapping values within the linear regimes of activation functions like Sigmoid or Tanh. To preserve network expressiveness, Batch Normalization introduces two learnable parameters per channel: scale $\gamma$ and shift $\beta$, computing $y_i = \gamma \hat{x}_i + \beta$. For standardized value $\hat{x} = 0.45$ with learned $\gamma = 2.0$ and $\beta = 1.0$, the layer outputs $2.0 \times 0.45 + 1.0 = 1.90$, allowing the network to recover any arbitrary representation.

Freeze the Calm

During training, each layer tracks exponential moving averages of batch means and variances ($\mu_{run} \approx 3.0, \sigma_{run} \approx 2.24$). At test time, these running statistics are frozen. When evaluating an individual test sample, the network standardizes inputs against the fixed historical population statistics rather than requiring mini-batches, ensuring deterministic, reproducible inference for single inputs.

Stride Five Times Longer

Batch Normalization significantly improves optimization geometry. By controlling gradient magnitudes and smoothing the loss landscape's Lipschitz curvature, the network avoids ill-conditioned ravines. While unnormalized networks must crawl at cautious learning rates like $\eta = 0.01$ to avoid divergence, normalized models glide stably with five times larger step sizes ($\eta = 0.05$), accelerating convergence.

Mind the Batch Size

Because Batch Normalization relies on mini-batch statistics, batch size directly affects estimation fidelity. Batches of 32 or larger provide dependable population approximations across the vertical batch dimension. When hardware memory limits batches below 32, batch statistics turn volatile; models switch to Layer Normalization, which standardizes horizontally across features within each individual sample independently of batch size.

Where It Breaks

The mathematical formulation fails entirely when the training batch size collapses to one ($m = 1$). A single sample equals its own mean ($x_1 = \mu$), producing a sample variance of zero ($\sigma^2 = 0$). Standardizing divides zero by $\sqrt{\epsilon}$, collapsing the activation $\hat{x}$ to zero and forcing output $y = \beta$. All input information is extinguished, completely halting forward and backward information flow.

The Quick Version

• Deep layers chase inputs that drift after every weight update.
• Mini-batch standardization enforces zero mean and unit variance.
• Learnable scale and shift restore full representational capacity.
• Frozen running statistics enable deterministic single-sample testing.
• Smoothed loss landscapes permit five-times-larger learning steps.
• Maintain batch size above 32 or switch to layer normalization.
• Single-sample training batches collapse variance to zero and fail.