Batch Norm & Dropout

What you are looking at

On the left is the network: two inputs, two hidden layers of ten ReLU units, one sigmoid output. It is training on forty points drawn from two overlapping noisy rings, with three hundred more points held out that it never sees.

On the right, the loss plot. The solid curve is the loss on the forty points the network is fitting; the dashed one is the loss on the three hundred it isn't. Press Play and watch what the two do differently.

Both curves fall at first. Then, somewhere around epoch a hundred, the solid one keeps falling and the dashed one turns and starts climbing. Nothing broke. The network is still getting better at the forty points, which is all it was ever asked to do — it is just doing that by memorising them, and memorised points say nothing about the three hundred it hasn't seen. The distance between the two curves is the generalisation gap, and it is the readout to keep your eye on.

Dropout

Move to the second stage and switch Dropout on. Units in the hidden layers start going dark, redrawn small with a 0 inside them, and the edges into and out of them disappear for that step. Every training step draws a fresh mask, so a different subset goes dark each time.

The surviving units show a number too — at the default Drop Probability of 0.5, they show 2.0. That is the scale factor $1/(1-p)$, and it is why the technique is called inverted dropout. Zeroing half a layer roughly halves the total signal reaching the next one, so the survivors are multiplied up to put the expectation back. Raise the drop probability and watch that number climb with it.

The point of all this is to stop units co-adapting. If a unit can count on a particular neighbour always being there, the two can learn to work only as a pair, and a pair that fires on one specific training point is exactly what memorisation looks like. Take the neighbour away at random and neither can rely on it, so each has to carry a feature that stands on its own.

A team where anyone might be off sick

Imagine a team where every member specialises so narrowly that nothing ships unless all ten are in the room. It works, right up until one person is away.

Now imagine the manager sends a random half of the team home every single day. Nobody can build a workflow that depends on one particular colleague, because that colleague is gone half the time. Everyone ends up covering more ground than strictly necessary, the team is slower at any one task, and it never stops entirely. That redundancy is the whole product.

Watch what happens to the training loss when you switch dropout on: it gets clearly worse. That is expected, and reading it as a failure is the most common mistake here. The network is being handicapped on purpose. The number that decides whether the trade was worth it is the dashed curve — and the gap between the two, which is the readout that arrives with this stage.

At inference, dropout is switched off completely. Every unit stays on and nothing is scaled, because the training-time multiplication already corrected for the missing units.

Batch normalisation

The third stage adds the histogram. It shows the first hidden layer's pre-activations — the raw weighted sums, before ReLU — pooled across every unit and every sample in the batch. With Batch Norm off there is exactly one distribution, because there is nothing to compare it against yet.

Switch it on and a second one appears, and the two readouts underneath say what happened: the pre-BN spread stays around 1.8 and the post-BN spread becomes 1.000. Batch norm takes the mean and variance of each unit across the batch, subtracts the one and divides by the square root of the other:

$\hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$

Then it applies a learned scale $\gamma$ and shift $\beta$:

$y = \gamma\hat{x} + \beta$

Which looks like undoing the work. It isn't. Forcing every layer's input to mean 0 and variance 1 is a constraint, and a constraint the network cannot escape costs it capacity — so $\gamma$ and $\beta$ give it a way back out, including all the way back to the identity if that turns out to be best. In this lab they start at exactly 1 and 0, which is why the post-BN readouts open at exactly 0.000 and 1.000 and drift away from there as training proceeds.

Two consequences worth noticing, because both are the sort of thing that only shows up when you try to use it:

The batch statistics are taken across the batch, which means one sample's normalised value depends on the other samples it happened to arrive with. During training that coupling is part of why batch norm regularises. At prediction time it would be a bug — the same input would get different answers depending on its batchmates, and a single input on its own has no variance at all. So training keeps a running average of the statistics it has seen, and inference uses that instead.

The same coupling runs backward. A gradient arriving at one sample's pre-activation depends on every other sample in the batch, which is where the four-term backward formula on the reference card comes from.

Batch norm is not a licence for any learning rate

Set Learning Rate to 10 and leave Batch Norm off. The training loss peaks at 7.9 inside four epochs — past the 5 the plot can draw, which is why both readouts start flagging themselves — and the pre-activation spread blows out from 1.8 to 9.6.

Switch batch norm on at the same rate. The peak is 2.8 and the loss never crosses 5. This is the headline claim for batch norm, and it is real: normalising the input to every layer makes the loss surface better conditioned, so a step that was catastrophic becomes survivable.

Now push the learning rate to 15. This time it is the normalised run that blows up and the plain one that survives. Batch norm widened the range of learning rates that work; it did not remove the limit, and past that limit which configuration fails first stops being predictable. Any claim that batch norm lets you stop tuning the learning rate is describing the middle of the range and ignoring the end of it.

Which one do you reach for

They are not alternatives, and they are not interchangeable — this lab has one control for each because they move different numbers.

Dropout addresses overfitting. Its target is the generalisation gap, it costs you training loss on purpose, and it does nothing whatsoever about a learning rate that is too large.

Batch normalisation addresses conditioning. Its target is the spread of the values one layer hands the next, it makes training faster and more tolerant of a large learning rate, and it happens to regularise a little as a side effect of coupling samples in a batch. It will not save a network that has enough capacity to memorise its training set.

Turn both on and you get both effects, which is why they so often appear together — but if the gap is your problem, reaching for normalisation will not close it.