Core ML

Dropout

Zero out a random slice of a layer on every training pass, and one set of weights quietly trains a huge family of thinned networks that vote at inference.

Why Does This Exist?

Twenty-eight thousand product reviews, one hidden layer of 512 units, training accuracy 99.4%. Feed it reviews from last month and it manages 84%. Nothing crashed. The layer learned those particular reviews instead of the language they were written in.

A hidden layer is a bank of detectors: each unit fires when it spots something worth reporting, and the next layer reads all 512 reports and makes the call. That's a multi-layer perceptron. With 512 free detectors against 28,000 examples there's room to memorise — overfitting.

But "it memorised" is too vague to fix. What goes wrong specifically is that units gang up. Unit 41 fires on the word "not", useless alone as a sentiment signal. Unit 88 learns a correction that only means anything when unit 41 fired. Together they nail 300 training reviews. That's a co-adapted pair, and a layer full of them is a set of private agreements that hold on the training set and snap everywhere else.

Think of It Like This

The kitchen where a third of the line calls in sick

Every service, a random third of your cooks don't show up, and you don't find out which third until the tickets start printing.

Run a kitchen that way for a year and nobody can be the only person who knows the sauce. Four people learn it instead, so the recipes stop living in individuals and start living in the room. You also aren't running one kitchen: each service is a different subset of the same staff, sharing one recipe book that every service improves.

Then the night the critic books, everybody shows up. Full line. Which is fine, except this kitchen has never cooked at full strength and is about to push out half again as much food as any training night. Somebody had better have thought about portions.

How It Actually Works

Sampling a thinned network on every pass

On each forward pass during training, for every unit, flip a coin. With probability $p$ , the drop rate, zero that unit's output for this pass — independently, per unit, per pass, and the backward pass routes gradient only through the survivors. A layer of $n$ units has $2^n$ on-off patterns, so 512 units gives $2^{512}$ thinned networks, and training visits a random handful of them, one per batch, each visit updating the same shared weights.

That's bagging with the expensive part removed. Bagging fits many independent models on resampled data and averages them, and it works because unshared errors cancel. Dropout's members share weights rather than being fitted separately, and you never average them explicitly: one full forward pass approximates the ensemble.

Now the co-adaptation story has teeth. Unit 88 can't build a correction leaning on unit 41, because unit 41 is gone on half the passes where unit 88 is present. Any strategy needing a specific partner gets punished, so what survives is a representation spread across the layer.

Why inference needs no adjustment

Say each of your 512 units outputs about 0.8 and you drop 30% of them. The sum reaching the next layer falls from roughly 410 to 287, and every downstream weight gets tuned against that smaller number. Switch dropout off and the sum snaps back to 410 — a scale the next layer was never calibrated on.

You could patch that at inference by multiplying every activation by $1-p$ , the keep probability. Nobody does. Frameworks patch it during training instead, dividing each survivor by $1-p$ so the expected sum lands where it would have with no dropout at all.

\mathbb{E}\big[\tilde{h}_i\big] = \frac{(1-p)\,h_i}{1-p} = h_i

$h_i$ is what unit $i$ would normally output, $\tilde{h}_i$ what it reports after dropout, and the expectation averages over the coin flips. That's inverted dropout, and it's why model.eval() is load-bearing rather than housekeeping. It turns dropout off.

Rates, and where it still earns a place

Inside transformer blocks, 0.1 to 0.3, on attention weights and feed-forward outputs. Historically 0.5 in wide fully-connected layers, where dropout started and where the number came from. Scale the rate with the width: dropping half of a 32-unit layer removes real capacity.

The trajectory is worth being honest about. In convolutional vision dropout has largely been displaced — batch normalisation and data augmentation do the regularising, and zeroing individual activations inside a spatially correlated feature map was never a clean fit. In transformer blocks it's still standard, and it composes with weight decay, which pulls on how large the weights are rather than which units are present.

Show Me the Code

The whole mechanism, plus the property that makes inference free: the expected sum doesn't move, only the per-pass spread grows.

1import numpy as np2
3
4def dropout(h: np.ndarray, p: float, training: bool, rng: np.random.Generator) -> np.ndarray:5    if not training:6        return h  # nothing to undo at inference — the rescaling already happened in training7    keep: np.ndarray = (rng.random(h.shape) > p) / (1.0 - p)  # inverted scaling8    return h * keep9
10
11rng = np.random.default_rng(0)12h: np.ndarray = np.full(512, 0.8)  # one hidden layer, every unit mildly active13
14for p in (0.0, 0.3, 0.5):15    passes = np.array([dropout(h, p, True, rng).sum() for _ in range(2000)])16    print(f"p={p}: train sum {passes.mean():6.1f} +/- {passes.std():4.1f}, eval sum {dropout(h, p, False, rng).sum():6.1f}")17    # -> p=0.0: train sum  409.6 +/-  0.0, eval sum  409.618    # -> p=0.3: train sum  409.5 +/- 12.0, eval sum  409.619    # -> p=0.5: train sum  410.1 +/- 17.9, eval sum  409.6

The mean holds at 410 for every rate. What moves is the spread: at $p=0.5$ a single pass lands about 18 away from a clean one. That noise is the regularisation, and it's why dropout usually needs more epochs before the loss settles.

Watch Out For

Evaluating a model that is still in training mode

The tell is a validation number that moves between two runs on identical data: 0.883, rerun unchanged, 0.891. That isn't noise in the data, it's dropout still firing — a random 30% of every layer missing from every prediction, so the score reads worse than the model deserves.

model.eval() before the validation loop, model.train() after. Wire a guard into the eval function rather than trusting yourself to remember, because there's no exception and no warning. The same switch freezes batch-norm statistics, so forgetting it costs twice.

Reading a training loss above the validation loss as a leak

Training loss 0.34, validation loss 0.29, and you start hunting for data leakage. Almost always there's nothing to find.

Training loss is measured on the thinned network, a third of every layer switched off; validation loss is measured on the whole network. Different models, so the inequality between them says nothing about leakage. Want them comparable? Evaluate the training set again in eval mode. If it still looks strange, then go looking.

The Quick Version

Each unit is zeroed independently with probability $p$ on every training pass, so every batch sees a different thinned network.
A layer of $n$ units defines $2^n$ subnetworks sharing one weight matrix, and the full network at inference approximates their average.
A cheap relative of bagging. The mechanism is anti-co-adaptation: no unit can lean on a specific other unit.
Inverted scaling divides survivors by $1-p$ during training, which is why inference needs no adjustment.
Eval mode is not optional. Forgetting it makes evaluation noisy and pessimistic.
0.1 to 0.3 in transformer blocks, 0.5 historically in wide fully-connected layers. Displaced in convolutional vision, still standard in transformers.