Dropout
How randomly turning off neurons prevents neural networks from memorizing data. See the training/eval asymmetry and the ensemble effect.
Without regularization, deep networks are lazy. A few neurons memorize the exact training data, and the rest of the network learns to rely completely on those few neurons. This "co-adaptation" means the model fits the training set perfectly but fails on new data.
The Dropout Mechanism
Dropout fixes this by randomly turning off a percentage of neurons (e.g., 50%) during every forward pass of training. Because no neuron can rely on its neighbors being there, every neuron is forced to learn useful, redundant features on its own.
The Evaluation Pass
At inference, you don't drop neurons. You need the full network. However, because twice as many neurons are now active (if drop rate was 50%), the total signal is twice as large. To compensate, the weights are scaled down by the retention probability (e.g., multiplying by 0.5) so the next layer sees the expected sum.
The Ensemble Effect
By dropping different neurons on every step, you are actually training millions of different sub-networks that all share weights. At inference time, using the full network is mathematically equivalent to taking an ensemble average of all those sub-networks, which vastly improves robustness.
Where It Breaks
If you forget to switch your model to evaluation mode (e.g., failing to call .eval() in PyTorch), dropout remains active. Your model will drop random neurons during inference, injecting massive noise and completely ruining the predictions.
The Quick Version
- Networks naturally overfit by letting a few neurons memorize the data (co-adaptation).
- Dropout randomly zeroes neurons during training to force redundancy.
- At evaluation, all neurons are used, but their outputs are scaled down to maintain the expected sum.
- Dropout acts as a massive ensemble of sub-networks sharing the same weights.
- Leaving dropout active during inference injects random noise and destroys accuracy.