Core ML

Vanishing Gradients

Backpropagation multiplies local derivatives layer by layer, and when those derivatives are consistently below one, the product shrinks toward zero the deeper it travels.

Each layer's backward pass multiplies the incoming gradient by a local derivative under one; thirty such multiplications in a row leave the earliest layer with a gradient forty orders of magnitude smaller than the last

Why Does This Exist?

Train a 30-layer sigmoid network to classify handwritten digits and something strange happens: the last few layers learn fine, loss drops, accuracy improves — but the first ten layers barely move, weights sitting close to their random initial values the entire run. It isn't a code bug. Backpropagation computes exactly the gradient the math says it should; that gradient is just a genuinely, catastrophically small number by the time it reaches those early layers.

Here's why. Backpropagation's whole trick is the chain rule: the gradient reaching any one layer is the gradient from the layer after it, multiplied by that layer's own local derivative. Chain thirty of those multiplications together, and if the typical local derivative is consistently below 1 — sigmoid's derivative peaks at exactly 0.25 and falls fast on either side of zero — the product shrinks with every layer it passes through. Not linearly. Exponentially. Thirty small numbers multiplied together isn't thirty times smaller than one; it's smaller by a factor that compounds, the same way a bank balance compounds interest, except downward and every single layer.

Think of It Like This

A message passed by thirty people, each one quieter than the last

Thirty people stand in a line, and a message gets whispered from the far end down to you. Each person, for whatever reason, repeats what they heard at only 80 percent of the volume they received it at — not because they're trying to be quiet, just a fact about how each of them whispers.

Person 30 hears the original message clearly and passes on 80 percent of its volume. Person 29 passes on 80 percent of that. By the time it reaches person 1, the volume has been multiplied by 0.8 twenty-nine times over — a number so small that person 1 hears something indistinguishable from silence, even though everyone in the chain did exactly what they were supposed to do.

Nobody in the middle made an error. The compounding itself, repeated enough times, is the entire problem — and it's exactly what happens to a gradient threading backward through a stack of layers whose local derivatives are each a little less than one.

How It Actually Works

The multiplication, made explicit

For a chain of layers, the gradient of the loss $L$ with respect to an early layer's pre-activation $z_1$ is a product of every local derivative between there and the output:

\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial z_T} \prod_{t=2}^{T} \frac{\partial z_t}{\partial z_{t-1}}

$z_T$ is the final layer's pre-activation, $T$ the depth, and each factor in the product is one layer's local derivative — for a layer with a sigmoid activation, that factor is $\sigma'(z) = \sigma(z)(1-\sigma(z))$ , which peaks at 0.25 when $z=0$ and falls toward zero as $z$ moves away from zero in either direction. Once a unit's pre-activations sit in that flatter region — saturated, in the usual term — its local derivative is small, and it stays small for every layer stacked below it.

Why depth turns a small number into a catastrophic one

A single factor of 0.25 isn't alarming on its own. The problem is that this same multiplication happens at every layer on the way back, so the effect compounds: thirty layers with a typical local derivative around 0.05 — a realistic value once several units are meaningfully saturated — multiply down to roughly $0.05^{30}$ , a number with about forty leading zeros after the decimal point. That's not a rounding error changing behavior slightly; it's a gradient so small that the corresponding weight update does effectively nothing, and the layer that received it never learns.

The three fixes, and what each one targets

Better weight initialization keeps pre-activations away from the saturated region at the start of training, so the local derivatives begin closer to 0.25 rather than near zero. Non-saturating activations — ReLU and its variants — have a local derivative of exactly 1 everywhere they're active, which removes the multiplicative shrinkage entirely for the units that fire. Residual connections attack the structure of the multiplication itself rather than any single factor in it: the +x skip path gives the gradient an additional route that bypasses the layer's own derivative altogether, so even a layer with a near-zero local derivative doesn't fully block the signal from reaching everything below it.

Show Me the Code

Thirty saturated sigmoid layers, and the gradient's magnitude at four depths, computed directly from the chain-rule product above.

1import numpy as np2
3
4def sigmoid_grad(z: np.ndarray) -> np.ndarray:5    s = 1.0 / (1.0 + np.exp(-z))6    return s * (1.0 - s)7
8
9rng = np.random.default_rng(0)10z = rng.normal(3.0, 0.5, size=30)  # pre-activations sitting in the saturated region11local = sigmoid_grad(z)12print(f"mean local derivative: {local.mean():.4f}")  # -> 0.050513
14g = 1.015for depth, factor in enumerate(local, start=1):16    g *= factor17    if depth in (5, 10, 20, 30):18        print(f"gradient after {depth:2d} layers: {g:.2e}")19# -> gradient after  5 layers: 1.70e-0720# -> gradient after 10 layers: 2.27e-1421# -> gradient after 20 layers: 5.38e-2722# -> gradient after 30 layers: 1.79e-40

Ten layers already costs seven orders of magnitude. By thirty, the gradient reaching that layer is smaller than a single bit of a 32-bit float can represent — not approximately zero for training purposes, but numerically zero.

Watch Out For

Blaming the data or the learning rate for early layers that never move

Training loss falls, later layers clearly learn, and the instinct is to raise the learning rate or collect more data, because nothing looks obviously broken. Neither fix touches the actual cause: the gradient reaching those early layers is many orders of magnitude smaller than the one reaching the output, so any single global learning rate that's sane for the last layer is far too small to move the first one in a reasonable number of steps.

The tell is checking each layer's gradient magnitude directly rather than only watching the loss curve — a healthy network has gradients within a couple of orders of magnitude of each other across depth; a vanishing-gradient network has them spanning tens of orders of magnitude, worse the earlier the layer.

Assuming ReLU alone eliminates the problem entirely

Switching every activation to ReLU is the single highest-leverage fix, since a firing ReLU unit has a local derivative of exactly 1 rather than something under 0.25. But a ReLU unit that has stopped firing — pushed permanently negative by a bad update — has a local derivative of exactly 0, which is its own way of blocking gradient flow, sometimes called a "dead unit." Very deep ReLU networks can still benefit from careful initialization and residual connections on top of the activation choice, rather than treating ReLU as a complete substitute for either.

The Quick Version

Backpropagation multiplies local derivatives together across every layer in the backward path; consistently small local derivatives shrink the product exponentially with depth.
Saturating activations like sigmoid and tanh produce small local derivatives whenever a unit's pre-activation sits far from zero — the region called saturation.
The shrinkage is multiplicative and compounding, so ten extra layers can cost many more orders of magnitude than the first ten did.
Better initialization, non-saturating activations, and residual connections each attack a different piece of the same multiplication.
Check per-layer gradient magnitudes directly — a healthy network's gradients stay within a couple of orders of magnitude across depth.