Skip to content
AI360Xpert
Core ML
Visual explainer

Vanishing Gradients

Why deep networks used to be impossible to train. See how multiplying small derivatives destroys the error signal, and how modern architecture fixes it.

The chain rule dictates that error signals must be multiplied layer by layer as they travel backward.
The chain rule dictates that error signals must be multiplied layer by layer as they travel backward.

To update a weight in the first layer, the network must pass the error signal backward through every layer above it. Because of the chain rule in calculus, this means multiplying the error by the derivative of every layer it passes through.

The Sigmoid Trap

The maximum slope of a Sigmoid function is 0.25, meaning at least 75% of the gradient is lost at each layer.
The maximum slope of a Sigmoid function is 0.25, meaning at least 75% of the gradient is lost at each layer.

Historically, networks used the Sigmoid activation function. But the maximum derivative (slope) of a Sigmoid is 0.25. When you multiply a number by 0.25 again and again, it shrinks exponentially.

The Wall

In deep networks, early layers receive effectively zero gradient and stop learning entirely.
In deep networks, early layers receive effectively zero gradient and stop learning entirely.

In a 10-layer network using Sigmoid, the error signal reaching the first layer is multiplied by 0.25100.25^{10}, which is practically zero. The late layers learn quickly, but the early layers are frozen. Because the late layers depend on the early ones extracting good features, the whole network fails.

The Modern Fix

The modern fix relies on ReLU activations (slope of 1) and residual connections to provide gradient superhighways.
The modern fix relies on ReLU activations (slope of 1) and residual connections to provide gradient superhighways.

The vanishing gradient problem was solved by two massive architectural shifts. First, replacing Sigmoid with ReLU, which has a constant slope of 1 for positive inputs, preventing the multiplication decay. Second, adding Residual Connections (ResNets) that provide a direct "superhighway" for the gradient to bypass layers entirely.

Where It Breaks

If an architecture is too deep for its chosen activation function, the loss flatlines early and training fails completely.
If an architecture is too deep for its chosen activation function, the loss flatlines early and training fails completely.

If you try to train a deep recurrent neural network (RNN) on a long sequence, or use the wrong activation function in a deep feedforward network, the loss will drop slightly and then flatline. The model hasn't converged; it has just lost its gradient signal and died.

The Quick Version

  • The chain rule requires multiplying derivatives to pass errors backward.
  • Sigmoid's maximum slope is 0.25, which destroys the gradient exponentially.
  • Vanishing gradients freeze the early layers, stopping the whole network from learning.
  • ReLU (slope = 1) and residual connections solved the problem for modern deep learning.
  • A flatlining loss early in training often indicates vanishing gradients, not convergence.

What to Read Next