Neural Networks
Skip / Residual Connections
How residual blocks provide a "gradient highway", completely solving the vanishing gradient problem in very deep neural networks.
Stage 1 of 4: The Shallow Network
2-layer network, skip connections off. Gradient at layer 0: 0.2500.
- Layer Activation
- Gradient Signal
- Vanished Gradient
In a shallow network, the error signal (gradient) easily reaches the first layer during the backward pass.
- depth 2, skip connections off
- layer 0 gradient 0.2500
- strong enough to learn from
Check your understanding
See why deep networks fail to train, and how adding skip connections lets gradients flow cleanly to early layers.
For a long time, researchers couldn't train deep neural networks. The problem was the Vanishing Gradient.
During training, the error signal (gradient) travels backward from the output to the input layer. But at every layer, it gets multiplied by weights and derivatives. If these numbers are smaller than 1, the gradient shrinks. In a 20-layer network, the gradient becomes microscopically small by the time it reaches the first layer. The early layers simply stop learning.
In 2015, ResNet solved this with Skip Connections (or Residual Connections). A skip connection literally bypasses a layer, adding the input directly to the output. During the backward pass, gradients flow freely down this shortcut, acting as a "gradient highway". This simple trick allows us to train networks with hundreds or thousands of layers!
Watch the "Layer 0 Gradient" readout as you push the network depth to 10 with skip connections off, then flip them on at the same depth — the number goes from vanishing to over 7 without changing anything else.
Reference
- Vanishing Gradient
- When gradients become too small to effectively update early layers.
- Skip Connection
- An architectural shortcut that adds an earlier layer's output to a later layer.
- ResNet
- Residual Network. The architecture that popularized skip connections.
Break it on purpose
Without skip connections, the gradient reaching the input layer of a 10-layer network drops below 0.01 — measured at 0.001. The early layers learn almost nothing, and the network fails to train. Turn skip connections on at the same depth and the same gradient jumps to over 7.