Deep Residual Learning

The Problem

By 2015, the deep learning community knew that deeper networks generally performed better. The intuition was simple: more layers mean the network can learn more complex and abstract features. However, researchers ran into a wall. When they tried to train networks deeper than 20-30 layers, the training error actually got worse.

This wasn't caused by overfitting (where training error is low but test error is high). This was a degradation problem: deep networks were simply harder to optimize. As gradients were backpropagated from the output layer back to the early layers, they would repeatedly multiply by small weights, causing them to shrink to zero. This is the notorious vanishing gradient problem. The early layers stopped learning entirely.

The Idea

Kaiming He and his team at Microsoft Research asked a clever question: if we take a shallow network and just add extra layers that do absolutely nothing (identity mappings), the deeper network should perform at least as well as the shallow one. So why can't a deep network learn to do this on its own?

They realized that it's mathematically difficult for a stack of non-linear layers (like convolutions and ReLUs) to learn an exact identity mapping (i.e., outputting exactly what was inputted).

Their solution was to change the architecture so that the network doesn't have to learn the identity mapping from scratch. They introduced skip connections (or shortcut connections).

How It Works

Instead of hoping the layers $F(x)$ will learn the desired underlying mapping $H(x)$, they let the layers fit a residual mapping: $F(x) = H(x) - x$.

To get the final output, they just add the original input $x$ back to the output of the layers: $H(x) = F(x) + x$.

In practice, this is implemented as a residual block. The input $x$ passes through a couple of convolutional layers. Meanwhile, the original $x$ bypasses those layers via a shortcut connection. The output of the convolutional layers and the shortcut connection are added together element-wise before the final activation function.

If the optimal mapping is closer to an identity mapping than to a zero mapping, it is much easier for the network to push the residual $F(x)$ to zero than it is to fit an identity mapping from scratch.

Why It Mattered

Residual blocks completely destroyed the depth barrier. Instead of being capped at ~30 layers, the authors successfully trained networks with 152 layers (ResNet-152) and even 1000 layers. ResNet-152 won the 2015 ImageNet competition by a landslide, reducing the top-5 error rate to 3.57% (surpassing human performance).

More importantly, the skip connections provided "superhighways" for gradients during backpropagation. Because the addition operation distributes the gradient equally, gradients could flow directly from the end of the network to the very beginning without vanishing.

What Came After

ResNet became the default architecture for almost all computer vision tasks. The concept of residual connections (or skip connections) was universally adopted. You will find them in almost every modern deep learning architecture today, including the Transformer and diffusion models (via the U-Net). Residual learning remains one of the most fundamental and enduring discoveries in deep learning.