Core ML

Residual Connections

Let a layer learn only the correction and pass the original input straight through untouched — the identity path that made 100-plus-layer networks trainable.

A residual block adds its input back onto whatever the layers inside it computed, so if those layers learn nothing at all, the block still passes the input through untouched

Why Does This Exist?

Say you're building an OCR system to read scanned receipts, and the current 19-layer network makes too many character-level mistakes on faded thermal paper. The obvious next move is more depth — deeper networks are, in theory, strictly more capable than shallow ones, because a deeper network can always fall back to computing exactly what a shallower one computes and simply do nothing with the extra layers.

Here's the wall, and it's the one researchers actually hit building VGG-19 in 2014: stack enough plain convolutional layers and, past a certain depth, training accuracy gets worse, not better. Not validation accuracy — that would just be overfitting, a familiar and separately-solved problem. Training accuracy. The network is doing a worse job of fitting the data it has already seen, with more layers meant to give it more room to fit that same data better.

That's the degradation problem, and the theoretical argument above says it shouldn't be possible. If the extra layers could simply learn to pass their input through unchanged — an identity mapping — the deeper network would match the shallower one's accuracy at worst, and improve on it wherever the extra capacity actually helped. So the extra layers, in practice, are failing to learn something that should be trivial: do nothing.

Think of It Like This

Grading a correction instead of a rewrite

Ask an editor to take a clean paragraph and reproduce it exactly, word for word, using only a red pen that adds or crosses out text — no copying, no starting fresh. That's a strange, fiddly task. Getting every single word to survive untouched through a process built for making changes takes real, deliberate effort, and it's easy to introduce a stray edit by accident.

Now instead, hand the editor the same paragraph and just say: "leave it as is, unless you see an improvement — then mark only the change." The default action, doing nothing, is now the easy path. Making zero edits is trivial. Making one small edit is nearly as easy. The editor's whole job just became "spot corrections," not "faithfully reproduce, then maybe improve."

A residual block is that second instruction, wired into the architecture instead of left as a hope.

How It Actually Works

Why identity is a hard thing to learn by accident

A plain stacked layer computes some function $F(x)$ — a weight matrix, a nonlinearity, maybe another matrix — and there is no reason a randomly initialized $F$ should land anywhere near the identity function. Reproducing $x$ exactly through a matrix multiply and a ReLU is a specific, narrow target inside a huge space of things $F$ could compute, and gradient descent has to discover that target from scratch, through every layer, with no help from the architecture itself. In a deep stack, several such layers each have to independently converge on "change nothing" for the whole stack to behave like its shallower counterpart — and empirically, that doesn't reliably happen.

The fix: make identity the free case

A residual block restructures the computation. Instead of asking a block of layers to output $F(x)$ directly, it outputs $F(x) + x$ — whatever the layers compute, plus the original input, added back in unchanged. The diagram above shows the shape: the input splits into two paths, one running straight through untouched (the skip connection), the other passing through the block's layers, and the two recombine by simple addition at the far end.

The payoff is immediate. If $F(x) = 0$ — every weight in the block is zero, or drifts toward zero — the block's output is exactly $x$ . Identity is no longer something the layers have to painstakingly reconstruct; it's the default they get for free when they've learned nothing at all. What the layers now have to learn is only the residual — the correction on top of identity — which is a far smaller, more tractable target than reconstructing the whole mapping from scratch. That reframing is where the name comes from: the block learns the residual, the input carries the rest.

A secondary benefit, worth naming separately

Residual connections also change backpropagation's arithmetic in a useful way. Because the forward computation is $F(x) + x$ , the gradient flowing backward through a residual block picks up an added term straight from that +x, alongside whatever gradient flows back through $F$ . Even when $F$ 's own gradient shrinks toward zero deep in a network — the vanishing-gradient problem, where each layer's local derivative multiplies into the next and a long chain of small factors compounds toward nothing — that added term keeps a usable signal reaching earlier layers, because it never has to pass through $F$ at all.

It's worth being precise about what this does and doesn't explain: the degradation problem above is about the optimizer's difficulty finding an identity mapping through nonlinear layers, and the original ResNet paper found it in networks whose gradients were not, by their own measurements, vanishing. Vanishing gradients is a real, separate difficulty that gets worse with depth, and normalization layers are the more direct fix for it. Residual connections happen to help with both — treat them as two related but distinct benefits of the same one-line change, not one problem with two names.

Show Me the Code

A minimal residual block, and the exact claim from this page — zero weights inside the block reproduce the input exactly — checked directly rather than asserted.

1import numpy as np2
3
4def residual_block(x: np.ndarray, w1: np.ndarray, w2: np.ndarray) -> np.ndarray:5    h = np.maximum(w1 @ x, 0.0)  # first layer + ReLU6    f_x = w2 @ h  # second layer, no activation before the addition7    return f_x + x  # the skip connection: add the untouched input back8
9
10x = np.array([0.6, -1.1, 2.0, 0.3])11zeros = np.zeros((4, 4))12
13output = residual_block(x, zeros, zeros)14print(np.allclose(output, x))  # -> True — F(x) = 0, so the block is exactly identity15
16rng = np.random.default_rng(0)17small_w1, small_w2 = rng.normal(scale=0.01, size=(4, 4)), rng.normal(scale=0.01, size=(4, 4))18nearly_identity = residual_block(x, small_w1, small_w2)19print(np.abs(nearly_identity - x).max() < 0.05)  # -> True — small weights, small correction

With zero weights the block is provably identity, no approximation. With small random weights it stays close to identity, because $F(x)$ is small when the weights are small — the block naturally starts near "do nothing" and moves away from there only as far as training pushes it.

Watch Out For

Mismatched shapes at the addition

The addition $F(x) + x$ only works when $F(x)$ and $x$ have identical shape. That's automatic when a block preserves both spatial dimensions and channel count, but plenty of real architectures deliberately change the channel count between stages — exactly the shape change in the CNN architecture lineage page's stage transitions.

The standard fix is a projection connection: a small 1×1 convolution applied to the skip path alone, whose only job is reshaping $x$ to match $F(x)$ 's new shape before the addition. It's a real, trained set of weights, not a free identity anymore — but it's still far cheaper than the full block, and it's needed only where the shape actually changes.

Treating the skip connection as a substitute for normalization

Residual connections and batch normalization solve adjacent but different problems — one makes identity easy to learn, the other keeps each layer's activations in a well-behaved range — and modern deep networks use both together for a reason. Removing normalization from a residual network and expecting the skip path alone to keep training stable is a common and costly assumption; very deep residual networks trained without any normalization are far harder to get converging at all.

The Quick Version

Stacking plain layers past a certain depth causes training accuracy itself to get worse — the degradation problem, distinct from overfitting.
In theory, extra layers could just learn identity and match a shallower network at worst. In practice, plain nonlinear layers struggle to converge on identity by accident.
A residual block computes $F(x) + x$ instead of $F(x)$ alone, so identity is the free default (at $F(x)=0$ ) and the layers only have to learn a correction.
As a secondary benefit, the +x term gives backpropagation a path that bypasses $F$ entirely, helping with vanishing gradients — a related but separate difficulty from degradation.
A shape change between the input and $F(x)$ needs a projection connection on the skip path — a small trained reshape, not a free identity anymore.