Residual and Dense Networks

Why Does This Exist?

Push a plain network past 20 layers and something odd happens: training accuracy gets worse, not just test accuracy. A 56-layer plain network loses to its own 20-layer sibling on the training set itself, so this isn't overfitting. The extra layers should at least be able to copy their inputs through unchanged, but fitting an identity mapping out of stacked nonlinear layers turns out to be genuinely hard for gradient descent.

Residual Networks (He et al., 2015) fixed this with a shortcut: add the block's input back to its output, so each block only has to learn the correction. With that change, 152-layer networks trained cleanly, and the identity path has sat inside nearly every architecture since. ResNeXt (2017) and DenseNet (2017) pushed the same idea further through parallel paths and dense concatenation.

This page covers all three wirings and the gradient math that makes them work. The 1x1 convolution page covers the bottleneck trick they rely on.

Think of It Like This

How It Actually Works

The residual block and the gradient highway

A residual block computes $y = \mathcal{F}(x) + x$, where $x$ is the block input, $\mathcal{F}$ is the learned correction (usually two or three convolutions), and $y$ is the output. When channels or resolution change, a $1 \times 1$ projection $W_s$ aligns the shortcut: $y = \mathcal{F}(x) + W_s x$.

The payoff shows in backpropagation. Let $\mathcal{E}$ be the scalar loss, $x_l$ an early layer, and $x_L$ a later one. The chain rule gives $\partial \mathcal{E} / \partial x_l = (\partial \mathcal{E} / \partial x_L) \cdot (1 + \partial \sum \mathcal{F} / \partial x_l)$. That lone $1$ is an identity path: even if the correction gradients shrink to nothing, the loss gradient flows straight back. Depth stops punishing training.

ResNeXt: cardinality instead of width

ResNeXt replaces one wide correction with $C$ narrow parallel paths: $y = x + \sum_{i=1}^{C} \mathcal{T}_i(x)$, where each $\mathcal{T}_i$ is a small bottleneck ($1 \times 1$, then $3 \times 3$, then $1 \times 1$) and $C$ is the cardinality, typically $32$. Widening cardinality buys more accuracy per parameter than widening channels, and the whole thing runs as one grouped convolution.

DenseNet: concatenate instead of add

DenseNet swaps addition for channel concatenation: layer $l$ receives every earlier output, $x_l = H_l([x_0, x_1, \ldots, x_{l-1}])$, where $H_l$ is that layer's convolution and $[\cdot]$ means stacking channels. If each layer adds $k$ feature maps (the growth rate, often $k = 32$), layer $l$ sees $k_0 + k(l-1)$ channels, with $k_0$ the input width. Transition layers ($1 \times 1$ convolution plus pooling) compress between dense blocks so memory stays livable.

Worked example: a bottleneck block's parameters

Trace a ResNet bottleneck on input $X$ with $256$ channels at $56 \times 56$, stride $1$. The correction path is Conv $256 \to 64$ ($1 \times 1$), Conv $64 \to 64$ ($3 \times 3$), Conv $64 \to 256$ ($1 \times 1$), with shapes matching so the shortcut is plain $x$:

• Reduce: $256 \times 64 \times 1 \times 1 = 16{,}384$ parameters, output $(64, 56, 56)$
• Spatial: $64 \times 64 \times 3 \times 3 = 36{,}864$ parameters, output $(64, 56, 56)$
• Expand: $64 \times 256 \times 1 \times 1 = 16{,}384$ parameters, output $(256, 56, 56)$
• Total: $16{,}384 + 36{,}864 + 16{,}384 = 69{,}632$, and the block returns $\mathcal{F}(X) + X$

Two plain $3 \times 3$ convolutions at $256$ channels would cost $2 \times (256 \times 256 \times 9) = 1{,}179{,}648$, over $16$ times more. The bottleneck compresses before the expensive spatial step, and the shortcut carries the rest for free.

Code

1# A residual block in pure Python: the correction plus the shortcut.2x = 2.03correction = 0.5  # what the layers learned; 0.0 would pass x through untouched4y = correction + x5print(y)6# -> 2.57
8# Bottleneck vs plain parameter counts from the worked example.9plain = 2 * (256 * 256 * 9)10bottleneck = 256 * 64 + 64 * 64 * 9 + 64 * 25611print(plain, bottleneck, round(plain / bottleneck, 1))12# -> 1179648 69632 16.9

Watch Out For

The Quick Version

• A residual block outputs $\mathcal{F}(x) + x$, so layers learn corrections and identity is the default.
• The chain rule gains a $+1$ identity term, and gradients reach early layers no matter the depth.
• ResNeXt splits the correction into $32$ parallel paths, buying capacity without a FLOP explosion.
• DenseNet concatenates all past outputs, maximizing reuse at the cost of activation memory.
• A $256$-channel bottleneck costs $69{,}632$ parameters versus $1{,}179{,}648$ for plain $3 \times 3$ layers.

What to Read Next

• Image Classification
• 1x1 Convolution
• ConvNeXt Modernized Convolutions