Fully Convolutional Networks

Fully convolutional networks turned a classifier into a dense pixel labeller by swapping fully connected layers for 1x1 convolutions plus learned upsampling.

A classifier backbone shrinks the image to a coarse score grid, 1x1 convolutions score it, and learned upsampling with skips restores a full mask

Why Does This Exist?

Image classifiers output one label per image, but semantic segmentation needs one label per pixel. The obvious fix of running a classifier on every sliding window repeats work thousands of times and still outputs blocky maps.

FCN (Long, Shelhamer, Darrell, 2015) showed the classifier already computes a spatial grid before its final layers. Convert those layers to convolutions, add learned upsampling, and the whole image is labelled in one forward pass. This page covers the conversion trick and the FCN-32s/16s/8s ladder; the encoder-decoder successors sharpen the idea further, and DeepLab adds atrous context on top.

Think of It Like This

A rubber stamp grid instead of one stamp at a time

Imagine grading a wall of exam papers pinned in a grid. The old way picks up one paper, stamps it, puts it back, and repeats 10,000 times. FCN is a stamp as wide as the wall: one press grades every paper at once, then a second expanding press sharpens the blurry edges back to full size.

Where it stops: the giant stamp still sees coarsely, so fine edges need the skip fusions described below.

How It Actually Works

A VGG-style backbone downsamples a $H \times W$ image by $32$ through five pooling stages, ending with a $H/32 \times W/32 \times 4096$ tensor. A fully connected layer reads that whole tensor at once, which is why it destroys spatial layout.

The convolutionalization trick

Replace each fully connected layer with a $1 \times 1$ convolution carrying the same weights. A layer that mapped $4096$ inputs to $1000$ scores becomes $1000$ filters of size $1 \times 1$ . The output changes from a $1000$ -vector to a $H/32 \times W/32 \times K$ score grid with $K$ classes. Nothing is relearned; the weights are reshaped, so ImageNet pretraining carries over.

Learned upsampling and the 32s/16s/8s ladder

A transposed convolution with stride $32$ stretches the coarse grid back to $H \times W$ (FCN-32s), but edges come out blobby. FCN-16s upsamples $2\times$ , adds the pool4 prediction map, then upsamples $16\times$ . FCN-8s repeats from pool3. Each skip re-injects a sharper map before the final stretch.

Worked example

Input $320 \times 320$ . Backbone output is $10 \times 10 \times 21$ for $21$ PASCAL classes, since $320 / 32 = 10$ . A pixel at coarse position $(4, 6)$ with winning class $15$ (person) and score $3.1$ against runner-up $1.2$ covers a $32 \times 32$ patch in the input: rows $4 \times 32 = 128$ to $159$ , columns $6 \times 32 = 192$ to $223$ . Bilinear initialization of the transposed filter spreads that $3.1$ smoothly, and the pool3 skip (at $40 \times 40$ ) corrects the patch border by up to $4$ pixels where the person meets the background.

Code

The point of this snippet is the coarse-grid geometry: standard-library-only arithmetic replays the worked example grid and patch bounds with no tensor package involved.

1H, W, stride, classes = 320, 320, 32, 212gh, gw = H // stride, W // stride3print((gh, gw, classes))4# -> (10, 10, 21)5
6r, c = 4, 67print((r * stride, (r + 1) * stride - 1, c * stride, (c + 1) * stride - 1))8# -> (128, 159, 192, 223)

The grid prints $10 \times 10 \times 21$ and the cell maps to rows 128 to 159 and columns 192 to 223, exactly the patch the prose assigns to coarse position $(4, 6)$ .

Watch Out For

Checkerboard artefacts from transposed convolutions

Stride- $32$ transposed convolutions with uneven kernel overlap paint a faint grid over the mask. The symptom is regular checkerboard flicker on flat regions like sky. Fix it by initializing with bilinear weights, keeping kernel size divisible by stride, or upsampling with bilinear interpolation followed by a $1 \times 1$ convolution.

Treating FCN-32s output as boundary truth

Coarse-only FCN-32s sits off by many pixels at edges, so thin structures must never be measured with it. Move to FCN-8s skips or a later encoder-decoder design before trusting boundaries.

The Quick Version

FCN replaces fully connected layers with $1 \times 1$ convolutions, keeping ImageNet weights and producing a spatial score grid.
Transposed convolutions learn the upsampling from $1/32$ resolution back to full size.
FCN-16s and FCN-8s fuse pool4 and pool3 maps to sharpen boundaries.
The worked grid proves it: $320 / 32 = 10$ , and cell $(4, 6)$ owns rows 128 to 159 and columns 192 to 223.
One forward pass labels the whole image instead of thousands of sliding windows.