DeepLab v1 to v3 Plus

DeepLab keeps resolution high with spaced-out atrous filters, sees many scales at once with ASPP, and sharpens edges with a small decoder in v3 Plus.

Spaced-out atrous filters hold the map at stride 16 while parallel ASPP branches at rates 6, 12 and 18 vote at three fields of view before the v3 Plus decoder restores edges

Why Does This Exist?

A classification backbone downsamples by $32$ , which is fine when the answer is one label and fatal when the answer is a label per pixel. Semantic segmentation needs boundaries where they actually are, not where a coarse grid guesses. The second problem is scale: one fixed filter cannot cover a distant pedestrian and a nearby bus in the same pass.

The DeepLab line (Chen and colleagues, 2015 to 2018) attacked both without exploding compute. Hold the output stride at $8$ or $16$ with spaced-out filters instead of striding, then probe every location at several fields of view at once. This page traces v1 through v3 Plus as one arc. The filter operation itself gets full treatment in dilated convolution, and the plain upsampling alternative lives in fully convolutional networks.

Think of It Like This

A surveyor with a zoom rake

You must map pebbles and boulders in one pass. A normal rake touches adjacent soil only. An atrous rake skips teeth: same handle weight, wider reach. ASPP hands you four rakes at once with gaps of $6$ , $12$ and $18$ inches plus one glance over the whole field, then merges the readings. v3 Plus adds a fine brush for the pebble outlines.

Where it stops: wider gaps sample sparsely, so very thin structures can fall between the teeth.

How It Actually Works

Atrous convolution

A $3 \times 3$ filter with dilation rate $r$ inserts $r - 1$ gaps between taps, reaching across $2r + 1$ pixels with the same $9$ weights. Rate $2$ covers $5 \times 5$ ; rate $6$ covers $13 \times 13$ . DeepLab replaces striding in the last backbone blocks with dilation, so the output stride stays at $16$ (v1 and v2) or $8$ (v3) instead of collapsing to $32$ . No extra weights, no extra downsampling, just a wider view per parameter.

ASPP and the version ladder

One scale is never enough, so v2 introduced Atrous Spatial Pyramid Pooling: parallel $3 \times 3$ branches at rates $6$ , $12$ , $18$ and $24$ plus a $1 \times 1$ branch, concatenated and fused with a final $1 \times 1$ convolution. Each location votes at several fields of view at once.

The versions stack like this. v1 pairs atrous convolutions with a dense CRF that snaps labels to image edges. v2 adds ASPP. v3 drops the CRF, adds image-level pooling to the pyramid, and modernizes training. v3 Plus adds a light decoder: upsample the ASPP output $4\times$ , fuse it with low-level backbone features, then stretch $4\times$ more to full resolution. The decoder is what recovered the sharp boundaries the CRF used to provide.

Worked example

Take a $512 \times 512$ image through a backbone at output stride $16$ . The map is $32 \times 32$ . An ASPP branch at rate $12$ spans $2 \times 12 + 1 = 25$ map pixels, which is $25 \times 16 = 400$ input pixels across: enough to cover a bus. The rate- $6$ branch spans $13 \times 16 = 208$ pixels for cars. The $1 \times 1$ branch reads local texture. Concatenated, one location combines the bus vote, the car vote and the road texture, and the v3 Plus decoder re-snaps the winning vote onto the low-level edge map.

Code

The point here is the reach check: rate $r$ on a $3 \times 3$ kernel always spans $2r + 1$ map pixels, scaled by the output stride into input pixels.

1def reach(rate: int) -> int:2    return 2 * rate + 1  # map pixels spanned by a 3x3 kernel3
4
5rates = [1, 6, 12, 18]6print([(r, reach(r), reach(r) * 16) for r in rates])7# -> [(1, 3, 48), (6, 13, 208), (12, 25, 400), (18, 37, 592)]

Read the last triple as: rate $18$ sees $37$ map pixels, which is $592$ input pixels at stride $16$ , wider than the $512$ -pixel image itself. That is your cue that the rate is too large for this stride.

Watch Out For

Gridding artefacts at large rates

Rates past about $18$ on a stride- $16$ map sample isolated pixels with dead gaps between taps. Symptom: striped misses on thin rails and wires while large regions look fine. Fix it by capping rates near the map size, keeping the image-pooling branch, or switching the output stride to $8$ .

Shipping v1 CRF settings with v3

The dense CRF helped v1 but v3 Plus is accurate without it, and stale CRF sigmas blur the sharp edges the decoder just recovered. Symptom: boundaries get worse after adding post-processing that was supposed to help. Fix it by retuning the CRF from scratch or dropping it when moving to v3 or v3 Plus.

The Quick Version

Atrous convolution widens the receptive field with no extra weights and no extra downsampling.
ASPP probes each location at rates $6$ , $12$ and $18$ plus image pooling, so one pass covers many object scales.
v1 used a dense CRF, v2 added ASPP, v3 modernized training, and v3 Plus added a light edge decoder.
Output stride $8$ or $16$ keeps maps dense enough for real boundaries.
A rate- $12$ branch at stride $16$ sees $400$ input pixels across, enough for a bus; the rate- $6$ branch sees $208$ for cars.
Over-large dilation rates cause gridding on thin structures.