SegFormer Segments Without Positions

SegFormer pairs a hierarchical transformer encoder with a tiny MLP decoder, so dense pixel masks come from four feature scales with no positional encodings at all.

SegFormer merges overlapped patches into four shrinking feature scales, then a two-layer MLP head fuses them into the mask with no positional encodings.

Why Does This Exist?

Plain vision transformers carry positional encodings tuned to one training size: test on a taller street scene and the position codes no longer line up, so dense prediction degrades exactly where resolution varies most. They also pair the encoder with a heavy decoder, which makes high-resolution masks slow. Semantic segmentation needs the opposite, one model that handles any input size with a cheap head.

SegFormer (2021) drops both crutches. Overlapped patch merging builds a four-scale hierarchy like a CNN pyramid, a Mix feedforward layer bakes locality into every block instead of position codes, and the decoder is two MLP layers over the fused scales. This page covers the encoder, the position-free trick, the MLP head, and the attention arithmetic; the classic dilated-convolution answer it displaced is the DeepLab family, and the query-based successor is Mask2Former.

Think of It Like This

Sketching districts, then inking with one thin pen

A cartographer draws a city at four zoom levels with overlapping pencil strokes, so no street falls on a gap between sheets. Then she inks the final borders with a single thin pen instead of unpacking a full paint set.

Overlapped merging is the overlapping pencil work, Mix-FFN is the local knowledge of which lines join up, and the MLP head is the thin pen. The analogy stops at drafts: a thin pen cannot fix wrong sketches, and likewise the light decoder only works because the four-scale encoder already holds clean boundaries.

How It Actually Works

Overlapped hierarchical encoder

The image is embedded as $4 \times 4$ patches with overlap between neighbors, so boundaries never sit exactly on a patch seam, then patch merging repeats at each of four stages to produce feature maps at $1/4$ , $1/8$ , $1/16$ , and $1/32$ of the input resolution. Self-attention at each stage uses key reduction: with token count $N$ and reduction ratio $R$ , keys shrink to $N/R$ and attention pairs fall from $N^2$ to $N^2/R$ . Coarse stages carry object context while the fine $1/4$ stage keeps edge detail.

Mix-FFN instead of position codes

Each transformer block replaces the plain feedforward network with Mix-FFN, which inserts a $3 \times 3$ depthwise convolution between the two linear layers. The convolution leaks each token's neighborhood into its own representation, giving the model a sense of locality from the data rather than from fixed position vectors. Because no block reads an absolute position, train and test resolutions can differ freely.

The two-layer MLP decoder

All four feature maps are projected to a shared width, upsampled to the $1/4$ scale, concatenated, and passed through two MLP layers that output the per-class mask, upsampled once more to full resolution. There are no dilated convolutions, no pyramid pooling, no refinement stages: the head assumes the encoder already separated the classes and just fuses scales.

Worked example

Take a $512 \times 512$ input. Stage one holds $128 \times 128 = 16{,}384$ tokens, so full self-attention would score $16{,}384^2 = 268{,}435{,}456$ pairs. With a key-reduction ratio of $R = 64$ , keys shrink to $16{,}384 / 64 = 256$ and pairs fall to $16{,}384 \times 256 = 4{,}194{,}304$ , which is $268{,}435{,}456 / 4{,}194{,}304 = 64$ times cheaper. That saving repeats at every stage with its own ratio, which is why the encoder stays practical on dense street-scene resolutions.

Code

1tokens = 128 * 128  # stage-one tokens for a 512 x 512 input2r = 64  # key-reduction ratio3
4full = tokens * tokens5reduced = tokens * (tokens // r)6print(full, reduced, full // reduced)7# -> 268435456 4194304 64

Watch Out For

Testing on sizes that break the merging grid

Overlapped merging and repeated halving assume dimensions divisible deep into the pyramid. The symptom is a strip of garbage predictions along the right or bottom edge on odd-sized inputs. Pad inputs to multiples of 32 before inference and crop the mask back afterward.

Training the transformer encoder from scratch on small data

The position-free design still needs data: a hierarchical transformer trained from random weights on a few hundred masks overfits while a CNN baseline generalizes. The symptom is training loss near zero with ragged validation masks. Start from ImageNet-pretrained Mix encoder weights and fine-tune, and only train from scratch with thousands of labeled scenes.

The Quick Version

SegFormer merges overlapped $4 \times 4$ patches into four scales from $1/4$ down to $1/32$ resolution.
Mix-FFN puts a $3 \times 3$ depthwise convolution inside each block, replacing positional encodings.
The decoder is two MLP layers over the four upsampled and fused scales.
Worked numbers: $16{,}384$ tokens with $R = 64$ cut attention pairs from 268M to 4.19M, 64 times cheaper.
Pad test inputs to multiples of 32, and fine-tune from pretrained encoders on small datasets.