Hybrids Stage Convs Before Attention

Hybrids run convolutions early for local bias and attention late for global range, beating either alone on data efficiency and compute cost.

Convolutions process 3136 tokens on 56 maps then attention handles 196 tokens on 14 maps, cutting attention pairs 256 times while keeping bias plus range

Why Does This Exist?

Pure CNNs bake in locality and translation equivariance but struggle to relate distant regions without deep stacks, while pure transformers model every pair of patches and therefore need large datasets to learn what convolutions get for free. Hybrids stage the two in order: convolutional layers shrink the maps where attention would be ruinous, then attention runs where the maps are small and global reasoning pays off. Families such as CoAtNet, CvT, and MaxViT differ in the block recipe, not the staging logic, and distilled transformers like DeiT attack the same data hunger from the training side.

This page covers the staging rule and its cost math. Convolution mechanics live in convolutional layers, classification heads in image classification, and pure transformer training in DeiT.

Think of It Like This

Local guides plus satellite view

A survey team walks the streets house by house for detail, then climbs to a hilltop to map how districts connect. Nobody starts from orbit when the street names are unknown, and nobody walks every road when one overlook shows the whole layout. One trip uses both in order.

Convolutions are the street walkers, attention is the hilltop view. Where it stops: swapping the order wastes money, just as running global attention on full-resolution maps burns compute before any detail is organized.

How It Actually Works

Early stages use MBConv or convolutional patch embeddings on large maps such as $56 \times 56$ and $28 \times 28$ , where cost grows linearly in tokens and the inductive bias matters most. Late stages switch to attention on small maps such as $14 \times 14$ and $7 \times 7$ , adding relative position bias or window partitions so the global mixing stays cheap. A pyramid or fused backbone then feeds the detection or classification head. The rule is order, not brand: convs first to compress, attention after to connect.

The savings come from token counts. Attention pairs grow quadratically, so shrinking the map edge by $4$ cuts tokens by $16$ and pairs by $256$ . That is why every hybrid family downsamples with convolutions before the first transformer block.

Worked example

Take a $56 \times 56$ map with $64$ channels and a late $14 \times 14$ map. Early tokens number $56 \times 56 = 3136$ ; a $3 \times 3$ convolution costs about $3136 \times 3 \times 3 \times 64 = 3136 \times 576 = 1806336$ , roughly $1.8$ M multiplies, linear in tokens. Late tokens number $14 \times 14 = 196$ ; full attention over them costs $196^2 = 38416$ pairs, about $38$ K. Attention over the early map would cost $3136^2 = 9834496$ pairs, about $9.8$ M. Token ratio $3136 / 196 = 16$ , pair ratio $9834496 / 38416 = 256$ : staging cuts attention pairs $256$ times while keeping early local bias and late global range.

Code

The fixture arithmetic with only the standard library:

1early = 56 * 56  # 3136 tokens2late = 14 * 14  # 196 tokens3conv_mults = early * 3 * 3 * 644early_pairs = early ** 25late_pairs = late ** 26print(f"tokens {early} vs {late}, ratio {early // late}")7# -> tokens 3136 vs 196, ratio 168print(f"conv {conv_mults}, pairs {early_pairs} vs {late_pairs}, ratio {early_pairs // late_pairs}")9# -> conv 1806336, pairs 9834496 vs 38416, ratio 256

Early convolutions stay linear at about $1.8$ M multiplies while late attention costs $38$ K pairs instead of $9.8$ M, a $256$ -times saving.

Watch Out For

Running attention before downsampling

Symptom: training crawls and memory explodes while accuracy barely moves, because full attention on $56 \times 56$ maps pays $9.8$ M pairs per layer for relationships convolutions would find cheaply. Keep convolutions first and verify the first attention block sees $14 \times 14$ maps or smaller before scaling up.

Treating every hybrid paper as a new principle

Symptom: the codebase collects one-off CoAtNet, CvT, and MaxViT branches that drift apart, when all three implement convs early and attention late with different blocks. Fix: keep one staging utility and link the block details to their component pages instead of copying each paper here.

The Quick Version

Hybrids run convolutions on large maps for local bias and attention on small maps for global range.
Early $3136$ tokens cost about $1.8$ M linear convolution multiplies at $3 \times 3 \times 64$ .
Late $196$ tokens cost $38$ K attention pairs instead of $9.8$ M, a $256$ -times saving.
Token ratio is $16$ and pair ratio is $256$ because attention is quadratic in tokens.
Keep the order convs first and attention after shrinking, or the savings vanish.