Core ML

Hierarchical Vision Transformers

Hierarchical Vision Transformers attend within small local windows and merge patches across stages, rebuilding a CNN-like pyramid of image resolutions.

Attention runs only inside small local windows of patches instead of across the whole grid, and patches merge into fewer, coarser tokens as the stages progress

Why Does This Exist?

A plain Vision Transformer computes full self-attention across every patch token against every other patch token, at one fixed resolution, all the way through the network. That has two costs that only show up once you try to use it for more than classification. First, the cost: attention over $n$ patches costs roughly $n^2$ , so doubling the image's side length (quadrupling the patch count) roughly sixteen-times the attention cost — fine for a 14×14 grid at classification resolution, punishing for the much finer grids that detection and segmentation need. Second, the shape of the output: a plain ViT produces one flat set of token representations, all at the same resolution, the whole way through. Detection and segmentation, though, are built around a CNN convention of a resolution pyramid — coarse, medium, and fine feature maps at different stages — because objects of different sizes are easiest to localize at different resolutions.

The wall is exactly that mismatch: tasks that need a multi-resolution pyramid meet an architecture that only produces one flat resolution, at a cost that gets prohibitive at the fine resolutions those tasks actually need. Hierarchical Vision Transformers — the Swin Transformer is the best-known example — fix both problems with two related moves: restrict attention to small local windows instead of the whole grid, and periodically merge patches into coarser tokens across stages.

Think of It Like This

Reading a newspaper page by section, then by page, then by issue

A plain Vision Transformer reads a newspaper the way someone would if they insisted on cross-referencing every single word against every other word on the page at once — technically thorough, but the cost explodes as the page gets denser, and it never steps back to notice the page is part of a larger section, which is part of a whole issue.

A hierarchical Vision Transformer reads the way an actual person does: closely within one paragraph first — a local window — then steps back and treats whole paragraphs as the new unit, comparing paragraph to paragraph rather than word to word, then steps back again to compare sections. Each step up trades fine detail for a wider view, and the reading at any one level only ever compares nearby units, never the whole page's words against each other directly. That's local windows plus periodic merging, exactly.

How It Actually Works

Restrict attention to local, non-overlapping windows

Instead of every patch attending to every other patch in the grid, patches are grouped into small local windows — 7×7 patches is Swin's default — and self-attention runs only within each window, never across window boundaries. A patch in one corner of the image simply never attends to a patch in the opposite corner within a single layer. Cost per window scales with the square of the window size, which is fixed and small, and the number of windows scales linearly with the total patch count — so total cost becomes linear in the number of patches, rather than quadratic, which is exactly the trade a plain ViT doesn't make.

Shift the windows between consecutive layers

A pure windowed design has an obvious gap: information genuinely never crosses a window boundary, so a pattern spanning two adjacent windows is invisible to any single attention layer. Swin's fix is to shift the window grid by half a window's width between consecutive layers, so a patch that sat at the edge of one window sits in the interior of a different window one layer later. Stack a few of these shifted layers and information has flowed across every original boundary, without ever paying for a single full-grid attention pass.

Merge patches into coarser tokens across stages

Periodically — after a fixed number of layers — neighboring groups of patches (commonly 2×2) are concatenated and linearly projected down into a single, coarser token, exactly mirroring what pooling does in a CNN: halve the spatial resolution, and typically widen the channel count to compensate. Repeating this across several stages produces exactly the multi-resolution pyramid a plain ViT never builds — an early stage with many fine tokens, a late stage with few coarse ones, and intermediate stages in between, each one a valid feature map a detection or segmentation head can attach to.

The tradeoff this buys

Restricting attention to local windows means a single layer's receptive field is smaller than a plain ViT's — no layer sees the whole image at once anymore. What recovers global context is depth: patch merging shrinks the grid stage by stage, so a fixed-size window covers a proportionally larger fraction of the (now smaller) image at each later stage, and the shifted-window trick keeps information flowing across every boundary in between. The architecture trades "every layer sees everything" for "every layer sees a lot less, cheaply, and depth reassembles the full picture" — the same bet a CNN's stacked local kernels make, applied to attention instead of convolution.

Show Me the Code

Comparing the attention cost of a full grid against a windowed version, at a resolution fine enough that the difference actually matters.

1import numpy as np2
3
4def full_attention_cost(num_patches: int) -> int:5    return num_patches ** 26
7
8def windowed_attention_cost(grid_size: int, window_size: int) -> int:9    num_windows = (grid_size // window_size) ** 210    tokens_per_window = window_size ** 211    return num_windows * tokens_per_window ** 212
13
14grid_size = 56                        # a fine early-stage grid, e.g. 224px images at patch size 415num_patches = grid_size ** 216full_cost = full_attention_cost(num_patches)17windowed_cost = windowed_attention_cost(grid_size, window_size=7)18print(full_cost)                      # -> 9834496 -- full grid, every patch against every patch19print(windowed_cost)                  # -> 153664 -- same grid, attention confined to 7x7 windows20print(full_cost // windowed_cost)     # -> 64 -- the windowed version costs 64x less at this resolution

That 64x gap only grows at finer resolutions, which is exactly why a plain ViT's full attention becomes impractical at the grid sizes detection and segmentation actually need.

Watch Out For

Using plain (non-shifted) windows and losing cross-window information

Windowing without the shift step confines every layer's receptive field to a fixed, unchanging set of patches — nothing a window doesn't contain ever gets seen by that window's tokens, in any layer. The failure is subtle: the model still trains and still produces reasonable-looking output, just with a real ceiling on how large a pattern it can represent, because no amount of depth alone crosses a boundary that stays in the same place every layer. The shift is not optional polish; it's what makes stacking windowed layers equivalent, over depth, to a much larger receptive field.

Expecting merged tokens to behave like the original patches

After a merge step, a token no longer corresponds to one original image patch — it's a learned combination of several. Code or intuition that assumes a fixed patch-to-pixel mapping at every stage (for indexing back into the original image, for instance) breaks silently once merging has happened, because the token count and its relationship to pixel coordinates changes at every stage boundary.

The Quick Version

Plain Vision Transformers pay quadratic attention cost and produce one flat resolution, which is wrong for detection and segmentation's multi-scale needs.
Hierarchical designs restrict attention to small local windows, making cost linear in patch count instead of quadratic.
Windows shift between consecutive layers so information still crosses window boundaries over depth.
Patches merge into coarser tokens across stages, rebuilding a CNN-like resolution pyramid.
The tradeoff: no single layer sees the whole image, but depth plus shifting reassembles global context cheaply.