Core ML

Attention Sinks and Streaming

The first few tokens in any sequence attract a disproportionate share of attention mass regardless of their content — they act as a sink that stabilises the softmax distribution, and evicting them from a sliding window cache causes the model to collapse.

Without attention sinks, evicting early tokens causes the model to collapse; pinning the first few tokens as sinks lets the window slide safely because those high-attention-mass tokens stay accessible regardless of their content

Why Does This Exist?

Sliding window attention makes inference over very long sequences tractable by keeping only the W most recent tokens in the KV cache. The idea is that local context is usually sufficient. What nobody predicted before measuring it: the moment the oldest tokens in the cache get evicted, model quality doesn't degrade gradually — it collapses. Perplexity spikes sharply the first time token 0 leaves the window.

This wasn't obvious because token 0 — often <bos>, a BOS token, or the first word of the document — has no special semantic meaning. Why would losing it break everything?

Xiao et al. (2023) measured attention weights across layers in LLMs and found the answer: the first few tokens absorb a large fraction of the total attention mass in almost every layer, for almost every input. Not because they're semantically important, but because they're always accessible. If every key you've seen is a potential recipient, and you need the softmax weights to sum to 1, having a few "dump" positions that reliably absorb excess attention weight is numerically convenient — the model learns to use them this way during training. These are the attention sinks.

Evict the sinks and the softmax distribution loses its dump buckets. Attention mass has nowhere to go, the distribution destabilises, and the model's output degrades sharply.

Think of It Like This

A vote where write-in candidates absorb spoiled ballots

Imagine an election where every voter must mark exactly one candidate, but some voters genuinely don't care who wins. Those voters write in "none of the above" — an absorber that keeps the formal sum-to-one rule intact without distorting the real preference signal. Remove that option and those voters are forced to pick a real candidate, adding noise to the count.

Attention sinks are the "none of the above" option for the softmax. Layers route some of their attention to the first tokens as a pressure valve. Remove those tokens and the model loses its pressure valve.

How It Actually Works

What the attention weights look like

Across many LLMs and many input types, the first one to four tokens consistently receive attention weights an order of magnitude higher than their position in the sequence would predict. A token at position 0 in a 4,096-token input may absorb 20–30% of a head's total attention weight, even if that token is just <bos>. This pattern persists across layers and heads.

Why eviction causes collapse

The softmax normalises attention weights to sum to one. If the sink tokens are present, they absorb a chunk of that mass; the remaining weights distribute across semantically relevant tokens. Without the sinks, the model has to assign that mass elsewhere, to tokens that the training distribution never prepared it to weight that heavily, and the output becomes incoherent.

The fix: pin the sinks

StreamingLLM's fix is minimal and elegant: always keep the first K tokens (typically K=4) pinned in the cache — they are never evicted, regardless of how long the sequence gets. The rest of the cache follows a rolling FIFO policy over the most recent W-K tokens. The total cache size is still bounded at K + (W - K) = W tokens.

With the sinks pinned, the model generates coherently at arbitrary length. The key insight is that you don't need to understand why the first tokens are sinks — you just need to not evict them.

Beyond StreamingLLM

Adding a small number of dedicated "sink tokens" directly in the architecture — learnable dummy positions that exist only to absorb attention mass — removes the dependency on <bos> happening to be in the right place. Mistral's sliding window uses this idea implicitly.

Show Me the Code

Measuring the attention-mass concentration at the first few positions with a toy attention computation.

1import numpy as np2
3rng = np.random.default_rng(11)4n, d = 16, 32    # 16-token sequence, d_head=325
6Q = rng.standard_normal((n, d)) * 0.027K = rng.standard_normal((n, d)) * 0.028
9# Bias first two key positions toward being attended to (simulating sink behaviour)10K[0] += 0.15   # first token biased to attract attention11K[1] += 0.0812
13scores = Q @ K.T / d**0.514weights = np.exp(scores - scores.max(-1, keepdims=True))15weights /= weights.sum(-1, keepdims=True)   # shape: (n, n)16
17avg_weight_by_key = weights.mean(0)  # average attention each key receives18for i in range(4):19    print(f"Position {i}: avg attention received = {avg_weight_by_key[i]:.4f}")20print(f"Rest (4-15): avg = {avg_weight_by_key[4:].mean():.4f}")21# Position 0: avg attention received = ~0.12  (5-8× higher than rest)22# Position 1: avg attention received = ~0.0923# Rest: avg per position ~0.015

The first positions attract substantially more attention than the uniform 1/N = 0.0625 baseline, which is the sink phenomenon in miniature.

Watch Out For

Assuming any eviction policy works with a sliding window

It's tempting to evict tokens in any order — perhaps by lowest attention score — assuming low-weight tokens matter least. Attention sinks are the counterexample: the first tokens can simultaneously have very high attention weight and no semantic importance. Evicting by low score keeps them around; evicting by oldest position removes them first. The right policy for streaming is always: pin the sinks, evict the oldest non-sink tokens.

Thinking sink-pinning gives you a large effective context

Pinning sinks and keeping the most recent W tokens does not give you W tokens of genuine long-range context. It gives you K tokens from the very beginning (often just BOS and a couple of setup tokens) plus W-K recent tokens. Anything in the middle is gone. For tasks that need information from the middle of a long document, a retrieval step or hierarchical architecture is needed — sink-pinning enables stable streaming, not true long-context recall.

The Quick Version

The first few tokens in a sequence absorb disproportionate attention mass because the model learns to use them as numerical pressure valves during training — these are attention sinks.
Evicting them from a sliding window KV cache causes the softmax distribution to destabilise and model output to collapse.
The fix is to pin the first K tokens (typically 4) permanently and evict only from the non-sink recent window.
Sink-pinning enables infinite-length streaming inference with bounded memory, but does not preserve mid-context recall — only local and BOS context is available.
Dedicated learnable sink tokens in the architecture replace the dependency on <bos> being in the right place.