Attention Sinks and Streaming
The first tokens of a sequence soak up most of the attention weight, so a sliding window that evicts them collapses. Pin the first few tokens in the cache and the window can slide forever.
Why Does This Exist?
When you generate text one token at a time, each step looks back at everything written so far. The KV cache makes that affordable by storing each past token's key and value vectors instead of recomputing them. But the cache grows with every token, and on a long stream it outgrows memory. That's the wall this idea gets past.
The obvious fix is sliding window attention: keep only the W most recent tokens and let the oldest slide out. Local context carries most conversations, so what could go wrong? Everything, it turns out. The moment the first tokens leave the cache, output quality falls off a cliff. Not a gradual fade. A collapse.
The reason is the subject of this page. The first few tokens quietly carry most of the attention weight in almost every layer, and the softmax has nowhere to put that weight once they are gone. You will see why they become attention sinks, how pinning them fixes streaming, and what pinning cannot give you. The cache mechanism itself lives on the KV cache page, the windowing policy on sliding window attention, and genuine long-range recall on long-context extension.
One prerequisite, in two sentences. For a single query, attention scores each past token, runs the scores through a softmax so the weights sum to 1, and returns a weighted blend of the tokens' values. If that machinery is hazy, read self-attention first. The rest of this page assumes you know the weights sum to 1.
Think of It Like This
A kitchen junk drawer
Every kitchen has one drawer where batteries, rubber bands, and takeout menus land. Nobody planned it. It is simply the first drawer by the door, so it is always there when your hands are full, and it keeps the counters clear for actual cooking.
Attention sinks are that drawer. Each query walks in carrying weight it must put somewhere, because the weights have to sum to 1, and the first tokens are always there. So the clutter lands on them and the content tokens stay clean. Take the drawer out and the clutter lands on the counters: some random token gets buried under weight it was never meant to hold.
Where it stops: you can open a junk drawer and still use what is inside. A sink token's content barely matters. Only its position does. And clearing a real drawer never breaks the kitchen, while evicting a sink breaks the whole distribution.
How It Actually Works
For one query vector (what the current token is looking for) and one key vector per past token (what token offers), the score is their scaled dot product:
Here is the head dimension, a fixed size like 64 or 128, and dividing by keeps the scores near unit variance so the softmax stays well behaved. The attention weight on token is then:
with the exponential function and the sum running over every key in the cache. Those weights sum to 1 by construction, and the layer output is the weighted blend of the value vectors . So far this is ordinary attention. The sink story starts with what the model does with the freedom inside it.
Why the mass piles onto the first tokens
Most queries match nothing decisively. When no key stands out, the softmax still has to hand out 100 percent of its weight, and spreading it evenly across thousands of tokens would wash the output out. During training the model finds a cheaper answer: give the earliest keys large dot products with almost everything, so they absorb the leftover mass by default.
The first tokens are the natural choice because every later query can see them. Under causal masking, token 0 is visible to all queries after it, where is the sequence length, while a middle token is visible to only about half that many. So the dump lands where it is always reachable: usually the first few positions, starting with the beginning-of-sequence token. This shows up in almost every layer and head, for almost every input. It isn't about meaning. A token can dominate a head's weight while contributing almost nothing semantically.
The fix: pin the first tokens
StreamingLLM keeps the first tokens permanently in the cache and runs the sliding window over the rest. With a total budget of slots, where is the window size, that means pinned sinks plus the most recent tokens, evicting oldest-first among the non-sink entries only. The default is . Memory stays bounded at entries no matter how long the stream runs, and the softmax keeps its dump buckets, so generation stays coherent indefinitely.
Name the price honestly. You get stability, not recall. The cache holds the opening tokens plus recent ones, and everything in the middle is gone. If your task needs facts from the middle of a long document, pinning will not save you. Reach for retrieval or long-context extension instead. The fix also assumes the sink sits among the first tokens. That holds in practice across layers and heads, but if some layer parks its mass elsewhere, that layer still wobbles.
Worked example
One query looks at four cached keys with scores . Exponentiate: . The sum is . Divide:
Token 0, the sink, holds 0.82 of the weight. Now evict it, the way a naive window does, and renormalize over . The exponentials are , the sum is , and the weights become . Tokens that held 0.07 suddenly carry 0.38 each, over a fivefold jump onto positions the model never trained to weight that heavily. That jump is the collapse.
Pin the sink instead and evict the oldest non-sink token. Renormalize over : exponentials , sum , weights . The mass stays parked where training put it, and the window is free to slide.
Code
This snippet reproduces the worked example so you can poke at it. Change the scores and watch where the mass goes.
import math
def softmax(xs: list[float]) -> list[float]: m = max(xs) exps = [math.exp(x - m) for x in xs] s = sum(exps) return [e / s for e in exps]
scores: list[float] = [3.0, 0.5, 0.5, 0.0]w = softmax(scores)print([round(v, 3) for v in w]) # -> [0.824, 0.068, 0.068, 0.041]print(round(sum(w), 3)) # -> 1.0print([round(v, 3) for v in softmax(scores[1:])]) # -> [0.384, 0.384, 0.233]print([round(v, 3) for v in softmax([scores[0]] + scores[2:])]) # -> [0.883, 0.073, 0.044]The evicted run matches the spill, and the pinned run matches . Same fixture as the diagram above, so the three check each other.
Watch Out For
Pinning the sinks and calling it long context
The symptom looks like memory but isn't. With sinks pinned and a window of recent tokens, your model chats fluently about the document opening and the last few pages, then answers questions about the middle with confident fabrications. The middle tokens are simply gone, and fluency hides their absence until someone checks a fact.
Treat sink pinning as stability, never as recall. If the task needs the middle, add a retrieval step or move to long-context extension. Judge a streaming setup by middle-context questions, not by how smooth the prose reads.
Pinning only the very first token
The symptom is a wobble that survives your fix. You pin token 0, stream again, and quality still dips on some inputs, because the sink mass spreads over the first few positions rather than sitting on exactly one. Token 0 holds most of it, but tokens 1 to 3 carry enough that evicting them still dents the distribution.
Pin the first 4 tokens, the StreamingLLM default, and evict oldest-first only among the rest. If the wobble persists after that, the problem sits elsewhere, so check the window size before blaming the sinks.
The Quick Version
- Attention sinks are the first few tokens of a sequence, which absorb most of a head's softmax weight as numerical ballast rather than as meaningful content.
- A sliding window that evicts them forces their weight onto random content tokens, and generation collapses instead of degrading gracefully.
- Pinning the first 4 tokens while the window slides over the rest keeps generation stable with bounded memory at any stream length.
- Pinned sinks buy stability, not recall: middle context is still evicted, so mid-document questions need retrieval or a longer true context.
- One query's weights always sum to 1, so every streaming fix is really a decision about where leftover weight is allowed to sit.