Efficient Streaming LMs with Attention Sinks
Discovered that LLMs naturally use the first few tokens as 'attention sinks' to dump excess attention scores, enabling a simple fix to keep models generating text infinitely without crashing.
Paper: Efficient Streaming Language Models with Attention Sinks
Authors: Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis · 2023
Read the paper
The Problem
Standard LLMs are trained with a fixed context window (e.g., 4K tokens). If you want an LLM to act as a continuous streaming agent (like an AI companion that runs forever), it will eventually run out of KV cache memory.
The obvious solution is Windowed Attention: when the cache is full, just evict the oldest tokens and only keep the most recent 4K tokens in memory. However, researchers found that the moment a model evicts the very first tokens of the sequence, its intelligence completely collapses, and it starts generating gibberish.
The Idea
The MIT and Meta researchers investigated why evicting the first token destroyed the model. They looked at the raw attention maps across all layers.
Because of the Softmax operation in the attention mechanism, all attention scores must sum to 1. Often, an attention head doesn't actually "want" to look at any of the previous tokens, but the math forces it to allocate its attention somewhere.
They discovered that the model naturally learns to use the very first token in the sequence (often a generic <s> or [BOS] token) as a "sink." Whenever an attention head doesn't have anything useful to look at, it dumps all of its excess attention probability onto that first token.
If you evict that first token to save memory, the model is suddenly forced to dump its excess attention onto a random semantic word, which completely corrupts the representations and causes the catastrophic collapse.
How It Works
The solution is remarkably simple. To achieve infinite streaming generation without the model collapsing, you just need to keep the attention sink.
The StreamingLLM framework modifies the KV cache eviction policy:
- It permanently pins the first 4 tokens of the prompt in the KV cache (the Attention Sinks).
- It keeps the most recent tokens in a rolling window.
- When new tokens arrive, it evicts the oldest tokens in the rolling window, but strictly preserves the pinned sink tokens.
By simply never deleting the first 4 tokens, the model can generate text smoothly for 4 million+ tokens with zero performance degradation, using a tiny, fixed amount of memory.
Why It Mattered
This paper solved a massive problem for long-running agentic AI. It allowed any off-the-shelf LLM (like Llama 2 or Mistral) to be instantly converted into an infinite-streaming model without requiring any fine-tuning or architectural changes.
It was also a profound interpretability finding. It revealed a fascinating quirk of how neural networks adapt to the strict mathematical constraints of the softmax function, completely independent of the actual semantic meaning of the text.
What Came After
StreamingLLM was widely integrated into serving frameworks for specific agentic use cases. While it solves the streaming generation problem (allowing infinite output), it does not allow the model to remember the evicted text. For tasks requiring true long-term memory, this approach is typically combined with external RAG databases or has been superseded by massive context models (like Gemini 1.5 Pro's 2M token window), assuming the user is willing to pay the massive compute cost to keep everything in context.