Attention Sink
Specific initial tokens in a sequence that inadvertently absorb massive amounts of attention weight simply because they serve as a dumping ground for mass.
Think of It Like This
Like a designated junk drawer in a kitchen where you blindly toss random miscellaneous items just because they have to go somewhere.
During training, softmax forces attention weights to sum to one, meaning a model has to allocate attention somewhere even if the current context isn't relevant. It learns to dump this excess probability onto the very first tokens of the sequence. If these early sink tokens are evicted from the KV cache during long-context generation, the entire attention mechanism destabilizes and the model collapses.