Sliding Window Attention
A localized attention mechanism where tokens only interact with a fixed number of neighboring tokens, drastically reducing compute for long sequences.
Think of It Like This
Like reading a scroll where you only keep the last 50 words in your active memory to understand the current sentence, ignoring the beginning.
Standard attention requires quadratic compute relative to sequence length. By restricting a token to only attend to the previous 'W' tokens, the compute scales linearly. Models like Mistral use this technique heavily. To maintain global context, these models often rely on the deeper layers to naturally aggregate the local windows over time.