PagedAttention
A memory management technique inspired by operating systems that partitions the KV cache of large language models into non-contiguous blocks of memory.
Think of It Like This
Like storing a massive file by splitting it into small chunks scattered across a hard drive, rather than requiring one huge block of free space.
Standard attention requires contiguous VRAM for a sequence's KV cache, leading to massive memory fragmentation and waste. PagedAttention solves this by dynamically allocating small, fixed-size blocks of memory for tokens as they are generated. This allows systems like vLLM to dramatically increase serving throughput and support complex decoding strategies.