KV Cache
A memory optimization in transformer models that stores previously computed Key and Value vectors during generation to prevent redundant calculations.
Think of It Like This
Like a mathematician keeping intermediate calculations on a scratchpad so they don't have to restart the whole equation for every new step.
In autoregressive generation, a transformer predicts one token at a time. Without caching, it would recompute the attention scores for all previous tokens at every step. The KV cache dramatically reduces computational overhead but consumes significant VRAM, creating a major bottleneck for long-context generation and high-throughput serving.