DeepSeek-V2

The Problem

While Grouped-Query Attention (GQA) significantly reduced the size of the KV cache compared to standard Multi-Head Attention, it was still a major bottleneck for serving massive models with long context windows. For an MoE model like Mixtral 8x22B, the KV cache for a 128K context window still consumes tens of gigabytes per user, making API serving incredibly expensive.

Simultaneously, standard MoE routing (like top-2 routing to 8 large experts) occasionally suffered from "knowledge collapse," where large chunks of specialized knowledge were awkwardly distributed among massive experts.

The Idea

The DeepSeek researchers completely reimagined both the attention mechanism and the MoE routing.

For attention, they realized that instead of caching massive, high-dimensional Key and Value vectors, they could compress all that information into a single, highly compressed latent vector, and dynamically decompress it on the fly during inference.

For MoE, instead of having a few large experts, they hypothesized that having a massive number of tiny, "fine-grained" experts would allow the router to isolate and combine knowledge much more precisely.

How It Works

1. Multi-Head Latent Attention (MLA): Instead of computing and caching Keys and Values, MLA takes the hidden state and projects it down into a tiny, shared latent vector (e.g., dimension 512). This latent vector is the only thing stored in the KV cache.

During generation, this single tiny vector is read from memory, broadcast to all attention heads, and rapidly multiplied by specific weight matrices to mathematically reconstruct the Keys and Values on the fly. Because math is cheap but memory bandwidth is expensive, this trade-off results in a 93% reduction in KV cache memory compared to MHA, blowing past the efficiency of GQA.

2. DeepSeekMoE: Instead of 8 large experts (like Mixtral), DeepSeek-V2 uses 160 tiny experts. The router selects the top 6 experts per token. Crucially, they introduced Shared Experts: a few experts that are always active for every token, ensuring that common knowledge and syntax are universally available without clogging up the routing network.

Why It Mattered

DeepSeek-V2 shocked the industry. With a total of 236B parameters but only 21B active parameters per token, it rivaled GPT-4 on reasoning and coding benchmarks, but at a fraction of the training and serving cost.

The introduction of MLA was the biggest architectural jump in attention mechanics since FlashAttention. By slashing the KV cache by over 90%, it allowed DeepSeek to price their API at a historically low cost ($0.14 per million tokens), fundamentally disrupting the LLM economic landscape.

What Came After

DeepSeek-V2's architecture was an overwhelming success and was rapidly succeeded by DeepSeek-V3 and DeepSeek-R1, which scaled the exact same MLA and fine-grained MoE architecture to achieve absolute state-of-the-art performance, confirming that this highly compressed, highly sparse architecture is currently the most efficient way to scale intelligence.