GQA

The Problem

During LLM generation (inference), the model must store the Key and Value (KV) vectors for every previous token in memory to avoid recalculating them. This is called the KV cache.

In standard Multi-Head Attention (MHA), every Query head has its own unique Key and Value head. As context windows grow to 8K, 32K, or 128K tokens, the memory required to store the KV cache for all those heads becomes massive, often exceeding the memory required for the model weights themselves. This strictly limits batch sizes and severely degrades serving throughput.

An existing alternative, Multi-Query Attention (MQA), solved the memory problem by forcing all Query heads to share a single, single Key and Value head. While MQA is incredibly fast and memory-efficient, it significantly degrades model quality and performance on complex reasoning tasks.

The Idea

The authors proposed a compromise that sits exactly between MHA and MQA: Grouped-Query Attention (GQA).

Instead of having one KV head per Query head (MHA), or one KV head for all Query heads (MQA), GQA divides the Query heads into groups. Each group of Query heads shares a single KV head.

How It Works

Imagine a model with 32 Query heads.

• MHA: 32 Query heads, 32 KV heads. (High quality, massive memory).
• MQA: 32 Query heads, 1 KV head. (Low quality, tiny memory).
• GQA-8: 32 Query heads divided into 8 groups of 4. Therefore, 8 KV heads. (High quality, low memory).

The researchers also showed how to effectively "uptrain" existing MHA checkpoints into GQA models. By mean-pooling the existing KV heads within a group to create a single shared KV head, and then pre-training for a small fraction of the original training steps, the model quickly adapts to the GQA structure.

Why It Mattered

GQA was a massive practical breakthrough for LLM serving infrastructure. The paper demonstrated that GQA achieves model quality virtually indistinguishable from MHA, while achieving inference speeds and memory footprints almost identical to MQA. It essentially solved the KV cache bottleneck without sacrificing intelligence.

What Came After

GQA was universally adopted almost immediately. Meta transitioned from MHA in Llama 1 to GQA in Llama 2 (70B) and Llama 3. Mistral, Gemma, and virtually every other modern open-weight model uses GQA as its default attention mechanism.

While GQA remains the standard, recent architectures like DeepSeek-V2 (Multi-Head Latent Attention) seek to compress the KV cache even further by projecting Keys and Values into a shared, highly compressed latent space, pushing the boundary beyond what simple head-grouping can achieve.