AWQ

The Problem

Like GPTQ, the goal was to shrink massive LLMs from 16-bit precision down to 4-bit precision so they could run on consumer GPUs.

While GPTQ achieved this by mathematically compensating for quantization errors using the Hessian matrix, calculating the inverse Hessian can be computationally intensive and occasionally prone to instability or overfitting to the calibration dataset. The MIT researchers wanted a method that was simpler, didn't require complex error-compensation updates, and generalized better across different tasks.

The Idea

The authors made a crucial observation: not all weights are equally important. A tiny fraction of weights (around 0.1% to 1%) are "salient." If you quantize these salient weights, the model's performance collapses. If you protect them (keep them in FP16), the model works perfectly.

However, hardware doesn't like mixed precision (having some weights in 4-bit and others in 16-bit is slow to compute). The breakthrough idea was: instead of keeping salient weights in FP16, we can just scale them up (multiply them by a large factor) before applying standard 4-bit quantization.

Because standard quantization groups weights and scales them based on their maximum value, artificially scaling up the salient weights ensures they suffer far less relative rounding error.

How It Works

The magic of AWQ is how it identifies which weights are salient.

Looking at the weights alone (their magnitude) doesn't work. The authors found that the importance of a weight is actually determined by the activations flowing through it.

1. They run a small calibration dataset through the model and observe the activations.
2. If a specific input channel has consistently massive activations, the weights connected to that channel are deemed highly salient.
3. AWQ mathematically searches for an optimal scaling factor to apply to these salient channels. It scales the weights up (to reduce quantization error) and simultaneously scales the incoming activations down (to preserve mathematical equivalence).
4. The scaled weights are then simply quantized to 4-bit using basic Round-to-Nearest (RTN).

Why It Mattered

AWQ achieved state-of-the-art 4-bit quantization results, often slightly outperforming GPTQ on specific models (like LLaMA), particularly on tasks requiring deep reasoning.

Because it relies purely on scaling rather than updating weights via the Hessian, it is incredibly robust. It doesn't overfit to the calibration data, and the quantization process itself is lightning fast. Furthermore, because the final result is just standard 4-bit weights and a scaling factor, it is highly optimized for fast inference on GPUs.

What Came After

AWQ and GPTQ currently share the crown for 4-bit weight-only quantization in the open-weight community. Serving engines like vLLM natively support both.

While these methods only quantize the weights, the frontier of quantization has since moved toward W8A8 (Weight-8, Activation-8) or W4A8 quantization (like FP8), which quantizes the activations as well, drastically speeding up the actual matrix multiplication compute, not just memory loading.