Gen AI

Quantization Quality Tradeoffs

Quantization shrinks a model's numeric precision to save memory, but the quality it costs shows up unevenly, worse in long context, code, and weaker languages.

Quantization does not cost every task the same amount of quality; short single-language answers barely move, long context and code drop more, and a low-resource language drops the most

Why Does This Exist?

A model's weights usually start out as 16-bit floats. Quantization converts them to a format with fewer bits — 8-bit, often 4-bit — so the model takes less memory and runs faster, which matters on a GPU short on VRAM, or an edge device with none.

Precision reduction is compression, and compression isn't free. Every weight gets rounded to the nearest value the smaller format can represent, injecting a small numerical error into every computation it touches. Less obvious, and the point of this page, is that the error doesn't land evenly: quantizing a model doesn't make it uniformly a little worse, it makes it a little worse at some things and a lot worse at others, and which is which is predictable.

Here's a concrete version. A team fine-tunes a coding-and-multilingual support assistant, then quantizes it to 4-bit for a memory-constrained edge device. They run a few quick English sanity prompts and it looks fine, so they ship it. Weeks later, it's quietly worse over long support conversations, writes code that's subtly broken in specific situations, and answers noticeably worse once a customer switches to their non-English support language. None of that showed up in the five-minute check, because that check is exactly the condition quantization damages least.

Think of It Like This

Reading through a lens with one fixed scratch

Put on glasses with one light scratch across each lens. A bold headline in large print still reads fine — the scratch doesn't blur enough to matter. A dense paragraph of small type is harder, and by the third page you're filling in gaps the scratch keeps introducing. Try to read sheet music, where a note's exact position is the entire content, and the scratch changes which note you think you're looking at. Try a script you're less practiced with, one whose strokes you were already relying on context to disambiguate, and the scratch costs you the most of all.

The scratch is quantization: one lens, one fixed distortion, applied everywhere at once. What changes case to case isn't the scratch — it's how much each kind of reading depended on the precision it just took away.

How It Actually Works

What quantization actually changes, and why it isn't uniform

Quantization rounds weights down to fewer representable values at every layer, injecting a small numerical error into nearly every matrix multiplication the model performs. That per-operation error is roughly constant — it doesn't care what task it's part of. What varies is how much a task's correctness depends on those errors staying small and not compounding. A short factual answer like "the capital of France" rides on one high-confidence token choice; a small nudge from rounding rarely flips it. A task built from thousands of smaller, less-certain decisions, each feeding into the next, has far more places for a small error to change the outcome.

Long context and code as the two sharpest failure surfaces

Two kinds of tasks expose this fastest. The first is long context. Every added token brings another round of attention over everything already generated, and a per-token error invisible at 500 tokens gets far more chances to matter by 50,000 — the error itself doesn't grow, but opportunities to matter stack up. A model can look fine on a short prompt and drift, subtly, over a real conversation — exactly what the team's deployment hit.

The second is code. Natural language tolerates a "close enough" word choice; a slightly different adjective rarely breaks a sentence. Code has no such slack. One wrong token can be a mismatched bracket, an off-by-one, or a flipped comparison operator, and the result isn't "slightly worse code" — it's code that produces the wrong answer, or doesn't run at all.

Multilingual degradation, and the evaluation discipline this implies

The same logic applies to language. A model's dominant training language — usually English — got the most training data and the most redundant ways to represent the same idea, leaving more slack for quantization to eat into before output changes. A lower-resource language had less redundancy to begin with, so its representation was already less robust before anyone touched a weight. Quantize the model and that thinner representation degrades faster — exactly what the team's assistant showed once customers switched languages.

The discipline this implies is simple, even though skipping it is common: evaluate a quantized model on held-out tasks, in the specific conditions it'll actually be used in, rather than a quick, single-language sanity check. That check just measures the condition quantization degrades least, so passing it tells you almost nothing about the conditions that matter.

Show Me the Code

A minimal tracker for per-category accuracy before and after quantization, so uneven degradation shows up as a number, not a guess.

1from dataclasses import dataclass2
3
4@dataclass5class BucketResult:6    category: str7    accuracy_before: float8    accuracy_after: float9
10    @property11    def delta(self) -> float:12        return self.accuracy_after - self.accuracy_before  # negative = degraded13
14
15def worst_buckets(results: list[BucketResult], top_n: int = 2) -> list[BucketResult]:16    """Return the buckets that dropped the most after quantization."""17    return sorted(results, key=lambda r: r.delta)[:top_n]18
19
20results = [21    BucketResult("short_english_qa", 0.94, 0.93),22    BucketResult("long_context", 0.91, 0.79),23    BucketResult("code_generation", 0.88, 0.74),24    BucketResult("low_resource_language", 0.86, 0.68),25]26
27for bucket in worst_buckets(results):28    print(bucket.category, round(bucket.delta, 2))29    # -> low_resource_language -0.1830    # -> code_generation -0.14

The English bucket barely moves; the other three don't — sorting by delta surfaces that ranking instead of hiding it inside one averaged score.

Watch Out For

Validating with a quick, single-language, short-context sanity check and shipping on it alone

A handful of short English prompts catches a badly broken quantization job, but is a poor way to catch a working-but-degraded one, since it's exactly the condition quantization damages least. Passing it means the model still functions — it says nothing about a long conversation, a real code task, or a different language, precisely where this page's failure modes live.

Assuming a quantization method that worked on one model or task generalizes untested

A 4-bit scheme that held up fine on one model's English benchmark doesn't automatically hold up on a different model, task, or language — how much quality a task loses depends on how much slack that model's representation of it had to begin with. Re-check on the new combination rather than carrying over a result from a different one.

The Quick Version

Quantization rounds weights to a lower-precision format to save memory and speed inference, introducing a small numerical error everywhere it happens.
The error size is roughly constant, but its effect on output isn't — precise, compounding computation loses more than short, high-confidence answers.
Long context degrades because small per-token errors compound across thousands of tokens; code degrades because exact syntax tolerates no "close enough" token.
Lower-resource languages degrade faster than the dominant training language, since their representation had less redundancy to fall back on.
Evaluate a quantized model in the conditions it'll actually ship in — long conversations, real code, the real target languages — not a quick single-language check.