Gen AI

Speculative Decoding

Instead of waiting for a massive 70B model to generate one token at a time, use a tiny 1B model to quickly guess the next 5 tokens, then have the massive model grade all 5 guesses at once in a single, lightning-fast parallel pass.

Why Does This Exist?

Generating text from a massive LLM (like a 70-billion parameter model) is agonizingly slow. Because of the autoregressive bottleneck—you cannot generate token 3 until you know what token 2 is—the GPU is forced to load the entire 140GB of model weights from memory into the compute cores for every single token. Memory bandwidth, not compute math, is the ultimate bottleneck.

But here is a structural secret about transformers: they process prompts in parallel. If you hand a transformer a 5-word prompt, it doesn't process them one by one. It processes all 5 words simultaneously in a single pass.

Speculative decoding exploits this perfectly. It realizes that if we can somehow guess the next few words using a cheap, tiny model, we can feed those guesses to the massive model as a "prompt". The massive model can then verify all of our guesses in parallel, in the exact same amount of time it would normally take to generate a single token.

Think of It Like This

The senior executive and the eager intern

Imagine a highly-paid, brilliant CEO (the 70B Target Model) writing an important company memo. Normally, the CEO sits alone, slowly typing one word at a time. It takes hours.

Now imagine the CEO hires a fast, eager intern (the 1B Draft Model). The intern rapidly types out the next 5 words they think the CEO wants to say, and hands the draft to the CEO. The CEO can instantly read all 5 words at once. If the intern was right, the CEO signs off on the whole block and moves on—5 words completed in the time it takes to read 1. If the intern makes a mistake on word 3, the CEO crosses out words 3, 4, and 5, corrects word 3 themselves, and tells the intern to try again.

Because reading (parallel verification) is infinitely faster than writing (autoregressive generation), the duo produces the memo in record time.

How It Actually Works

1. The Draft Phase

The system runs a tiny, highly efficient "draft" model (e.g., a 1B parameter version of the same architecture). Because the model is so small, its weights easily fit in the fastest layers of GPU cache, allowing it to autoregressively generate 3 to 5 tokens incredibly fast. (e.g., "The", "cat", "sat", "on").

2. The Verification Phase

The system takes the original context plus the 4 drafted tokens and passes them into the massive "target" model (e.g., 70B parameters). The target model runs a single, parallel forward pass over the entire sequence.

At every position, the target model outputs its own probability distribution. The system compares the draft model's token against the target model's distribution.

3. Acceptance and Rejection

The math is clever. If the target model agrees with the draft token (meaning the token falls within the target's acceptable probability distribution), the token is accepted. If the draft token is wildly wrong, it is rejected. At the exact moment of rejection, the system throws away that token and all tokens drafted after it. However, because the target model already computed its own probability distribution for that position during the verification pass, the system simply samples the correct token directly from the target model's output.

The Guarantee

The most magical part of the speculative decoding math is that it is mathematically lossless. Due to a technique called modified rejection sampling, the final generated text is mathematically identical to what the 70B model would have generated on its own. You lose absolutely zero quality, you just get the text 2x to 3x faster.

Show Me the Code

This code conceptually models the verification phase, showing how a sequence of drafts is accepted until a failure occurs, at which point the target model's own calculation is substituted in.

1def verify_speculative_draft(draft_tokens: list[str], target_model_predictions: list[str]):2    accepted = []3    4    # Iterate through the parallel verification results5    for i, draft_tok in enumerate(draft_tokens):6        target_tok = target_model_predictions[i]7        8        if draft_tok == target_tok:9            print(f"Step {i}: Draft '{draft_tok}' ACCEPTED.")10            accepted.append(draft_tok)11        else:12            print(f"Step {i}: Draft '{draft_tok}' REJECTED. Target wanted '{target_tok}'.")13            # We take the target model's correct prediction, append it, 14            # and instantly discard the rest of the draft.15            accepted.append(target_tok)16            break 17            18    return accepted19
20# The intern drafted 4 words21draft = ["The", "cat", "sat", "on"]22
23# The CEO verified them all in a single parallel pass24target_preds = ["The", "cat", "jumped", "over"]25
26final_tokens = verify_speculative_draft(draft, target_preds)27print(f"Final Output: {final_tokens}")28
29# -> Step 0: Draft 'The' ACCEPTED.30# -> Step 1: Draft 'cat' ACCEPTED.31# -> Step 2: Draft 'sat' REJECTED. Target wanted 'jumped'.32# -> Final Output: ['The', 'cat', 'jumped']

In a single GPU pass, we secured 3 valid tokens instead of 1. If we hadn't used speculative decoding, getting those 3 tokens would have required 3 separate, massive memory-bandwidth-choking passes through the 70B model.

Watch Out For

Choosing a bad draft model

The entire speedup hinges on the "acceptance rate" (how often the CEO agrees with the intern). If your draft model is too stupid, it will get every word wrong. You will spend compute time running the draft model, only for the target model to instantly reject word 1 every time, making your overall system slower. Draft models must be highly aligned with their target models (often trained on the exact same data) to achieve the 60%+ acceptance rates required for massive speedups.

The Quick Version

Generating text token-by-token from a massive LLM is bounded by memory bandwidth, not compute.
Speculative decoding uses a tiny, fast model to draft 3 to 5 tokens sequentially.
The massive target model processes those drafted tokens in a single, parallel batch, verifying them simultaneously.
When the target model rejects a token, it seamlessly substitutes its own correct calculation and discards the rest of the draft.
The math guarantees zero loss in output quality, effectively delivering a 2x-3x speedup for "free" (at the cost of slightly more complex serving infrastructure).