Paper breakdown

Fast Inference via Speculative Decoding

Introduced Speculative Decoding, a technique that drastically speeds up LLM inference by using a small, fast draft model to generate tokens, which a larger model then verifies in parallel.

Paper: Fast Inference from Transformers via Speculative Decoding

Authors: Yaniv Leviathan, Matan Kalman, Yossi Matias · 2022

Read the paper

Speculative decoding separated generation (drafting) from verification, achieving massive latency reductions.

The Problem

LLM generation is inherently autoregressive—you must generate token 1 before you can generate token 2. This means inference is bound by memory bandwidth, not compute. When generating text, a massive model (like a 70B parameter model) must load its entire 70B weights from GPU memory to compute just one single token.

Most of the time, the GPU's compute cores are sitting idle waiting for weights to arrive. You can't speed this up by throwing more FLOPs at it; you are fundamentally limited by how fast memory can be read.

The Idea

The Google researchers realized that verifying tokens is much faster than generating them. If you give an LLM a sequence of 5 tokens, it can verify whether it agrees with all 5 tokens in a single forward pass (in parallel), using its weights just once.

Speculative Decoding proposes using a tiny, fast "draft" model to quickly guess the next few tokens, and then using the massive, slow "target" model to verify them all at once.

How It Works

Drafting: A small, fast model (e.g., a 1B parameter model) autoregressively generates a short sequence of tokens (e.g., 5 tokens) very quickly.
Verification: The large, slow target model (e.g., a 70B parameter model) runs a single forward pass over those 5 draft tokens. It checks if it would have generated those same tokens.
Accept/Reject: The target model accepts the draft tokens up to the first token it disagrees with. It keeps the accepted tokens, outputs its own corrected token for the point of disagreement, and discards the rest.
Repeat: The process starts again from the corrected token.

Crucially, because the target model ultimately makes the final decision, the mathematical distribution of the final generated text is provably identical to what the target model would have generated on its own. There is zero degradation in quality.

Why It Mattered

Speculative decoding achieved 2x-3x speedups in inference latency for massive models without changing their weights or degrading their output quality. It turned a memory-bandwidth problem into a compute problem by keeping the GPU cores busy verifying multiple tokens at once.

What Came After

Speculative decoding spawned an entire subfield of inference optimization. Researchers developed advanced variants like Medusa (which trains multiple specialized heads on the target model itself to draft tokens, avoiding the need for a separate draft model) and EAGLE. It is now a standard feature in high-performance serving engines like vLLM and TensorRT-LLM.