Decoding Strategies

Why Does This Exist?

How LLMs work established that a model's forward pass produces a probability distribution over the next token — but a distribution isn't a token. Something has to choose one specific token from that distribution before generation continues, and that choice is a separate decision from everything the model itself computed. The same distribution can turn into wildly different generated text depending on how that final choice is made.

This matters because the choice isn't neutral. Always picking the single most probable token sounds like the obviously correct choice, but it produces noticeably duller, more repetitive text than sampling with some randomness does — a genuinely counterintuitive finding that took the field a while to fully internalize. The decoding strategy is a separate lever from the model itself, and different tasks call for different settings on it.

Think of It Like This

How It Actually Works

Greedy: always the single best next token

Greedy decoding picks $\arg\max$ over the probability distribution at every step — whatever token the model currently rates most likely, every time, with no randomness at all. It's simple, fast, and completely deterministic: the same prompt always produces the exact same output. Its failure mode is also well documented: greedy decoding tends toward repetitive, generic text, because the single highest-probability token at each step is often the safest, blandest continuation, and committing to it at every step compounds that blandness across the whole output.

Sampling: drawing from the full distribution

Rather than always taking the top token, sampling draws randomly from the distribution itself, weighted by each token's probability — a token rated at 30% gets chosen roughly 30% of the time, not 0% just because it wasn't the single highest. This introduces genuine variation between runs on an identical prompt, and produces text that reads more natural and less repetitive than greedy output, at the cost of losing perfect determinism. Temperature controls how sharply sampling favors high-probability tokens: dividing logits by temperature before the softmax sharpens the distribution as temperature drops toward zero, and flattens it toward uniform randomness as temperature rises.

Beam search: tracking several candidates at once

Beam search maintains several candidate partial sequences — "beams" — in parallel, rather than committing to one choice immediately. At each step, every current beam extends by its most likely next tokens, the candidates get scored by running total probability, and only the top-scoring beams survive into the next step. This lets beam search recover from a locally suboptimal choice a greedy approach would be stuck with, which is why it excels at translation, where there's often one clearly correct output worth searching for. For open-ended generation, beam search tends toward oddly generic, repetitive text for reasons related to greedy's failure — it still optimizes for the highest-probability overall sequence, which biases toward safe, common phrasing.

Everything else is a refinement of these three

Top-k sampling restricts sampling to only the $k$ most probable tokens before drawing, cutting off the long unlikely tail. Top-p (nucleus) sampling keeps whatever smallest set of tokens accounts for a cumulative probability $p$, adapting the cutoff to how spread out the distribution is at each step. Both refine plain sampling to avoid the rare risk of drawing an extremely unlikely, nonsensical token by chance. Every technique here chooses, differently, among the same distribution the forward pass already produced — none change what the model computed, only what gets selected from it.

Show Me the Code

Greedy and temperature-scaled sampling implemented directly over the same distribution, so the difference in mechanism is explicit.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e = np.exp(z - z.max())6    return e / e.sum()7
8
9def greedy_decode(logits: np.ndarray) -> int:10    return int(np.argmax(logits))                        # always the single highest score11
12
13def sample_decode(logits: np.ndarray, temperature: float, rng: np.random.Generator) -> int:14    probs = softmax(logits / temperature)                  # temperature reshapes the distribution15    return int(rng.choice(len(probs), p=probs))16
17
18logits = np.array([2.0, 3.5, 1.0, 0.5, 2.8])                # five candidate next tokens19rng = np.random.default_rng(0)20
21print(greedy_decode(logits))                                 # -> 1 — always this one, every call22print(sample_decode(logits, temperature=1.0, rng=rng))        # -> varies run to run23print(sample_decode(logits, temperature=0.01, rng=rng))       # -> almost always 1, approaching greedy

Greedy decoding on the same logits always returns index 1, deterministically. Sampling at a low temperature converges toward that same deterministic behavior, while a higher temperature spreads choices more evenly across the candidates — the same underlying distribution, read differently by each strategy.

Watch Out For

The Quick Version

• The model's forward pass produces a probability distribution; decoding strategy is the separate choice of how to turn that into an actual token.
• Greedy decoding always takes the single highest-probability token — deterministic, but prone to repetitive, generic output.
• Sampling draws randomly weighted by probability; temperature controls how sharply it favors high-probability tokens.
• Beam search tracks several candidate sequences in parallel, useful when there's a genuinely correct answer to converge on, less useful for open-ended generation.
• Top-k and top-p sampling are refinements that cut off the unlikely tail of the distribution before drawing.

What to Read Next

• How LLMs Work is the pipeline that produces the probability distribution every decoding strategy on this page operates over.
• KV Cache is what makes the repeated generation loop these strategies run inside of efficient at each step.
• In-Context Learning shapes the distribution decoding strategies choose from, based purely on prompt content.
• Context Windows limits how much prior context can influence the distribution these strategies sample from.