Transformer Decoding

Generating text one token at a time means making one decision per token. But the model's output is not a word — it is a score for every token in the vocabulary. How you turn those scores into a choice changes the tone, the variety and the coherence of everything that follows, and it does so without touching the model at all.

What is producing these numbers

The model here is not a transformer. It is a bigram count model over seven short sentences: it has seen which words followed which, and logit = log P(next | current) comes straight out of those counts, with a small constant added to every pair so nothing is impossible.

That is a deliberate substitution, and it is worth being clear about what survives it. A transformer conditions on the whole context rather than the previous word, and it has billions of parameters rather than a table of nine. What it produces at each step is nevertheless the same kind of object: one distribution over the vocabulary. Every decoding strategy below acts on that object and nothing else — so on this model they do exactly what they do on a real one, at a scale you can read off the screen.

Nine tokens, and the corpus is:

• the model reads the sentence
• the model writes the sentence
• the sentence teaches the model
• a model reads a word
• a word follows a word
• the model writes a word
• the model writes a sentence

Greedy decoding has to repeat itself

The simplest strategy is to take the tallest bar every time. It is deterministic, it always picks the most likely continuation, and it cannot work.

Press Step and watch. the to model to writes to a to word to follows to a to word to follows. By the ninth token the text is periodic and it will stay periodic forever.

This is not bad luck with this corpus. Greedy decoding is a function from the current token to the next one, the same function every time, over a finite vocabulary. Iterate any such function and you must eventually land on a token you have already visited — and from there the sequence you produce is the one you produced last time you were there. The loop is arithmetic, not a modelling failure. Real models are conditioned on more than one previous token, which makes the state space enormous rather than nine, so it takes longer — but greedy decoding on a long generation still degenerates, and this is why.

Note also what does not help: no Top-K or Top-P setting changes greedy's output at all. Truncation zeroes the tail and rescales what is left; it never changes which token is at the top.

Temperature: sampling, and how sharply

Draw the token instead of taking the maximum, and the loop breaks — the walk stops being deterministic, so it stops being forced to close.

Temperature divides the logits before softmax. Below 1 it exaggerates the differences and pushes sampling back toward greedy; above 1 it compresses them and pushes toward uniform. Effective choices puts a number on where you are: it is exp(H), the perplexity of the distribution, running from 1 (all the mass on one token) to 9 (spread evenly). From a it reads 1.00 at temperature 0.1, 2.68 at 1.0, and 5.12 at 2.0.

The cost of flattening, and what truncation is for

Flattening does not only add variety. It moves probability onto continuations the corpus never contained — and that is a number too. Off-corpus mass is the total probability sitting on tokens this word has never been followed by: 1.9% from a at temperature 1, 8.8% at 1.5, 17.4% at 2.0. Every one of those percent is a chance, this step, of writing a pair the model has no evidence for.

That is the problem Top-K and Top-P exist to solve, and both do it by zeroing part of the distribution and renormalising what remains.

Top-K keeps a fixed number of the highest-scoring tokens. Set it to 3 and the six least likely go hatched and stop being drawable, whatever the temperature. From a at temperature 1.5 it takes the off-corpus mass from 8.8% to zero.

Top-P keeps tokens until their cumulative probability reaches P — a fixed mass rather than a fixed count, so it takes two tokens when the model is confident and many when it is not. That adaptivity is usually what you want. It also has a consequence worth seeing: as the distribution flattens, the nucleus has to grow to hold the same 90%, so Top-P loosens exactly when the tail is at its worst. From model, P = 0.9 keeps 2 tokens at temperature 1 and 7 at temperature 2. A hard Top-K of 3 keeps 3 at both.

The band, not the setting

There is no best configuration, and both walls are real:

• Below about temperature 0.5, sampling is greedy in all but name and the text loops.
• Above about 1.2 with nothing truncated, unseen transitions start appearing — one by 1.2, three by 1.5, four by 2.0 over the same eleven steps.
• Over-truncate and you are back at the first wall: Top-K of 2 loops at every temperature, because two options is not enough room to leave the cycle.

The usable band is roughly temperature 0.8 to 1.2 with Top-K at 3 or more. Which is to say: enough randomness to avoid saying the same thing forever, and enough restraint to only say things the model has evidence for.