Top-k / Top-p
Two primary decoding strategies used during text generation to truncate the vocabulary distribution, balancing model creativity and factual coherence.
Think of It Like This
Like a manager asking a team for ideas but instantly dismissing the bottom 90% worst suggestions before picking one randomly.
Top-k sampling rigidly forces the model to only select from the 'k' most probable tokens. Top-p (nucleus) sampling selects from the smallest set of tokens whose cumulative probability exceeds 'p'. Both methods prevent the long tail of highly unlikely, nonsensical words from ever being generated, stabilizing the output.