Temperature & Sampling

Why Does This Exist?

When an LLM finishes its forward pass, it outputs a list of raw, unnormalized numbers called logits—one for every word in its vocabulary. To pick the next word, we run these logits through a function called the softmax, which squashes them into a proper probability distribution that sums to 1.0 (e.g., "apple" is 80% likely, "banana" is 15% likely, "car" is 5% likely).

If we sample from this distribution natively, we get whatever balance of predictability and randomness the model naturally learned during training. But what if we want the model to write a highly creative, unpredictable poem? Or what if we want it to output strict, predictable JSON code?

Temperature exists to give the user a dial to override the model's natural confidence. By applying a simple mathematical division before the softmax, we can artificially sharpen the distribution (making the model fiercely confident in its top choices) or flatten it (forcing the model to gamble on unlikely words).

Think of It Like This

How It Actually Works

The Math: $q_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$

The mathematical implementation of temperature is remarkably simple. We take the raw logits ($z$) and divide them by the temperature ($T$) before feeding them into the standard exponential softmax function.

$T < 1$: Sharpening the distribution (Cold)

If you divide a set of numbers by a decimal like 0.5, you are effectively multiplying them by 2. This stretches the distance between the highest logit and the rest of the pack. When these stretched numbers are exponentiated in the softmax, the leading token dominates entirely, often jumping to a 99% probability. The generation behaves almost exactly like greedy decoding, reliably outputting the single safest answer. Use cases: Code generation, data extraction, math, factual Q&A.

$T = 1$: Natively calibrated (Neutral)

Dividing by 1 leaves the logits unchanged. You get exactly what the model learned during training.

$T > 1$: Flattening the distribution (Hot)

If you divide the logits by a large number like 2.0, you shrink the distance between them. The highest logit and the lowest logit are pulled closer to zero, making them mathematically closer together. When exponentiated, the resulting probabilities flatten out toward a uniform distribution. A token that was natively a 1% chance might suddenly become a 15% chance. The model will frequently choose strange, unlikely words. If pushed too high (e.g., T=5), the model devolves into complete gibberish. Use cases: Brainstorming, creative writing, breaking out of repetitive loops.

Show Me the Code

You can see exactly how the temperature parameter reshapes a set of raw logits before sampling.

1import numpy as np2
3def temperature_softmax(logits: np.ndarray, temperature: float) -> np.ndarray:4    # Divide the logits by the temperature5    scaled_logits = logits / temperature6    7    # Standard softmax math (subtracting max for numerical stability)8    e = np.exp(scaled_logits - np.max(scaled_logits))9    return e / np.sum(e)10
11# Raw output from the model: Token 0 is the favorite, Token 1 is second.12logits = np.array([4.0, 2.0, -1.0, -3.0]) 13
14# T = 1.0 (Standard)15print(f"T=1.0 : {temperature_softmax(logits, 1.0).round(3)}")16# -> [0.88 0.119 0.006 0.001] (Token 0 has a commanding 88% lead)17
18# T = 0.2 (Cold/Greedy-like)19print(f"T=0.2 : {temperature_softmax(logits, 0.2).round(3)}")20# -> [1. 0. 0. 0.] (Token 0 is now 100% certain. Token 1 is eradicated)21
22# T = 3.0 (Hot/Creative)23print(f"T=3.0 : {temperature_softmax(logits, 3.0).round(3)}")24# -> [0.575 0.295 0.108 0.056] (Token 0 dropped to 57%. The rare tokens now have a real chance)

Notice how T=0.2 entirely destroys the chance of drawing anything other than the highest-ranked token, while T=3.0 gives even the deeply negative logits a fighting chance.

Watch Out For

The Quick Version

• Temperature is a mathematical dial that reshapes a model's confidence before it chooses the next word.
• The raw logits are divided by the temperature value prior to being passed into the softmax function.
• A low temperature (T < 1) stretches the logits, making the top choice overwhelmingly likely (predictable, safe).
• A high temperature (T > 1) compresses the logits, flattening the probabilities and giving rare words a chance (creative, erratic).
• Most APIs map a temperature of exactly 0 to a purely deterministic greedy decoding path.

What to Read Next

• Top-K Sampling explains another method to control randomness, typically applied after temperature.
• Decoding Strategies gives the overarching view of how all these sampling techniques compare to greedy and beam search.
• Top-P Sampling (upcoming) is the dynamic safety net that is almost universally paired with high temperatures to prevent gibberish.