Gen AI

Perplexity

Take the average negative log-probability a model assigns its own held-out text and exponentiate it — a confident, correct model scores close to one, a confused model scores much higher.

Perplexity is the exponent of the average negative log-probability the model assigned its own held-out text — a confident, correct model scores close to one, a confused model scores much higher

Why Does This Exist?

Training a model on next-token prediction produces a very specific kind of output: at every position, a full probability distribution over the entire vocabulary for what comes next. Comparing two trained models by eyeballing a handful of their generated outputs is subjective and doesn't scale — you need a single number that summarizes, quantitatively, how well a model's predicted distributions actually matched real text it wasn't trained on.

Perplexity is that number, and it comes directly from the training loss itself rather than being a separately invented metric. It answers one specific, narrow question: given text the model has never seen during training, how surprised was the model, on average, by what actually came next at each position? A model that consistently puts high probability on the actual next token is rarely surprised, and scores low. A model that's essentially guessing scores high.

Think of It Like This

Betting confidently on the right outcome, versus hedging blindly

Imagine two people betting on a series of coin-like outcomes, where one person has genuine insight into what's about to happen and bets confidently and correctly almost every time, while the other has no real insight and spreads their bets thinly across every possibility just to avoid ever being catastrophically wrong. The confident, correct bettor wins big and consistently. The hedging bettor never loses badly, but never wins convincingly either — their money is smeared across outcomes that mostly don't happen.

Perplexity measures which kind of bettor a language model is being, at every position in a piece of text it's shown for the first time. A model that puts strong, correct probability on the actual next token — confident and right — earns a low perplexity. A model whose predicted distribution is flat and uncommitted, or worse, confidently wrong, earns a high one.

How It Actually Works

From cross-entropy to perplexity, directly

Recall the next-token prediction loss: average cross-entropy across positions in a sequence. Perplexity is defined as simply the exponent of that same average:

\text{Perplexity} = \exp\left( \frac{1}{n} \sum_{t=1}^{n} -\log p_\theta(x_{t+1} \mid x_1, \ldots, x_t) \right)

This isn't a separately designed metric — it's the training loss itself, reshaped into a number with a more intuitive scale. Cross-entropy is measured in nats (or bits, with a different log base), which is an awkward unit to build intuition around. Exponentiating it converts that awkward log-scale quantity into something closer to "effective number of equally likely choices the model was uncertain between," which is far easier to reason about directly.

Reading the number

A perplexity of 1 means the model assigned probability 1 to every actual next token — perfect, zero-uncertainty prediction, which essentially never happens on real text. A perplexity of $V$ (the vocabulary size) means the model's predictions were, on average, no better than picking uniformly at random across the entire vocabulary — the worst realistic baseline. Real, well-trained models land somewhere between these extremes, and lower is better: it means the model's predicted distributions consistently concentrated probability mass on whatever token actually came next in unseen text.

What perplexity measures, and the gap it leaves open

Perplexity is fundamentally an intrinsic evaluation: it's computed purely from probabilities the model assigns to text, without reference to any downstream task or any notion of correctness beyond "did the model expect this." A model can achieve excellent perplexity on held-out text — meaning it's very good at predicting plausible continuations — while still producing factually wrong statements, because perplexity has no mechanism for checking truth against anything outside the text distribution itself. It measures calibration to a training-like distribution, not accuracy on any external standard, which is exactly why it's paired with separate, task-specific evaluations rather than used alone.

Show Me the Code

Computing perplexity directly from per-token log-probabilities, so the exponent-of-average-negative-log-prob relationship is visible rather than asserted.

1import numpy as np2
3
4def perplexity(log_probs_of_true_tokens: np.ndarray) -> float:5    """log_probs_of_true_tokens: the log-probability the model assigned to each actual next token."""6    avg_neg_log_prob = -log_probs_of_true_tokens.mean()7    return float(np.exp(avg_neg_log_prob))8
9
10confident_and_right = np.log(np.array([0.85, 0.90, 0.80, 0.88]))    # high probability, every time11confused = np.log(np.array([0.10, 0.05, 0.15, 0.08]))                # low probability, every time12
13print(round(perplexity(confident_and_right), 2))   # -> 1.18 — close to 1, barely surprised14print(round(perplexity(confused), 2))                # -> 11.29 — much higher, consistently surprised

The confident model's perplexity sits close to 1; the confused model's sits over ten times higher, purely because it assigned much lower probability to the tokens that actually appeared — no other information entered the calculation.

Watch Out For

Comparing perplexity across different tokenizers

Perplexity is computed per token, and different tokenizers segment the same text into different numbers of tokens — a model using a coarser tokenizer that produces fewer, larger tokens per sentence will tend to report a different perplexity than one using a finer tokenizer on identical text, independent of which model is actually "better" at predicting language. Comparing perplexity scores across models that use different tokenizers, without correcting for this, is a common and misleading mistake.

Treating low perplexity as evidence of factual accuracy

A model can be extremely well-calibrated — consistently assigning high probability to whatever token plausibly comes next — while being confidently wrong about facts, because perplexity only checks agreement with the statistical patterns of held-out text, never against any external ground truth. A model trained heavily on a corpus containing a widespread factual error can achieve excellent perplexity while faithfully reproducing that error. Perplexity answers "was the model surprised by this text," never "was this text true."

The Quick Version

Perplexity is the exponent of the average negative log-probability a model assigns to real, unseen next tokens.
It's derived directly from the cross-entropy training loss, reshaped into a more interpretable scale.
Lower is better: a perplexity near 1 means confident, correct predictions; a perplexity near the vocabulary size means essentially random guessing.
It's an intrinsic metric — computed purely from predicted probabilities, with no reference to task performance or factual correctness.
Comparing perplexity across models with different tokenizers is unreliable, since token segmentation itself affects the number.