Math

Entropy

How surprised you should expect to be by the next outcome, measured as a number — highest when everything is equally likely, zero when you already know the answer.

Why Does This Exist?

Uncertainty is something you deal with constantly and rarely measure. "The model is unsure about this input" is a vague statement — until you notice that a prediction of $[0.51, 0.49]$ and one of $[0.99, 0.01]$ differ in a way you can put a number on.

Entropy is that number, and it is the foundation under a surprising amount of practical machinery:

Cross-entropy loss is entropy's direct descendant, which makes this the page underneath the loss function used by essentially every classifier.
Decision trees choose splits by information gain, which is the entropy you removed by splitting.
Perplexity, the standard number quoted for language models, is entropy exponentiated. A perplexity of 20 means the model is as uncertain as if it were choosing uniformly among 20 words.
Active learning picks the samples with the highest predictive entropy, because those are the ones the model has the least idea about and therefore the most to learn from.

The insight that makes all of it work is Shannon's: information and surprise are the same quantity. Learning something you already expected teaches you nothing. Learning something you thought was unlikely teaches you a lot.

Think of It Like This

Twenty questions, played optimally

Someone picks a card and you must identify it with yes-or-no questions. Played well, you halve the possibilities each time — is it a heart or a diamond, then is it high or low, and so on. Fifty-two cards take about six questions.

Entropy is the average number of questions you need. It is the cost of resolving your uncertainty, denominated in yes-or-no answers — which is exactly what a bit is.

Now change the game. Someone tells you the card is almost certainly the ace of spades, with a small chance of anything else. Your first question is obvious, and you are usually done in one. The uncertainty collapsed, so the entropy did too.

Push it further: if you are told with certainty which card it is, you need zero questions. Entropy zero. And the hardest possible version is the one where all fifty-two are equally likely — no question is better than any other, and you are stuck paying full price. Uniform is always the worst case, and that is the single most useful fact about entropy.

How It Actually Works

Start with the surprise of one outcome. Shannon's requirement was that surprise should shrink as probability grows, be zero for a certainty, and add when independent events combine. Exactly one function does all three:

\text{surprise}(x) = -\log P(x)

In plain words: the rarer the outcome, the bigger the number, and a certainty scores zero because $\log 1 = 0$ . The additivity is why it has to be a logarithm — independent probabilities multiply, and logs turn multiplication into addition, so two independent surprises sum.

Entropy is then the expected surprise, averaged over everything that might happen:

H(P) = -\sum_i P(x_i) \log P(x_i)

In plain words: how surprised you should expect to be, weighting each outcome's surprise by its own probability. Note the weighting carefully — a vanishingly rare outcome is enormously surprising, but it contributes almost nothing to the average because it almost never happens. The convention $0 \log 0 = 0$ handles the limit, and it is the right limit.

Bits or nats — pick one and say so

The log's base sets the unit, and both bases are in active use:

Base 2 gives bits. A fair coin is exactly 1 bit. Use this when reasoning about information or communicating results.
Base $e$ gives nats. Every ML framework uses this, because $\ln$ has the derivative $1/x$ and that keeps the gradients clean.

They differ by a constant factor, $1 \text{ bit} = \ln 2 \approx 0.693$ nats, so nothing structural changes — but quoting one when your audience assumes the other is a 44% error in the number. Frameworks default to nats silently, which is why a binary classifier's loss maxes out around 0.693 rather than 1.0.

Two properties that do all the work

Entropy is maximised by the uniform distribution. With $k$ equally likely outcomes, $H = \log k$ — the largest value any distribution over $k$ outcomes can reach. This is why "the model output was near-uniform" is a precise statement about maximum confusion, and why a language model's perplexity is compared against its vocabulary size.

Entropy is zero exactly when the outcome is certain. One probability at 1 and the rest at 0 gives $H = 0$ . It cannot go negative for a discrete distribution: every $P \le 1$ , so every $\log P \le 0$ , and the minus sign makes each term non-negative.

Between those bounds, entropy decreases as the distribution concentrates. That is the entire behaviour.

Worked example

A fair coin, $P = [0.5, 0.5]$ , in bits:

H = -(0.5 \log_2 0.5 + 0.5 \log_2 0.5) = -(0.5 \times -1 + 0.5 \times -1) = 1

In plain words: one yes-or-no question resolves it, exactly. This is the definition of a bit, and it is the maximum for two outcomes.

Now a biased coin, $P = [0.9, 0.1]$ :

H = -(0.9 \log_2 0.9 + 0.1 \log_2 0.1) = -(0.9 \times -0.152 + 0.1 \times -3.322) \approx 0.469

Under half a bit. Guessing "heads" is right 90% of the time, so most of the time you learn very little. Look at the two terms though: the rare tails outcome is 3.32 bits of surprise when it happens — over twenty times as surprising as heads — yet it contributes only 0.332 to the average because it is rare. Surprise is weighted by its own probability, and that weighting is the whole formula.

A uniform distribution over four classes gives $H = \log_2 4 = 2$ bits, the maximum for four outcomes. In nats the fair coin is $\ln 2 \approx 0.693$ — same uncertainty, different ruler.

Show Me the Code

1import numpy as np2
3
4def entropy(p: np.ndarray, base: float = 2.0) -> float:5    p = p[p > 0]  # 0*log(0) is defined as 0, so drop those terms entirely6    return float(-np.sum(p * np.log(p) / np.log(base)))7
8
9print(entropy(np.array([0.5, 0.5])))                    # -> 1.0    fair coin, bits10print(entropy(np.array([0.9, 0.1])))                    # -> 0.4689...11print(entropy(np.array([0.25, 0.25, 0.25, 0.25])))      # -> 2.0    uniform over 412print(entropy(np.array([1.0, 0.0])))                    # -> 0.0    certainty13print(entropy(np.array([0.5, 0.5]), base=np.e))         # -> 0.6931...  same, in nats

Every number matches the hand calculation. The filter on the first line is not a hack — it implements the $0 \log 0 = 0$ convention, and without it NumPy returns nan and poisons the sum.

Watch Out For

Letting a zero probability produce NaN

np.log(0) is -inf, and 0 * -inf is nan rather than 0. So the textbook formula transcribed literally returns nan the moment any outcome has probability exactly zero — which is common, because one-hot targets are mostly zeros.

A nan here is worse than a crash. It propagates through every subsequent operation, and if it reaches a gradient it silently converts every parameter it touches to nan. A model that was training fine produces nan loss from one step to the next, and the cause is several functions upstream.

Two correct fixes, and it matters which you use. Dropping the zero terms is mathematically exact, since their true contribution is zero. Clamping with np.log(p + 1e-12) is approximate and biases the result slightly, but it keeps the expression differentiable — which is why frameworks prefer it inside a loss. Never clamp by adding epsilon to a probability without renormalising if you also need the distribution to sum to 1.

Comparing entropies computed in different bases or over different alphabets

Two numbers that both say "entropy" are frequently not comparable.

The base problem is the easy one: 1.0 in bits and 1.0 in nats are different uncertainties, and the ratio is 0.693. Mixing them is a 44% error, and it happens whenever results move between a paper written in bits and code written in nats.

The alphabet problem is subtler and more damaging. Maximum entropy is $\log k$ , so it grows with the number of outcomes. An entropy of 3 bits is near-total confusion over 8 classes and near-certainty over 1,000. So a raw entropy says nothing about how confident a model is until you know $k$ — which is exactly why language model perplexity is reported alongside vocabulary size, and why comparing perplexities across models with different tokenizers is meaningless.

If you need a confidence number that compares across different numbers of classes, normalise by the maximum: $H / \log k$ lands in $[0, 1]$ regardless of $k$ . State which quantity you are reporting.

The Quick Version

Surprise is $-\log P(x)$ : rare outcomes are surprising, certainties carry zero information.
Entropy is the expected surprise, so rare outcomes are surprising but contribute little to the average.
It has to be a logarithm because independent surprises must add while probabilities multiply.
Base 2 gives bits, base $e$ gives nats. Frameworks use nats. $1 \text{ bit} = 0.693$ nats.
Maximum is the uniform distribution, at $\log k$ for $k$ outcomes. Minimum is 0 at certainty. Never negative.
Perplexity is exponentiated entropy: "as uncertain as choosing uniformly among this many options".
$0 \log 0 = 0$ , but code returns nan. Drop zero terms, or clamp if you need differentiability.
Entropy is not comparable across different bases or different numbers of classes. Normalise by $\log k$ if you need that.