Entropy

Written $H(p) = -\sum_i p_i \log p_i$, it answers a concrete question: if you had to encode outcomes from this distribution, how long would the average message be? A fair coin is 1 bit. A coin that lands heads 99% of the time is 0.08 bits, because you can say "heads again" cheaply and almost never need anything else. A distribution with all its mass on one outcome is 0 bits — there is nothing to transmit.

The base of the logarithm sets the unit. Base 2 gives bits, base $e$ gives nats, and deep learning libraries use nats throughout. A number without its unit is not comparable to anything.

Two uses dominate in practice. Entropy is the floor on cross-entropy loss: a model trained on genuinely ambiguous labels cannot drive the loss below $H(p)$, so a validation loss that stops falling at 0.3 may have converged rather than stalled. And the entropy of a model's own output distribution is a cheap uncertainty signal — high entropy means the model is spreading its probability mass, which is what active learning and abstention thresholds select on.