Core ML

Softmax Regression

One score per class, exponentiated then divided by their total. The classes now compete for a fixed budget of one, so only the gaps between them matter.

Exponentiating four class scores and dividing by their total keeps the ranking and the rough shape but turns them into shares that add to exactly one, so the classes compete for a fixed budget

Why Does This Exist?

Every incoming support ticket goes to one of four queues: billing, bug, feature request, account access. One ticket, one queue, and you want four numbers saying how likely each is.

That's the case for this page, and it's the shape of nearly every classifier you'll ever ship. The last layer of a neural-network classifier is exactly this. ImageNet's thousand classes, a language model's next-token distribution over 100,000 tokens: same arithmetic, just wider.

You know the two-class version. Logistic regression computes one score, squashes it through a sigmoid, and reads the result as the probability of "yes". The obvious move for four queues is four separate logistic models. Run it and the answers don't add up, literally: 0.7, 0.6, 0.2, 0.4 sums to 1.9, with no rule for turning that into a decision. Nothing told the four models they were rivals.

Softmax is the fix. Give each class its own score, exponentiate so every number is positive, divide by the total. Now they sum to one, and pushing one up pushes the others down. Those raw pre-softmax scores are logits, a word you'll meet in every library doc.

Think of It Like This

One bag of flour, four loaves

Four bakers, one bag of flour, and the whole bag gets used. Give the bug queue more and something else gets less. There's no way to be generous twice.

Each baker's share depends on how loudly they argue, on a scale with no ceiling. Here's what makes softmax softmax: the arguments get amplified before the split, so a two-point lead doesn't buy slightly more flour, it buys a lot more. Score 4 against score 2 becomes a 7-to-1 share, not 2-to-1. And if every baker gets five points louder the split is identical, because the relative arguments haven't moved. Only differences reach the flour.

How It Actually Works

Score every class, then take shares

One weight vector per class, so a ticket $x$ with $K = 4$ queues gets $K$ logits:

z_k = w_k^\top x + b_k

$w_k$ is the weight vector for class $k$ and $b_k$ its intercept. Each $z_k$ is unbounded, and they aren't comparable yet. Softmax makes them comparable:

p_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}

The numerator is positive whatever $z_k$ was. The denominator is shared, so the ranking survives untouched and the four values now sum to exactly 1. With $K = 2$ the whole thing collapses back to the sigmoid, since $p_1 = \sigma(z_1 - z_2)$ — logistic regression was always the two-class case.

Shifts are free, and that's not a curiosity

Subtract any constant $c$ from all four logits and the answer doesn't move:

\frac{e^{z_k - c}}{\sum_j e^{z_j - c}} = \frac{e^{-c}\,e^{z_k}}{e^{-c}\sum_j e^{z_j}} = p_k

The $e^{-c}$ cancels top and bottom. Softmax reads only the gaps between logits, never their level.

Every serious implementation cashes that in. Double-precision exp overflows to infinity north of an input of 709, and a logit of 800 isn't exotic in an unregularised model or a network mid-training. Feed it in directly and you get inf over inf, so nan out, poisoning every gradient downstream. Subtract the largest logit first and the biggest input to exp is 0, while shift-invariance guarantees the answer is unchanged. That one subtraction is the whole of what people mean by a "numerically stable softmax".

One spare degree of freedom, and a gradient you've seen

Because only gaps matter, adding the same vector to all four weight rows leaves predictions bit-for-bit identical. The parameters are over-determined: $K$ rows describe a function $K - 1$ rows already pin down. Nothing breaks, but the coefficients aren't unique, so "the weight on billing" means nothing on its own. Some libraries remove the slack by fixing one class's row at zero, making every remaining coefficient a comparison against that reference class.

The loss is categorical cross-entropy, $L = -\log p_{y}$ with $y$ the true queue. Only the true class's probability enters, because the one-hot label zeroes out the rest. The gradient is the one from the two-class page, now per class:

\frac{\partial L}{\partial z_k} = p_k - y_k

Predicted minus actual. Chain it through the linear layer and the weight gradient is $(p_k - y_k)\,x$ . That's why the two always ship together: the $e^{z}$ derivatives cancel against the $\log$ and you're left with subtraction.

Show Me the Code

Four logits for one ticket, the stable version, then the naive version failing.

1import numpy as np2
3def softmax(z: np.ndarray) -> np.ndarray:4    z = z - z.max()  # shift-invariant, so this is free and it stops exp from overflowing5    e = np.exp(z)6    return e / e.sum()7
8scores = np.array([2.0, 4.0, 1.0, 3.0])  # billing, bug, feature, access9print(np.round(softmax(scores), 3))10print(np.round(softmax(scores + 800.0), 3))  # same gaps, so the same answer11with np.errstate(over="ignore", invalid="ignore"):12    raw = np.exp(scores + 800.0)  # what the textbook formula does, unshifted13    print(raw / raw.sum())14# -> [0.087 0.644 0.032 0.237]15# -> [0.087 0.644 0.032 0.237]16# -> [nan nan nan nan]

A one-point lead, 4 against 3, becomes 0.644 against 0.237, nearly three to one. Shift all four by 800 and the stable version doesn't blink; the unshifted one returns four nans.

Watch Out For

Exponentiating raw logits straight into nan

Training runs fine for a while, then the loss prints nan and every metric flatlines. The usual cause is a hand-rolled softmax hit by a logit large enough to overflow exp, and it doesn't take a bug to get there: remove weight decay, let the learning rate run hot, or feed an unscaled feature, and logits in the high hundreds turn up on their own. One bad row in a batch is enough, so it looks intermittent and people chase the data instead of the arithmetic.

Subtract the max, or better, never write the softmax at all. Every framework ships a fused loss that takes logits and does the stable computation internally: PyTorch's CrossEntropyLoss, TensorFlow's from_logits=True. The mirror-image mistake is quieter — applying your own softmax, then handing the probabilities to one of those fused losses. Nothing errors, gradients shrink toward nothing, and the model plateaus early with no clue why.

Using softmax where labels can co-occur

A ticket says the app crashed and the customer was double-charged. Both queues are correct, and softmax cannot say so, because the shares must sum to one: the only way to raise billing is to take probability away from bug. In the metrics that reads as decent top-1 accuracy beside terrible recall on second labels.

The problem isn't the model, it's the question. "Which one of these?" is softmax. "Which of these apply?" is $K$ independent sigmoids with binary cross-entropy on each and a threshold per label, so two labels can both sit at 0.9. Tagging and content moderation are both the second kind, and the bug survives review because a softmax model on multi-label data still returns a plausible-looking answer.

The Quick Version

One logit per class, exponentiate, divide by the total. Positive, sums to one, ranking preserved.
Logits are the raw pre-softmax scores, and library APIs care which of the two you hand them.
Softmax reads only the gaps between logits, which is why you subtract the max before exponentiating.
Skip that and large logits overflow to inf, then nan, then every weight downstream.
One redundant degree of freedom, so a class's coefficients need a reference class to mean anything.
Paired with cross-entropy the gradient is $p_k - y_k$ , predicted minus actual, per class.
Mutually exclusive labels want softmax. Co-occurring labels want independent sigmoids.