Core ML

Naive Bayes

It treats every word as independent evidence, which is plainly false. The probabilities come out wrong, and the class that wins usually still wins correctly.

Each word contributes a likelihood per class and the class with the larger product wins, so a single word never seen in a class zeroes that class out entirely unless smoothing adds a pseudo-count

Why Does This Exist?

A support inbox takes 300 tickets a day, and for two years a human has routed each one to the billing queue or the outage queue. That's 40,000 labelled examples, and you'd like the routing done before anyone reads it.

Start from the exactly-correct answer. Bayes' theorem says the probability a ticket is billing, given its words, is proportional to two things multiplied: how often billing tickets turn up at all (the prior), and how likely those words are in one (the likelihood).

Now compute the second one. The ticket reads "invoice charged twice after the timeout last night", and that exact sequence has appeared zero times in your 40,000. So has every other message anyone will ever send you. The exact answer needs a count of something that never happened, and more data won't fix it, because the space of messages grows faster than any inbox.

That's the wall. The fix is one assumption: treat each word as independent evidence, given the class, so the whole message's probability is the product of each word's.

P(C \mid w_1, \ldots, w_n) \propto P(C) \prod_{i=1}^{n} P(w_i \mid C)

$C$ is the class, $w_i$ the $i$ -th word, $P(C)$ the prior, and each $P(w_i \mid C)$ a count you actually have — every quantity on the right is one pass over 40,000 tickets.

And the assumption is false. "Invoice" and "charged" turn up together constantly, so seeing both is nowhere near twice the evidence of seeing one. The model treats it as exactly twice.

Think of It Like This

Eight witnesses, one rumour

A panel has to decide which of two contractors damaged a wall. Twelve people give statements: eight name the first, four the second.

Eight to four sounds decisive. Then you learn all eight stood in the same room and heard it from the same site foreman. One observation, repeated eight times, counted as eight.

Two questions. Was the strength of the evidence real? No — their confidence is badly inflated. Did they pick the right contractor? Quite possibly, because the foreman did see something and all eight pushed the same way.

That's the independence assumption. It counts correlated evidence more than once, which wrecks the number and usually leaves the verdict standing.

How It Actually Works

The counts, and what each variant assumes

Training is counting. Per class, tally the tickets (the prior) and how often each vocabulary word appears (the likelihoods). One pass, no optimiser, no learning rate.

What you count depends on the variant, and each is a claim about a feature. Multinomial treats it as a count, so a ticket saying "refund" four times differs from one saying it once — the choice for text, and what most text representations hand you. Bernoulli treats it as present or absent, using absent words as evidence too, which suits subject lines. Gaussian fits a mean and variance per feature per class and reads the likelihood off a bell curve.

Wrong probabilities, right winner

Double-counting inflates the score of whichever class the correlated words favour. When "invoice" and "charged" both appear, billing gets both full boosts, so a ticket that deserved 0.7 comes out at 0.9999.

But you're taking an argmax. All that matters is whether billing beats outage, and the correlated evidence pushed the already-winning class further ahead rather than reordering the two. The ranking survives what the calibration doesn't.

Hence the split personality: a respectable classifier and a terrible probability estimator in one object.

Zeroes, pseudo-counts, and logs

A single zero destroys everything. "Timeout" never appeared in a billing ticket, so its likelihood is 0, so the whole billing product is 0 however strongly "invoice" and "charged" pointed there. One unseen word overrules the rest. Not reduces. Annihilates.

Laplace smoothing is the repair, as blunt as it sounds: add a pseudo-count of 1 to every word in every class before dividing. Now "timeout" in billing gets 1 out of 80 instead of 0 out of 74 — small, survivable, and honest that you can't rule it out from 40,000 examples. Use a smaller pseudo-count if you trust your counts more. The point is that it isn't zero.

Work in logs. Multiply a hundred numbers around 0.001 and you get ten to the minus three hundred, which flattens to exactly 0 in double precision and takes your comparison with it (numerical overflow covers that floor). Add logarithms instead.

Show Me the Code

Six words, two classes, one ticket with a word never seen in billing. Scored without a pseudo-count, then with one.

1import numpy as np2
3def log_score(counts: np.ndarray, doc: list[int], alpha: float) -> np.ndarray:4    total = counts.sum(axis=1, keepdims=True) + alpha * counts.shape[1]5    with np.errstate(divide="ignore"):               # alpha=0 gives log(0) on purpose, see below6        return np.log((counts + alpha) / total)[:, doc].sum(axis=1)7
8vocab = ["invoice", "refund", "charged", "crash", "timeout", "stacktrace"]9counts = np.array([[31.0, 24.0, 18.0, 1.0, 0.0, 0.0],    # word counts in billing tickets10                   [1.0, 2.0, 1.0, 27.0, 19.0, 22.0]])   # word counts in outage tickets11ticket = [0, 2, 4]                                       # "invoice", "charged", "timeout"12for alpha in (0.0, 1.0):13    s = log_score(counts, ticket, alpha)14    print(f"alpha={alpha:.0f}  billing {s[0]:8.2f}  outage {s[1]:8.2f}  "15          f"-> {['billing', 'outage'][int(np.argmax(s))]}")16# -> alpha=0  billing     -inf  outage    -9.89  -> outage17# -> alpha=1  billing    -6.74  outage    -8.69  -> billing

A ticket about an invoice being charged routes to outage, because one of its words was never seen in billing and minus infinity beats everything else. Add 1 to every count and billing wins by two log units.

Watch Out For

Using the output probability for anything but ranking

The model returns 0.9999 and someone builds on it: auto-route above 0.95, multiply by a cost, feed it to an ensemble as a feature. All three break quietly.

Double-counting pushes those numbers toward 0 and 1, and the effect grows with the number of correlated features, so a long ticket is more overconfident than a short one. The symptom is a confidence histogram piled at both ends, on a model whose accuracy is 88%. If you need the number, run probability calibration on held-out data or use logistic regression instead.

Building the vocabulary from training data and skipping the smoothing

This hides until deployment, because training and validation tickets share a vocabulary by construction: every word in a validation ticket was in the training set, so no likelihood is ever zero and no test catches it.

Then a customer writes "webhook" for the first time. Unsmoothed, that word has count zero in every class, so every class scores minus infinity and the argmax returns whichever index comes first — the ticket routes by array order. One new product name can misroute a morning's tickets. Smooth every class, and decide what happens to a word outside the vocabulary: dropping it is defensible, letting it zero the row is not a decision.

The Quick Version

The exact Bayes answer needs the probability of a whole message, never observed. The naive assumption replaces it with a product over single words.
The assumption is false and the model works anyway: double-counted evidence inflates the winning score without usually changing which class wins.
A decent classifier, a bad probability estimator. Use the argmax, not the number.
Multinomial for counts, Bernoulli for presence-or-absence, Gaussian for continuous features.
One unseen word gives a zero likelihood, and a single zero annihilates the product. Add a pseudo-count to every category, and add logs rather than multiplying probabilities.
Training is counting: fast, light on data, still a real text baseline.