Threshold Selection

A score is not a decision. The cutoff that minimizes cost comes from the prices of the two mistakes, and 0.5 is just the default the library shipped.

Two overlapping score distributions share one axis, so wherever you put the cut you are choosing how many real positives to miss against how many clean cases to review

Why Does This Exist?

A card-fraud model scores each of the day's 200,000 transactions between 0 and 1. Somewhere a line of code turns that into a yes or a no, and in almost every codebase it reads score above 0.5. Nobody chose 0.5. It arrived as a library default and survived because it looked neutral.

It is not neutral. Move the line and two things trade against each other. A false positive is a clean transaction you flag: review labor plus a text the customer did not need, call it 2 dollars. A false negative is fraud you wave through, an average chargeback of 180 dollars. Ninety times apart in price, and 0.5 treats them as identical. Same weights, same ranking quality, yet a different line moves both the fraud team's day and the loss number.

One assumption throughout: the score means something. If 0.70 is not a 70 percent chance, the arithmetic below computes with something that is not a probability, and probability calibration is what fixes that first.

Think of It Like This

The sensitivity dial inside a smoke alarm

Open a smoke alarm and there is a sensitivity setting. Crank it up and the thing screams at burnt toast, so within a week somebody pulls the battery and you have no alarm at all. Crank it down and a real fire gets a long head start before anyone knows.

No single setting is correct, and that is the point, because it depends on the room. In a data center a fire costs millions and a nuisance alarm costs a technician a walk down the hall, so you set it twitchy. Next to a bedroom, false alarms destroy trust in the alarm, so you back it off and accept a slower response.

Same sensor, same physics, opposite settings, because the two mistakes cost different amounts in the two rooms. That is exactly the choice score above 0.5 makes for you, without asking which room you are in.

How It Actually Works

The arithmetic behind the cut

Call a false positive cost $C_{FP}$ , the 2 dollars of review, and a false negative cost $C_{FN}$ , the 180-dollar chargeback. For a transaction scored at probability $p$ you have two options, each with an expected cost.

Flag it: you are wrong with probability $1 - p$ , so the cost is $(1-p) \, C_{FP}$ . Wave it through: you are wrong with probability $p$ , so the cost is $p \, C_{FN}$ . Flag whenever flagging is cheaper, and rearranging that comparison leaves one number:

t^{*} = \frac{C_{FP}}{C_{FP} + C_{FN}}

$t^{*}$ minimizes expected cost, and notice what is missing from it: nothing about the model, nothing about the data, nothing about how rare fraud is. Just the ratio of the two prices. For 2 against 180 that is $2/182$ , which is about 0.011. Any problem where one mistake costs far more than the other lands nowhere near the middle.

The framings production actually uses

Expected cost is the honest framing when you can price both mistakes. Often you cannot, and three alternatives each earn their place.

Maximize F-beta. It blends precision, meaning of what you flagged how much was fraud, with recall, meaning of the fraud how much you caught. The parameter $\beta$ sets how much more recall counts. Reach for it when recall clearly matters more and you cannot name a price. The single-number summaries live in classification metrics.

Hold a precision floor. Reviewers disengage below 30 percent precision is a real constraint, often the realest on the table. Take the lowest threshold that clears it, and read that off the precision recall curve.

Fix an alert budget. The team reviews 200 a day, so send the top 200. That is a rank cut, not a probability cut, and it is what most deployed systems do, because capacity is the constraint that does not move.

Where a threshold goes stale

Choose it on validation data, then leave it alone. It is a decision fitted to data like any other.

Then watch prevalence. Precision depends on the base rate as much as on the model: hold the threshold fixed, halve the fraction of fraudulent transactions, and precision roughly halves too. Reviewers see twice the noise and nothing in the code changed.

That drift is the strongest practical argument for the alert budget. A rank cut says the worst 200 today and normalizes itself, because when scores drift up or down together, the top 200 is still the top 200. A probability cut says everything above 0.011, and that moves with the score distribution.

Code

Ten scored transactions, two cuts, and the cheaper line is the one the ratio predicts.

1COST_FP, COST_FN = 2.0, 180.0  # review labor vs an average chargeback2
3
4def cost_at(scores: list[float], labels: list[int], t: float) -> float:5    total = 0.06    for p, y in zip(scores, labels):7        flag = p >= t8        if flag and y == 0:9            total += COST_FP10        if not flag and y == 1:11            total += COST_FN12    return total13
14
15scores = [0.005, 0.008, 0.02, 0.05, 0.3, 0.7, 0.002, 0.9, 0.04, 0.6]16labels = [0, 1, 1, 0, 1, 1, 0, 1, 0, 0]17t_star = COST_FP / (COST_FP + COST_FN)18
19print(f"{t_star:.4f}")20# -> 0.011021print(f"{cost_at(scores, labels, 0.5):.0f}")22# -> 54223print(f"{cost_at(scores, labels, round(t_star, 4)):.0f}")24# -> 186

The minimum sits at 0.0110, which is $C_{FP}/(C_{FP}+C_{FN})$ to four places, and it costs 186 against 542 for the default. Trace the two lines by hand: at 0.5 the three missed frauds at 0.008, 0.02 and 0.3 cost 540 plus one false alarm, while at 0.011 only the 0.008 fraud slips through plus three cheap false alarms.

Watch Out For

Tuning the threshold on the test set

Symptom: you sweep 200 thresholds, pick the best F2, and report that F2 as the model's score. It reads as a measurement, but it is the maximum of 200 noisy numbers, and a maximum over noise sits high by construction. The number looks strong and the launch disappoints.

Fix: choosing a threshold is model selection, so it belongs on validation data. Pick the cut there, freeze it, then evaluate that frozen cut once on test. The plumbing for the split lives in train, validation and test splits.

Shipping a threshold and never revisiting it

Symptom: the cut was right in spring, and by autumn recall is down eight points with precision down six. No deploy happened and no test failed, because every gate in the pipeline compares the model to itself while the world moved around it.

Fix: track flagged volume and the precision of the flagged set on a rolling window, and alert when either leaves its band, since volume moves first. Then refit the cut on recent labels on a schedule instead of after a complaint. A rank-based budget absorbs much of that drift for free.

The Quick Version

0.5 is a library default, not a decision, because it assumes both mistakes cost the same.
With a calibrated score and two prices, the expected-cost-minimizing cut is $C_{FP}/(C_{FP}+C_{FN})$ , and nothing about the model appears in it. For 2 against 180 that is 0.011.
When prices are unknown, maximize F-beta, hold a precision floor, or fix an alert budget and take the top N, which is what most deployed systems use.
Pick the cut on validation data, since tuning it on test is model selection in disguise.
Precision moves with prevalence at a fixed threshold, so a probability cut decays where a rank cut holds.