Data Concepts

Class Imbalance

At 0.3% fraud, a model that always says no scores 99.7% accurate and is worth nothing. Most imbalance problems are metric problems, so fix the metric first.

Two classes at a hundred to one overlap on a single score axis, and sliding the decision threshold is what changes how many frauds you catch and how many false alarms you pay for

Why Does This Exist?

Two million card transactions, 6,000 of them fraudulent. That's 0.3%.

Train anything on that and the shortest path to a low loss is predicting "not fraud" every time. Accuracy 99.7%, better than most published numbers, and it catches zero fraud.

Here's the position this page takes, against how the topic usually gets taught: most imbalance problems are metric problems, and the first fix is the metric and the threshold, not the data. Resampling is fourth on the list below, not first. It invents rows, and inventing rows belongs after the free options.

Think of It Like This

A smoke alarm nobody has adjusted

A smoke alarm that never sounds is correct on 364 days out of 365. Score it on accuracy and it's excellent. Score it on the day that mattered and it's a plastic box.

Now turn the sensitivity dial. Wind it up and burnt toast sets it off every Tuesday, annoying enough that somebody unplugs it. Wind it down and you're back to the plastic box.

No setting is right in the abstract. One becomes right once you've decided what a false alarm costs against what a missed fire costs. That decision is yours, not the model's, and no amount of extra training data makes it for you.

How It Actually Works

The order of preference

Fix the metric. Accuracy is meaningless at a 0.3% base rate. Use precision, recall, F1, and precision-recall AUC rather than ROC AUC when positives are rare. Why: ROC AUC is built on the false-positive rate, whose denominator is the enormous negative class, so 330 extra false alarms against 9,900 negatives barely move it while precision drops through the floor. Precision-recall AUC puts the rare class in both denominators. Report per-class recall too.
Move the threshold. A probabilistic model hands you a dial, and 0.5 was never a considered choice — it's the midpoint of the output range, nothing more. Sweep it on validation, plot precision against recall, pick the point matching the cost you carry. Free, and usually the largest win available.
Weight the classes. class_weight="balanced", or a cost-sensitive loss with the real currency in it. The objective gets reweighted so a missed positive hurts more. No rows invented, no data thrown away.
Resample. Undersample the majority: cheap, discards data, fine when the majority is huge. Oversample the minority by duplication: no new information, and it overfits those exact rows. Or SMOTE and its variants, interpolating new minority points between near neighbours, so they invent rows that can land in the wrong region and they need scaled features because neighbours are defined by distance.
Focal loss, for the deep-learning case. It down-weights the negatives the model already gets right, so the gradient keeps coming from the hard rows.

Two structural rules people break

Resample the training fold only. Never validation, never test. A resampled validation set no longer carries the production base rate, so precision measured on it describes a dataset you invented. Recall survives; precision doesn't, because its denominator moved.

Recalibrate if you need probabilities. Resampling and class weighting both shift predicted scores upward, which is the point of them. If something downstream multiplies your output by a transaction value, that shift is a systematic error. Calibrate on untouched, real-base-rate data.

When imbalance isn't your problem

A 1:1000 ratio with 50,000 positive rows is usually fine. Plenty of minority signal in absolute terms, and a reasonably separable boundary gets found. Imbalance bites on a small absolute minority count, not on a large ratio.

At the far end, a few dozen positives among millions, stop treating it as classification. Anomaly detection frames it as learn normal, flag the distance, and needs no positive labels.

Worked example

10,000 validation rows, 100 of them positive, so a 1% base rate.

At threshold 0.5 the model predicts positive zero times. TP 0, FP 0, FN 100, TN 9,900. Accuracy $9900/10000 = 0.99$ , recall $0/100 = 0$ , precision undefined because you never made a positive call.

Drop the threshold to 0.08 and it flags 400 rows, 70 of them real. TP 70, FP 330, FN 30, TN 9,570. Recall $70/100 = 0.70$ , precision $70/400 = 0.175$ , F1 $2(0.175 \times 0.70)/0.875 = 0.28$ , accuracy $9640/10000 = 0.964$ .

Accuracy went down, 0.99 to 0.964, while the model went from worthless to catching 70% of the fraud. The false-positive rate at that point is $330/9900 = 0.033$ , tiny, so ROC AUC hardly notices. Meanwhile 330 of your 400 alerts are wrong, which is what the fraud team experiences.

Show Me the Code

1import numpy as np2
3def report(tp: int, fp: int, fn: int, tn: int) -> tuple[float, ...]:  # acc, prec, rec, fpr4    m: np.ndarray = np.array([tp, fp, fn, tn], dtype=float)5    acc: float = float((m[0] + m[3]) / m.sum())6    prec: float = float(m[0] / (m[0] + m[1])) if m[0] + m[1] else float("nan")7    rec: float = float(m[0] / (m[0] + m[2]))8    fpr: float = float(m[1] / (m[1] + m[3]))9    return round(acc, 4), round(prec, 4), round(rec, 4), round(fpr, 4)10
11print(report(0, 0, 100, 9_900))     # -> (0.99, nan, 0.0, 0.0)      threshold 0.5012print(report(70, 330, 30, 9_570))   # -> (0.964, 0.175, 0.7, 0.0333) threshold 0.08

Same model, two thresholds. Accuracy falls, fpr barely moves, and the only figures that moved usefully are precision and recall.

Watch Out For

Running SMOTE before the split, or before the fold

SMOTE builds a synthetic minority row by taking a real one and stepping partway toward a neighbour. Do that on the whole frame, then split, and a point interpolated from row 412 lands in validation while row 412 sits in training, a few percent away in feature space.

Your model now gets scored on near-copies of rows it trained on. Recall comes back at 0.95, and production recall is 0.31. It's data leakage with a resampling library in front of it, and cross-validation won't catch it, because the contamination happened before the loop began.

SMOTE is a transform with fitted state. It belongs inside the Pipeline, on the training fold only.

Shipping on ROC AUC at a 1% base rate

ROC AUC 0.94 reads well in a review, so the model ships. The fraud team then gets 400 alerts a day and 330 are nothing.

The arithmetic explains it. The false-positive rate divides by 9,900 negatives, so hundreds of wrong alerts move it by 0.03. Precision divides by the 400 rows you flagged, so the same hundreds put it at 0.175. Both describe the model honestly; only one describes the queue somebody works through.

Report precision and recall at your operating threshold, plus precision-recall AUC for a threshold-free summary. Keep ROC AUC for roughly balanced problems, where its denominators are comparable.

The Quick Version

Imbalance is a metric problem before it's a data problem. Fix the metric and the threshold first.
Accuracy at a 0.3% base rate is 99.7% for a model that predicts nothing. Never report it alone.
ROC AUC divides false positives by the huge negative class, so it barely moves while precision collapses. Prefer precision-recall AUC when positives are rare.
0.5 is a default, not a decision. Sweep the threshold on validation against real costs.
Class weights and cost-sensitive losses reweight the objective without inventing rows. Try them before resampling.
Undersampling discards data, duplication overfits, and SMOTE invents rows that can land in the wrong region and needs scaled features.
Resample the training fold only. A resampled validation set has the wrong base rate, so its precision means nothing.
Resampling and weighting both shift predicted scores. Recalibrate on untouched data if anything downstream reads them as probabilities.
A large ratio with many minority rows is fine. A few dozen positives is anomaly detection, not classification.