Data Leakage

Why Does This Exist?

A hospital readmission model, AUC 0.97 on held-out patients, signed off, deployed. In production it barely clears the base rate. Nothing in validation flagged it, because what was wrong is what made validation look good.

The whole page fits in one sentence. Leakage is the model having access at training time to information it will not have at prediction time. Every named variety is a different route across that line. The taxonomy below is plumbing; the definition is the idea.

Which is why it's the nastiest bug class in applied work. A crash announces itself. A weak feature shows up as a flat score. Leakage pays you a bonus.

Think of It Like This

How It Actually Works

Take a feature. Fix the instant you'd have to predict. Ask whether its value was computable from data existing at that instant. If not, you have leakage, and the name depends only on how the value got there.

Six routes across the line

• Target leakage. A feature computed from the outcome, or a proxy for it. discharge_notes_length predicts readmission because sicker patients get longer notes, written after the fact. total_delivery_duration for a lateness model is the label in other units. account_closed_date in a churn table is the label with a timestamp. They pass review because the names sound reasonable.
• Temporal leakage. Any aggregate or window including events after the prediction time: mean_order_value over a customer's whole history, a courier's late rate computed including the delivery you're scoring. The structural version is a random split on time-ordered rows, so the model trains on next week and gets tested on last week.
• Group leakage. The same patient, user or device on both sides of the split. The model memorises the entity rather than the pattern, then scores well on a test set full of entities it met.
• Preprocessing leakage. Any fitted transform fitted before the split, or outside the fold: a scaler that saw the test set's mean, an imputer whose median came from every row, a target encoder, a feature selector, a Box-Cox exponent, a quantile mapping. Every one is a parameter estimated from held-out data. Most common route by a wide margin.
• Duplicate leakage. The same row, or a near-identical one, in train and test, so your test score becomes a memorisation check. It's why deduplication is a validation concern, not tidiness.
• Train-serving skew. The production twin. Nothing leaked into training; serving computes a feature differently, so the model meets a distribution it never trained on. See online/offline skew.

The diagnosis protocol

This order, because the cheap checks catch most of it.

1. Treat a suspiciously high score as a symptom, not a success. Beating the published field on a first attempt with library defaults means a bug.
2. Read feature importance. One column carrying nearly all of it is the signature.
3. Re-split by time, then by group. A collapse localises the problem in one move.
4. Ask of every column: was this value knowable at the prediction instant? Ask with whoever owns the source table.
5. Look for exact and near-duplicate rows across the split boundary.
6. Confirm every fitted step sits inside a Pipeline, so it refits per split.

Not one of those is a metric, and that's the point. Leakage isn't detectable from the validation number, because it's the class of bug that makes that number better.

Worked example

A delivery lateness model. 1,000 orders, 380 late, where late means over 45 minutes door to door. Guessing the majority class gets 620 right, so 0.62. Train on distance, hour, courier and prep time: 720 correct, accuracy 0.72. Roughly what the problem allows.

Add total_delivery_duration and accuracy jumps to 0.99, so 990 of 1,000. One split at 45 minutes reproduces the label; the ten misses are orders the app flagged a minute either side of the logged duration.

Now backtest it: train on January through April, score May, features computed as of the moment each order was placed. That last clause is what a temporal split forces on you, which is why re-splitting is a test rather than a reshuffle. The duration isn't knowable at order time, so it arrives empty, the trees fall back on distance and hour, and accuracy lands at 0.71. That 0.28 gap was the label renamed.

Show Me the Code

1import numpy as np2
3minutes: np.ndarray = np.array([31, 52, 44, 68, 22, 47, 39, 55, 41, 60])          # logged AFTER delivery4distance: np.ndarray = np.array([2.1, 8.4, 3.0, 9.9, 1.2, 4.4, 6.2, 7.1, 3.3, 8.8])  # known when ordered5is_late: np.ndarray = (minutes > 45).astype(int)          # the label is DEFINED from minutes6knowable: dict[str, bool] = {"minutes": False, "distance": True}7
8def threshold_accuracy(x: np.ndarray, cut: float) -> float:9    return float(((x > cut).astype(int) == is_late).mean())10
11print(is_late.tolist())                     # -> [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]12print(threshold_accuracy(minutes, 45.0))    # -> 1.0   one split rebuilds the label exactly13print(threshold_accuracy(distance, 5.0))    # -> 0.8   the honest column, and it is weaker14for name, ok in knowable.items():15    if not ok:16        print(f"drop {name}")               # -> drop minutes

The last three lines are the only check that works, and it isn't a metric. It's a question asked of a column name.

Watch Out For

The Quick Version

• One definition covers every case: at training time the model holds information it won't have when it predicts.
• Target leakage is a feature computed from the outcome or a proxy for it, under a sensible business name.
• Temporal leakage is any window reaching past the prediction moment, plus a random split on time-ordered rows.
• Group leakage puts the same entity on both sides, so memorising looks like generalising.
• Preprocessing leakage is a fitted transform fitted before the split or outside the fold. Most frequent by far.
• Duplicates across the split make the test set a memorisation check; skew is the same failure in production.
• Diagnose in order: distrust the score, read importances, re-split by time and group, audit knowability, hunt duplicates, check the Pipeline.
• No metric detects leakage, because leakage improves the metric.

What to Read Next

• Train, Validation and Test Splits is the boundary all of this is defined against.
• Cross-Validation Schemes has the grouped and time-series folds that catch two routes.
• Target Encoding is the sharpest single instance, worked end to end.
• Feature Construction is where as-of computation stops temporal leakage.
• Online/Offline Skew is the production-side twin.
• Deduplication at Scale finds near-duplicates straddling a split.
• Definitions worth a look: Data Leakage, Train-Test Split, and K-Fold Cross-Validation.