Data Concepts

Train, Validation and Test Splits

Three sets, because you need somewhere to make decisions and somewhere that has never informed one. Look at the test set twice and you no longer have one.

Train fits parameters, validation makes every choice, and the sealed test block answers once, and the strip underneath cuts the same rows chronologically because a random cut lets the model read the future

Why Does This Exist?

You can't measure a model on the rows it learned from. Everyone knows that one, and it's why the first split exists.

The second exists for a reason people skip past: you also can't measure a model on the rows you used to choose it. Picking a learning rate, dropping a feature, moving a threshold, deciding when to stop — each is information flowing from a set of rows into the thing you ship.

So three roles. Train fits parameters. Validation chooses everything you tune: hyperparameters, features, thresholds, model family, early stopping. Test estimates the chosen thing — once, on rows that never informed a decision.

Think of It Like This

Past papers and the real exam

You revise from the textbook. That's your training data.

Then you sit past papers, mark them, and notice you keep dropping marks on one topic, so you go back and revise it. The papers didn't teach you the material. They told you what to do next, and they're used up the moment they do it.

The real exam happens once, on a paper you've never seen, and the grade means something because of that. Hand a student the real paper as practice and they'll score brilliantly, measuring how well they memorised one paper.

How It Actually Works

Typical shape: 60% train, 20% validation, 20% test. The numbers aren't sacred. What matters is that each set is big enough for its job, and that the third stays shut.

The one-test-set rule, as arithmetic

This gets taught as etiquette. It's measurement.

Take 1,000 test rows and a model at 90% accuracy. The standard error is $\sqrt{0.9 \times 0.1 / 1000} = 0.0095$ , so 0.95 percentage points. A 0.5-point "improvement" is half a standard error, indistinguishable from noise. Resolving it needs about 14,400 test rows.

Now peek. Try 20 variants and keep the best, and you've selected the maximum of 20 noisy draws — expected value about 1.87 standard errors, so the winner carries roughly 1.8 percentage points of pure luck. Enough peeks and test is as optimistic as validation.

Why a random split can be a lie

A random split assumes rows are exchangeable: any row is as likely to be a future row as any other. Three structures break that.

Time. Split at random and training rows sit either side of every validation row, so the model interpolates a period it should forecast. Cut chronologically. See time series data.
Groups. Multiple rows per patient, user, or device. A random split puts the same entity on both sides, so the model learns the entity rather than the pattern. Split by group.
Imbalance and rare strata. At 2% positives a random split can leave a class barely present, or absent. Stratify. See imbalanced data.

Grouped, time-ordered and imbalanced together is the normal case, not the exotic one.

The practical rules

Split first, before exploration and before any fitted transform. A scaler, imputer, or encoder each learns a statistic, and learning it from all the rows moves information across the boundary. See data leakage.

Record the seed, or you can't tell a better model from a friendlier split.

Make test look like production, not like train. Recent rows, real class balance, messy ones included.

Size it by the difference you care about, working backwards from that standard error. If you can't afford one that resolves your smallest interesting difference, say so instead of reporting three decimals.

Keep a temporal holdout as the final gate, even after cross-validating. The last unseen slice of calendar says whether the model survives recent change.

Worked example

1,000 rows at 2% positives, so 20 positives. A random 20% test split takes 200 rows: expected 4 positives, standard deviation 1.77, and roughly a 1-in-90 chance of none at all.

Draw ten random splits and the count lands on [5, 4, 4, 4, 4, 2, 3, 2, 3, 2]. At 2 positives, one classification moves recall by 50 points. At 5, by 20.

Stratify, and test holds exactly 4 every time. Still noisy, but not noisy in a way you'd mistake for a result.

Show Me the Code

1import numpy as np2
3y: np.ndarray = np.zeros(1000, dtype=int)4y[:20] = 1                                          # 1,000 rows, 20 positives, so 2%5
6def random_test_positives(seed: int) -> int:7    test = np.random.default_rng(seed).permutation(y.size)[:200]8    return int(y[test].sum())9
10def stratified_test_positives(seed: int) -> int:11    rng = np.random.default_rng(seed)12    pos = rng.permutation(np.flatnonzero(y == 1))[:4]     # 20% of the positives13    neg = rng.permutation(np.flatnonzero(y == 0))[:196]   # 20% of the negatives14    return int(y[np.concatenate([pos, neg])].sum())15
16draws = [random_test_positives(s) for s in range(10)]17print(draws)                           # -> [5, 4, 4, 4, 4, 2, 3, 2, 3, 2]18print(min(draws), max(draws))           # -> 2 519print(sorted({stratified_test_positives(s) for s in range(10)}))  # -> [4]

The seed is the only thing that changed between 5 positives and 2. Stratifying collapses that spread to one number.

Watch Out For

Tuning against the test set, including the slow version

The obvious form evaluates on test inside the loop. Rare, and a reviewer catches it.

The common form takes three months. A model ships, misses target, and gets retried against the same held-out quarter every week. Nobody wrote a loop and nobody thought they were tuning, but twenty attempts against one holdout is twenty peeks, and the winner clears the bar partly on the 1.8 points of luck the arithmetic predicts. It underperforms by about that much, and the post-mortem blames drift.

No code change catches this, so the fix is procedural. Make every decision on validation or on cross-validation folds, then touch test once.

A random split on grouped or time-ordered rows

train_test_split(X, y, random_state=42) on a table with 40 rows per patient, or one where every row carries a date.

Grouped: patient 1174 has rows in train and rows in test. The model memorises 1174, scores well on those rows, then has nothing for a patient it hasn't seen. Time-ordered: validation rows sit between training rows, so the model interpolates a period it should be extrapolating into.

The symptom is the same either way: validation far better than production, and the number moves a lot when you re-split by group or by date. Do that as a test. If the score drops 8 points, those 8 points were never yours.

The Quick Version

Train fits parameters, validation makes every choice, and test estimates the chosen thing once. Anything you tune is data entering the model through you.
Test reuse is measurement, not manners: 20 peeks at a 1,000-row test set buys about 1.8 points of optimism.
At 1,000 test rows and 90% accuracy the standard error is 0.95 points, so a 0.5-point win is unmeasurable; 14,400 rows would resolve it.
A random split assumes exchangeable rows. Time means a chronological cut, groups mean grouped splitting, imbalance means stratifying. Often all three.
Split before exploring and before fitting any transform, record the seed, make test resemble production, and keep a temporal holdout as the final gate.
1,000 rows at 2% positives: a random 20% test split gave 2 to 5 positives across ten seeds. Stratified gave 4 every time.
Validation far better than production, and a score that swings when you re-split by group or date, is the diagnosis.