Data Concepts

Cross-Validation Schemes

Cross-validation gives you a mean and a spread from the same rows. The spread is the half people ignore, and it's what tells you a 0.3-point win is noise.

Five folds validate the same rows five times as the block slides along, and the time-series strip underneath never trains after the block it validates

Why Does This Exist?

You held out 20% for validation, scored 0.847, and now you're choosing between models that differ by 0.3 points. How much would 0.847 move on a different 20%? No idea.

Cross-validation answers that. Rotate the validation block through the data, fit K times, and you get K scores from the same rows: a mean and a spread.

The spread is the half everyone drops from the report, and the more useful half. At a fold standard deviation of 2 points, a 5-fold mean has standard error $2/\sqrt{5} = 0.89$ , so a 0.3-point difference is a third of a standard error. It means nothing.

Scope note: this page is about partitioning the data honestly. Picking a model with those partitions is another subject.

Think of It Like This

Five judges, or one

You want to know how good a dish is, so one person tastes it. They say 7 out of 10.

Ask five and you learn something the single taster couldn't. Say 7, 7, 8, 7, 7 and you have a solid dish. Say 3, 9, 8, 4, 9 and the mean is still about 7 while describing nothing — it lands for some palates and fails badly for others, and which you heard first was luck.

Two dishes averaging 6.8 and 7.1 aren't ranked. They're tied, and the spread is what tells you so.

How It Actually Works

Each rung fixes something the one below it gets wrong. Climb until your data stops breaking the assumption.

The ladder

K-fold. K parts; train on K−1, validate on the one left out, rotate, so every row is validated once. Small K is pessimistic because each model sees less data; large K has low bias but the folds correlate. Five or ten.
Stratified K-fold. Each fold at the class proportions of the whole, the default for classification. At 2% positives a plain fold can arrive with almost no positives, and that score is noise the mean absorbs. See imbalanced data.
Grouped K-fold. No entity split across folds — patient, user, device, session. Split one and the model memorises it in training and recognises it in validation, so you measured recall of memory. Stratified-grouped does both, approximately.
Leave-one-out and repeats. K = n gives the lowest bias and a noisy estimate: each fold scores one row, and the n models are nearly identical so their errors correlate. Repeated K-fold reruns 5-fold under several shuffles and pools the scores.
Time-series split. Never shuffled, training always earlier than validation. An expanding window grows from the start of the series; a rolling window keeps training length fixed and slides forward. See time series data.
Purged and embargoed. For labels that span time. If the label is "did the price rise over the next 5 days", Monday's row and Wednesday's share three days of outcome, so rows either side of a boundary correlate. Purging drops training rows whose label window overlaps validation; an embargo drops a further gap.
Nested. Outer loop estimates, inner loop tunes on the outer loop's training rows only, because one set of folds can't do both honestly. Five outer, five inner, a 20-point grid: 505 fits.

Every fitted step refits inside the fold

This silently invalidates more results than any wrong choice of K.

A scaler learns a mean, an imputer a median, an encoder a category list, feature selection which columns survive, and target encoding an average of the label. All fitted state, all refitting inside every fold on that fold's training rows only. Pipeline plus cross_val_score does that; a hand-rolled loop usually doesn't, because the tempting place for preprocessing is above the loop.

Worked example

Twenty rows, indices 0 to 19, five folds. Unshuffled, the validation blocks are [0,1,2,3], [4,5,6,7], [8,9,10,11], [12,13,14,15], [16,17,18,19]. Now make those rows five users with four each: user 0 owns 0 to 3, user 1 owns 4 to 7. The folds land one user apiece, because the table was sorted that way. Luck.

Shuffle and it falls apart. Fold one becomes [2, 4, 6, 19] — users 0, 1 and 4 — and the user counts across folds run [3, 4, 3, 2, 3]. User 1's rows 4 and 6 validate while 5 and 7 train.

Show Me the Code

1import numpy as np2
3n, k = 20, 54groups: np.ndarray = np.repeat(np.arange(5), 4)   # 5 users, 4 consecutive rows each5
6def folds(seed: int | None) -> list[np.ndarray]:7    idx = np.arange(n) if seed is None else np.random.default_rng(seed).permutation(n)8    return np.array_split(idx, k)                 # 20 rows / 5 folds = 4 rows each9
10def users_per_fold(parts: list[np.ndarray]) -> list[int]:11    return [int(np.unique(groups[p]).size) for p in parts]12
13ordered, shuffled = folds(None), folds(0)14print([sorted(p.tolist()) for p in ordered[:2]])  # -> [[0, 1, 2, 3], [4, 5, 6, 7]]15print(users_per_fold(ordered))                    # -> [1, 1, 1, 1, 1]16print(sorted(shuffled[0].tolist()))               # -> [2, 4, 6, 19]17print(users_per_fold(shuffled))                   # -> [3, 4, 3, 2, 3]

One user per fold before the shuffle, three or four after. shuffle=True is the default people reach for and exactly what breaks grouped data.

Watch Out For

Shuffling rows that came in an order

KFold(n_splits=5, shuffle=True, random_state=42) on a table sorted by date. The snippet everyone has memorised.

Fold three now trains on March and December and validates on June. The model interpolates: it knows what happened either side of the period it's scoring, which production never does. No amount of tuning reproduces that number later.

What keeps it around is that it looks stable. All five folds agree because all five cheat the same way, and a tight spread reads like a trustworthy result. Use a time-series split and compare: the gap is the size of the lie. See data leakage.

Preprocessing above the loop, cross-validating below it

Scale on line 20. Select the top 30 features by correlation with the target on line 25. Call cross_val_score on line 40.

Every fold's validation rows contributed to that mean, that standard deviation, and that feature ranking. Selection is the worst, because it looked at the target across all rows: with enough candidates, some pure-noise features rank well on the full data and then score well on every fold.

The failure mode is uniform: all K scores are optimistic together, so the usual signal of folds disagreeing is absent. Put every fitted step in a Pipeline and keep a test set out of the loop.

The Quick Version

One validation split gives one noisy number. Cross-validation gives a mean and a spread, and the spread is the half that matters: at a fold standard deviation of 2 points, a 5-fold mean has standard error 0.89.
K-fold validates every row once. K=5 or 10; small K is pessimistic, large K correlates the folds. Leave-one-out is the K=n limit: low bias, noisy estimate.
Stratified is the default for classification; grouped is mandatory when rows share an entity.
Time-series split never shuffles: training sits earlier than validation, expanding or rolling.
Purged and embargoed splits handle labels that span time, where adjacent rows share outcome.
Nested CV tunes inside and estimates outside. Five by five on a 20-point grid is 505 fits.
Every fitted step refits inside the fold: scaler, imputer, encoder, selector, target encoder. That's Pipeline.
Twenty rows, five users: one user per fold unshuffled, three or four after shuffle=True.