Core ML

Nested Cross-Validation

Tuning against the same fold that reports the final score lets a search overfit the split. Two loops, one inside the other, keep the reported number honest.

An inner loop tunes a hyperparameter using only the outer loop's training rows, and the outer loop's held-out fold sees the chosen setting for the first time when it reports the final score

Why Does This Exist?

Cross-validation schemes names nested CV as the top rung of its ladder without explaining why the rung is needed. Hyperparameter tuning picks a setting by validation score without asking what happens when the same fold that picked the setting also reports how good it is. This page is where those two gaps meet.

Here's the case we'll carry down the page. You have 80 rows and 60 candidate features — a genomics-style panel where samples are expensive and columns are cheap — and you're tuning a ridge penalty over 30 candidate values with ordinary 5-fold cross-validation. Each fold picks whichever penalty scores best on that fold's own validation rows, then reports that same score as the model's generalization estimate. The number comes back at 0.75. Deploy the chosen setting on genuinely fresh data and it lands at 0.55.

Nothing was miscoded. The evaluation just asked the search to grade its own homework.

Think of It Like This

Letting a student see the exam before choosing which chapter to study

A student has one evening and thirty chapters to choose from, and can only study one before tomorrow's exam. Handed the exam questions tonight, they pick whichever chapter happens to cover what's actually being asked — not necessarily the chapter that would help most on a different exam covering the same subject, just the one that wins on this specific paper.

Tomorrow's grade looks excellent. It measures how well the student can pick a chapter after seeing the questions, not how well they know the subject. A fair test hides the real exam until after the choice is made, and grades the choice against a different set of practice questions the student never saw while choosing.

How It Actually Works

Two loops, and which rows each one is allowed to touch

Nested cross-validation runs an outer loop that estimates generalization and an inner loop, nested inside each outer fold, that tunes the hyperparameter — and the load-bearing rule is that the inner loop only ever sees the outer loop's training rows. For each outer fold: split the training rows again into inner folds, sweep every candidate hyperparameter across those inner folds, pick the winner, refit on the full outer-training set with that winner, and only then score on the outer fold's validation rows — rows the entire search process, inner and outer, never touched while choosing anything.

The outer score is what a model like this — including its own tuning process — would actually score on new data. The choice of hyperparameter is allowed to vary from outer fold to outer fold; nested CV isn't trying to hand you one final setting, it's trying to hand you an honest number for the procedure of searching and fitting.

Why the naive version is optimistic, and when the gap is largest

Ordinary cross-validation used for both tuning and reporting lets the search see the exact rows it will later be graded on, so among many candidate hyperparameters, whichever one happens to fit that fold's particular noise best will often win — a form of the same overfitting a single model can do to its own training rows, just one level up, applied to a search instead of a parameter vector. The gap between the naive and nested numbers grows with the size of the search space (more candidates means more chances one flukes a fold) and shrinks with the amount of data (more rows per fold means less room for any one candidate to specialize to noise). Few rows and many candidates, exactly this page's case, is the worst combination.

The cost, and where it's worth paying

Five outer folds times five inner folds times thirty candidate settings is 750 model fits, not the 150 an unnested search would need — nested CV multiplies cost by the outer fold count. That expense buys an unbiased generalization estimate, not a final model to ship; once nested CV has told you the honest number, the model you actually deploy is refit once, tuned by ordinary cross-validation on the full dataset. Skip the nested wrapper when data is abundant relative to the search space, where the naive number and the honest one are close enough not to matter; reach for it specifically when reporting a number that will drive a decision, on a dataset small enough that the gap could be large.

Show Me the Code

The same ridge search, small data and a wide sweep, scored the naive way and the nested way.

1import numpy as np2
3rng = np.random.default_rng(11)4n, p = 80, 60  # few rows, many candidate features -- room for the search itself to overfit5x = rng.normal(size=(n, p))6y = (x[:, 0] - 0.5 * x[:, 1] + rng.normal(0.0, 1.5, n) > 0).astype(int)7alphas = np.logspace(-3, 3, 30)8
9def acc(x_tr: np.ndarray, y_tr: np.ndarray, x_va: np.ndarray, y_va: np.ndarray, alpha: float) -> float:10    xb = np.c_[np.ones(len(x_tr)), x_tr]11    w = np.linalg.solve(xb.T @ xb + alpha * np.eye(xb.shape[1]), xb.T @ y_tr)12    return float(((np.c_[np.ones(len(x_va)), x_va] @ w > 0.5).astype(int) == y_va).mean())13
14folds = np.array_split(rng.permutation(n), 5)15naive, nested = [], []16for i, val in enumerate(folds):17    train = np.concatenate([f for j, f in enumerate(folds) if j != i])18    naive.append(max(acc(x[train], y[train], x[val], y[val], a) for a in alphas))  # tunes on val itself19
20    inner = np.array_split(rng.permutation(train), 4)  # tunes on train only, then scores on val21    scored = {a: np.mean([acc(x[np.concatenate([g for k, g in enumerate(inner) if k != j])],22                               y[np.concatenate([g for k, g in enumerate(inner) if k != j])],23                               x[inner[j]], y[inner[j]], a) for j in range(4)]) for a in alphas}24    nested.append(acc(x[train], y[train], x[val], y[val], max(scored, key=scored.get)))25
26print(f"naive  (alpha tuned against the same fold it's scored on): {np.mean(naive):.2f}")27print(f"nested (alpha tuned inside the outer fold's training rows): {np.mean(nested):.2f}")28# -> naive  (alpha tuned against the same fold it's scored on): 0.7529# -> nested (alpha tuned inside the outer fold's training rows): 0.55

Twenty points separate the two numbers on identical data and an identical model family. The naive score let each outer fold's penalty choice see the exact rows it would later be graded on; the nested score never let that happen, and it's the one that predicts what a fresh batch of samples would actually return.

Watch Out For

Running GridSearchCV once and reporting its best_score_ as the final number

A single GridSearchCV call does exactly the naive thing this page warns about: its folds both choose the hyperparameter and report the score used to choose it. best_score_ is a tuning diagnostic, not a generalization estimate, and reporting it as the latter is the most common way this mistake reaches a document someone acts on. Wrap the whole call inside an outer cross_val_score loop before quoting a number.

Nesting nested CV inside itself out of caution

Once the mistake is visible, it's tempting to add a third loop "to be safe" — nesting a second hyperparameter's search inside the inner loop. Every genuinely separate tuning decision needs its own honest nesting level, but stacking loops past what the decisions actually require only multiplies compute; a single inner loop can search a joint grid over several hyperparameters at once instead of nesting one loop per hyperparameter.

The Quick Version

An outer loop estimates generalization; an inner loop, nested inside each outer fold, tunes the hyperparameter using only that fold's training rows.
Tuning and reporting against the same fold lets the search overfit the split — the hyperparameter that wins is partly the one that flukes that fold's noise, not the one that generalizes best.
The gap between naive and nested estimates grows with search-space size and shrinks with data volume; small data plus a wide sweep is the worst case.
Nested CV costs roughly (outer folds) times an ordinary search, and it buys an honest estimate, not a model to ship — refit once on the full data afterward.
GridSearchCV.best_score_ used alone is the naive number; wrap it in an outer evaluation loop before quoting it as generalization performance.