Core ML

Time Series Cross-Validation

Shuffling a time-ordered fold lets a model peek at the future to score the past. Expanding and sliding windows keep training strictly earlier than validation.

An expanding window always trains on everything before the point it validates, while an ordinary shuffled fold lets rows from both sides of that point train on each other

Why Does This Exist?

Cross-validation schemes covers the ladder of folds — k-fold, stratified, grouped — and names time-series split as the row where shuffling stops being safe. This page is that row, in full.

Here's the case we'll carry down the page. You're forecasting tomorrow's electricity demand from the last 90 days of readings, and you want an honest estimate of how well a model will do before it ships. Five-fold cross-validation on this series, shuffled the ordinary way, hands one fold March and December in training and June in validation. The model interpolates: it has seen weather either side of the week it's scoring, which production never will. The number comes back looking excellent and means nothing.

The fix isn't a different metric. It's a different definition of which rows are allowed to train on which.

Think of It Like This

Grading a weather forecaster with next week's paper

A forecaster predicts tomorrow's rain each morning, and at the end of the month you want to grade the run. Grading fairly means checking each day's forecast only against what was known that morning — yesterday's readings and before.

Shuffle the month's days into five piles and grade each pile using the other four as "training" for a hand-fitted rule, and the forecaster's Tuesday forecast gets built with the help of Thursday's actual rainfall. Of course it looks accurate. It was allowed to read next week's paper before writing this week's forecast.

A fair grading walks forward one day at a time, using only what happened before. That's the whole of what an expanding window enforces mechanically.

How It Actually Works

Two windows, one rule in common

Both variants share the rule that matters: every validation block sits strictly after every row used to train the model scored on it. They differ only in how much history each fold's training set keeps.

An expanding window starts training at the beginning of the series and grows it with each fold — fold one trains on the first tenth, fold two on the first fifth, and so on, always validating on the block immediately after. A sliding window keeps the training length fixed and slides both training and validation forward together, dropping old rows as it goes. Expanding uses more data per fold and suits a stable process; sliding matches training data volume to what production will actually have and adapts faster if the underlying process drifts.

Walk-forward validation is the same idea done for real deployment

Where cross-validation asks "how good is this model on average", walk-forward validation simulates the actual production schedule: retrain (or not) on a fixed cadence, predict the next block, record the score, advance, repeat. It's the same expanding or sliding window mechanic, run at the granularity a system would actually retrain at — weekly, say, rather than five arbitrary folds — so the reported number matches the process that will really run.

Why the naive number was optimistic, and by how much it lies

A shuffled fold on time-ordered data lets training rows surround the validation block on both sides, so the model effectively interpolates rather than extrapolates. Interpolation is a strictly easier problem than forecasting forward, and the gap between the two is not a rounding error — it's the entire difference between a number that describes the past and one that describes what will actually happen when the model runs forward in time. The size of the inflation depends on how strongly the series is autocorrelated and whether the underlying process has structural breaks a shuffled fold would smear across the whole dataset instead of isolating.

The one thing to check before trusting either window

Neither window fixes the model's exposure to a genuine one-off level shift — a policy change, a currency shock, a physical plant closure — sitting in the training data of a fold that validates on the other side of it. That's not a leakage problem cross-validation windows solve; it's a problem checked separately by inspecting the series for known breaks and either fitting a break-aware model or excluding the affected training rows explicitly.

Show Me the Code

The same lag-1 model, scored two ways on a series with a structural break near the end: an ordinary shuffled fold, and an expanding window.

1import numpy as np2
3rng = np.random.default_rng(3)4n = 5005t = np.arange(n)6regime = (t > 350).astype(float)                            # a structural break near the end7series = 0.02 * t + 8.0 * regime + rng.normal(0.0, 1.0, n)   # slow drift, then a level shift8lag1 = np.r_[series[0], series[:-1]]                         # yesterday's value predicts today's9
10def mae(train: np.ndarray, val: np.ndarray) -> float:11    w = np.polyfit(lag1[train], series[train], 1)12    return float(np.mean(np.abs(np.polyval(w, lag1[val]) - series[val])))13
14def shuffled_kfold(k: int = 5) -> float:15    folds = np.array_split(rng.permutation(n), k)16    return float(np.mean([mae(np.concatenate([f for j, f in enumerate(folds) if j != i]), v)17                           for i, v in enumerate(folds)]))18
19def expanding_window(k: int = 5) -> float:20    cuts = np.linspace(n // (k + 1), n, k + 1).astype(int)21    return float(np.mean([mae(np.arange(0, cuts[i]), np.arange(cuts[i], cuts[i + 1])) for i in range(k)]))22
23print(f"shuffled k-fold MAE:   {shuffled_kfold():.2f}")24print(f"expanding-window MAE:  {expanding_window():.2f}")25# -> shuffled k-fold MAE:   1.1526# -> expanding-window MAE:  1.66

The shuffled fold reports a smaller error because some of its folds train on rows from after the level shift while validating on rows from before it, borrowing information a real forecast could never have. The expanding window's larger, honest number is the one that predicts what deploying this model would actually look like.

Watch Out For

Reaching for the shuffle default without noticing the data is a series

KFold(shuffle=True) is the snippet everyone has memorised, and it runs without error on time-ordered data — there's no exception to catch. The result looks stable, sometimes more stable than a time-series split, because every fold cheats the same way and their scores agree. Check whether a "date" or "timestamp" column exists and whether rows are naturally ordered before choosing a splitter; if either is true, shuffling is very likely wrong.

Treating a single early fold's score as representative

The first few folds of an expanding window train on very little history, so their scores are noisy and often worse than later folds purely from having less data, not from the model being wrong. Reading the earliest fold's number as "the" score, or stopping the evaluation before the window has grown to a realistic training size, understates how the model will actually perform once it has the history production will give it. Report the trend across folds, and weight the later, larger-window folds more heavily when it's time to summarize into one number.

The Quick Version

Every validation block must sit strictly after every row that trained the model scored on it — the one rule both windows enforce.
An expanding window grows training data fold over fold; a sliding window keeps training length fixed and moves both windows forward together.
Walk-forward validation is the same mechanic run at the cadence a real deployment would actually retrain at.
A shuffled fold on time-ordered data lets the model interpolate between rows on both sides of the point it's scoring, which inflates the reported score in a way no amount of tuning later corrects.
Neither window alone protects against a genuine structural break landing across a fold boundary; check the series for known breaks separately.