Foundations

Train/Test Split & CV

A held-out score is only as honest as the split that produced it — how you cut the data decides what the number means.

Class A sample
Class B sample

Dataset size

How many samples there are to divide up.

Samples40

Forty samples, each with one feature and one of two labels. Class A outnumbers class B three to one, and the row is sorted by class — every A, then every B. Drag the size slider; the imbalance holds at any size.

Check your understanding

How data partitioning works. Splitting testing data correctly ensures that you can robustly evaluate how a model will perform on unseen scenarios in the wild.

We rely on machine learning models because we expect they have grasped the fundamental patterns of a task and can generalize well to data they have not seen. If we evaluated a model using the exact same data it learned from, we would have no way of knowing whether the model actually learned anything universal or if it just perfectly memorized the examples.

By strategically partitioning data into separate buckets, we can cleanly isolate a model's training data from its testing evaluation data.

Shuffle and Stratify

Look at the initial dataset pattern. Real-world dataset records often arrive grouped sequentially or chronologically (for example, all positive samples collected in the morning, and negative samples in the evening). If you simply draw a cut down the middle without shuffling, your Training Set might get 100% of one class, leaving none for testing!

Shuffle: Mixes the dataset uniformly so drawing any boundary results in a randomized array of samples.
Stratify: Mathematically spaces samples based on their class labels. This ensures that no matter where the train/test split lands, both sets will maintain the exact same proportional representation of positive vs negative examples as the whole dataset.

Cross-Validation

A single random split still holds a structural problem for small datasets: holding out 20% of your data means deliberately depriving the learning algorithm of a chunk of critical teaching examples.

K-Fold Cross-Validation solves this. By dividing the total dataset into $K$ equal chunks (folds), the model operates on a rotational system: one chunk is dynamically picked to act as the testing hold-out, and the remaining 4 chunks train the model. This process rotates $K$ times, testing every data point precisely once, ending up with $K$ unique test scores whose statistical mean serves as the ultimate benchmark.

Reference

Held-out score: accuracy on samples the model was not fitted to
k-fold CV: split into k folds; each takes a turn as the held-out set, then average
Stratified split: every fold keeps the whole dataset's class proportions
Why the spread matters: mean ± std across folds; a low mean with a high std means one fold failed badly
Decision stump: the model here: predict by comparing one feature against one threshold

Break it on purpose

Leave Shuffle and Stratify off at k = 4. The dataset is sorted by class, so three folds are entirely class A and the last is entirely class B. The model trained for that last fold has never seen a single B, so it labels everything A and scores 0 — while the mean across folds still reads a comfortable 0.68. Turn Stratify on and the same model scores 0.9 with the spread collapsing from 0.41 to 0.07.