Core ML

Model Selection and Validation

Choosing between candidate models requires a rigorous protocol — the choice must be made on a validation set that is never touched during training and never confused with the test set used for the final performance estimate.

Three-way data split: training data trains all candidate models; validation data selects the best model and tunes hyperparameters; the test set is used only once for the final performance estimate — conflating any two of these roles produces an optimistic bias in the reported metric

Why Does This Exist?

Three candidate models: random forest, gradient boosted tree, logistic regression. You evaluate all three on the test set, pick the random forest because it has the highest accuracy, report 91.3% accuracy. The problem: the test set was used to make a decision (which model to deploy). The 91.3% is now an optimistic estimate — the random forest was effectively selected because it got lucky on the test set, and that luck won't generalise to production.

Model selection is the process of choosing between candidates. Validation is the process of estimating how well the chosen model will perform on unseen data. These are two different questions, and they need two different data splits.

Think of It Like This

A university exam where students choose which exam to sit after seeing all the papers

If students can look at ten exam papers and then choose which one to sit, the pass rate from the chosen exams overstates their actual knowledge — each student picks the exam they happen to know best. The exam result is no longer an unbiased test of the curriculum.

Using the test set for model selection is the same mistake. You're choosing the model that got lucky on the test set, and then reporting its test set performance as the final accuracy.

How It Actually Works

The three-way split

For most practical purposes: split data into training, validation, and test sets. The ratio depends on total dataset size.

Training set (typically 60–70%): train all candidate models.
Validation set (typically 10–20%): select between candidates, tune hyperparameters.
Test set (typically 10–20%): estimate final performance. Touch once.

The validation set is used many times (for every model and every hyperparameter combination). The test set is used once. If the test set is examined during development — even just to spot-check — it is no longer an unbiased estimator.

Cross-validation for model selection

When the dataset is too small for a three-way split to give a reliable validation estimate, use k-fold cross-validation on the training data:

Split training data into $k$ folds.
Train each candidate model $k$ times, each time holding out one fold as validation.
Average the validation metric over all $k$ folds.
Select the candidate with the best cross-validation score.
Retrain the selected model on all training data.
Evaluate once on the test set.

A common mistake: running k-fold cross-validation on the entire dataset (including the test set) and reporting the cross-validation score as the final estimate. This leaks the test set.

Information criteria

For models fitted by maximum likelihood, AIC and BIC penalise the likelihood by model complexity:

\text{AIC} = 2k - 2\ln\hat{L} \qquad \text{BIC} = k\ln n - 2\ln\hat{L}

where $k$ is the number of parameters and $\hat{L}$ is the maximised likelihood. BIC penalises more heavily for larger $k$ and favours simpler models. Both avoid the need for a separate validation set, but only apply to models that maximise a likelihood.

Nested cross-validation

When both model selection and hyperparameter tuning are required, a single k-fold validation loop conflates the two. Nested cross-validation uses an outer loop for model selection and an inner loop for hyperparameter tuning — ensuring neither leaks into the other's evaluation. Expensive but unbiased. See Nested Cross-Validation for the full procedure.

Show Me the Code

1import numpy as np2
3
4def k_fold_cv(X, y, model_fn, k=5, seed=0):5    """Returns mean and std of validation metric across k folds."""6    rng = np.random.default_rng(seed)7    idx = rng.permutation(len(y))8    folds = np.array_split(idx, k)9    scores = []10    for i in range(k):11        val_idx = folds[i]12        train_idx = np.concatenate([folds[j] for j in range(k) if j != i])13        model = model_fn()14        model.fit(X[train_idx], y[train_idx])15        score = model.score(X[val_idx], y[val_idx])16        scores.append(score)17    return float(np.mean(scores)), float(np.std(scores))18
19
20# Usage sketch21from sklearn.ensemble import RandomForestClassifier22from sklearn.linear_model import LogisticRegression23import sklearn.datasets as ds24
25X, y = ds.make_classification(n_samples=500, n_features=10, random_state=0)26
27for name, fn in [("LR", lambda: LogisticRegression(max_iter=1000)),28                 ("RF", lambda: RandomForestClassifier(n_estimators=50))]:29    mean, std = k_fold_cv(X, y, fn)30    print(f"{name}: CV accuracy = {mean:.3f} ± {std:.3f}")31# Pick the model with higher CV accuracy — test set still untouched

Watch Out For

Touching the test set more than once

Every time you examine the test set performance and make a subsequent modelling decision, you leak the test set into your model selection process. "The RF had 91.3% on the test set; let me increase the number of trees and try again" means the second model was selected partly because of test set information. Keep a lab notebook. Note the first time you look at the test set — everything before that is model selection, and everything after that should not exist.

k-fold CV on time-series data with random fold assignment

Random k-fold shuffles time-series data and creates future-into-past leakage — a validation fold may contain observations from before the training fold. Use time-series cross-validation (expanding window or sliding window) that always trains on past data and validates on future data. See Time-Series Cross-Validation.

The Quick Version

Three-way split: training (fit), validation (select), test (report). Never use the test set for any decision.
Cross-validation replaces the validation split when data is scarce. Run it on training data only.
AIC/BIC are likelihood-based alternatives to a validation split for parametric models.
Nested cross-validation separates model selection from hyperparameter tuning — the gold standard for small datasets.
The reported test set accuracy is only meaningful if the test set was never consulted during development.