Foundations
Bias-Variance Tradeoff
Why doesn't a model that perfectly memorizes the training data generalize well? See exactly how model complexity trades off between capturing the broad signal and obsessing over the noise.
How model complexity drives a tradeoff between capturing the underlying signal (bias) and overreacting to noise (variance).
Stage 1 of 5: The Data
Degree 1. Train MSE: 0.094. Val MSE: 0.083.
- Train Data
- Val Data
A scatter plot of training and validation data sampled from a true hidden function with some noise.
Check your understanding
2 questions in the bank. Each attempt draws a fresh set in a fresh order, so a second go is a real second go.
Overfitting and Underfitting
When building a model, the goal is to uncover the underlying pattern holding the data together while ignoring the random noise that happened to be recorded alongside it.
This simulation lets you play this balance manually. A flat line (degree 1) assumes the whole world is linear. It doesn't capture the subtle pattern, so it has high bias—it is underfitting.
If you push the complexity up, you give the model enough flexibility to hit every single point. It drives the training error to near zero. But these models are brittle—a tiny change in the data makes the curve wildly different. This is high variance—overfitting.
Try pushing the degree high, then hit Resample Data. Look at how the faint ghost copies vary wildly, while a simple model stays much more consistent.
Reference
- Bias
- Error from erroneous assumptions (e.g. assuming linear when underlying function is curved)
- Variance
- Error from sensitivity to small fluctuations in the training set
- Overfitting
- Modeling noise instead of the underlying signal, characterized by low train error and high val error
- Underfitting
- Modeling neither noise nor the signal, characterized by high train and high val error
Break it on purpose
Pushing the degree up to 9 for a tiny N produces a wild overfit, where the validation error skyrockets and ghost models span everywhere.