Bias-Variance Tradeoff
Underfitting vs overfitting as darts on a board — why complexity alone doesn't win.
When evaluating a model, error comes from two main sources. High bias means the model is consistently off-target. High variance means the model's predictions are scattered and highly sensitive to the specific training data.
Underfitting
A model with high bias makes strong assumptions about the data. If you fit a straight line to curved data, it doesn't matter how much data you collect. The model is simply too rigid to capture the true underlying pattern.
Overfitting
A model with high variance is overly complex. It threads a wavy line through every single training point, perfectly memorizing the noise. When given new data, its predictions will wildly fluctuate and fail.
The sweet spot
As you increase model complexity, bias goes down but variance goes up. The total test error forms a U-curve. The optimal model is found at the bottom of this curve, where the combined error is minimized.
Where It Breaks
You cannot decrease both simultaneously just by changing model complexity. Pushing bias down forces variance up, and vice versa. It is a fundamental tradeoff you cannot escape without getting more data.
The Quick Version
- Bias is systematic error; variance is sensitivity to noise.
- High bias leads to underfitting because the model is too rigid.
- High variance leads to overfitting because the model memorizes noise.
- The total error forms a U-curve against complexity.
- You must trade one against the other to find the minimum error.