Learning Curves
Diagnosing bias vs variance from data volume — what more data actually fixes.
Before spending money on labelling more data, you should ask if it will actually help. A learning curve answers this by plotting model performance against the size of the training set.
High Bias (Underfitting)
If a model is too simple to capture the underlying pattern, both training and validation error will converge early and plateau at a high error rate. Adding data beyond this point provides zero benefit.
High Variance (Overfitting)
If a model is too complex, it memorises the noise. Training error will be very low, but validation error stays high, creating a large gap. In this scenario, adding more data will force the model to generalise, pulling the curves together.
Where It Breaks
The single most expensive mistake in ML is gathering more data to fix a high-bias model. If your learning curves have already flattened together, more data will not move them. You must increase model complexity or add better features instead.
The Quick Version
- Learning Curve: Plots error against training set size.
- High Bias: Curves converge quickly but remain at a high error.
- High Variance: Large gap between training and validation error.
- The Fix: More data only cures high variance (overfitting).
- The Failure: Wasting money on data when the model is underfitting.