Bias vs Variance
Understanding the fundamental tradeoff between a model making systematic errors and being overly sensitive to noise.
Verdict: Attack Bias first. An underfitting model (high bias) learns nothing useful. Once Bias is low, tackle Variance to prevent overfitting.
The Short Answer
Every machine learning model's total error can be mathematically decomposed into three parts: Bias, Variance, and Irreducible Noise.
Bias is the error introduced by assuming a complex problem is simple. A model with high bias pays very little attention to the training data and oversimplifies things (underfitting). It consistently misses the mark in the exact same way.
Variance is the error introduced by a model being too sensitive to small fluctuations in the training set. A model with high variance pays too much attention to the training data (overfitting), learning random noise instead of the true signal.
Where They Differ
| Feature | Bias | Variance |
|---|---|---|
| Definition | Error from erroneous assumptions (oversimplifying). | Error from sensitivity to small fluctuations in training data. |
| Symptom | Model performs poorly on both training and test data. | Model performs great on training data, poorly on test data. |
| Common Name | Underfitting. | Overfitting. |
| Bullseye Metaphor | Hits are clustered, but far from the center. | Hits are scattered all over the board. |
| Fix | Make the model more complex. | Get more data, or simplify the model (Regularization). |
The Tradeoff
It is called the Bias-Variance Tradeoff because decreasing one usually increases the other.
If you have a high-bias model (e.g., a straight line trying to fit a curved dataset), you can reduce bias by adding complexity—like switching to a high-degree polynomial. But if you make it too complex, it will bend to hit every single noisy data point perfectly. Your bias is now zero, but your variance has exploded.
The goal of machine learning is to find the sweet spot: the level of model complexity that minimizes the total error (Bias + Variance).
Choose to attack A When
(When to reduce Bias)
- Training error is unacceptably high: The model isn't even learning the training data.
- You are using a very simple algorithm: For example, Linear Regression on a highly non-linear dataset.
- How to fix it: Add more parameters (more layers to a neural network, deeper trees), add new features, or train longer.
Choose to attack B When
(When to reduce Variance)
- Training error is near zero, but validation error is high: The model memorized the training data but failed to generalize.
- You are using a highly complex algorithm: For example, an unpruned Decision Tree or a massive Neural Network on a small dataset.
- How to fix it: Get more training data. If you can't, use Regularization (L1/L2, Dropout), prune trees, or use Ensembling techniques (like Random Forests).
What People Get Wrong
Trying to minimize both simultaneously from the start
In practice, you rarely attack both at once. The standard modern workflow is:
- Make your model complex enough to completely overfit your training data (drive Bias to near zero).
- Apply regularization techniques (Dropout, weight decay) to generalize it (drive Variance down).
Confusing Model Bias with Societal Bias
In everyday language, "bias" means prejudice. In machine learning ethics, it also means prejudice (e.g., a model favoring certain demographics). But in the Bias-Variance Tradeoff, bias is a purely mathematical term meaning systematic statistical error.