Bias-Variance Tradeoff

When you expand model capacity, training error decreases monotonically toward zero. Evaluating on unseen test data reveals a different trajectory: test error bottoms out at an optimal complexity level ($0.27$) before rebounding sharply to $0.58$. Minimizing training error alone selects an over-parameterized model that has memorized idiosyncratic training noise rather than the underlying data distribution.

Shift, Not Just Spread

Target geometry cleanly separates the two error modes. Bias represents systematic error: overly simplistic model assumptions cause the entire cluster of predictions to miss the true target center by a constant offset, regardless of sample fluctuations. Variance measures sensitivity to the training sample: predictions scatter widely across trials, even if their empirical centroid remains centered on the bullseye. Effective modeling requires diagnosing whether errors stem from systematic inflexibility or unstable variance.

Bias Squared Plus Variance

For squared error loss, expected generalization error decomposes mathematically into three additive components:

$\mathbb{E}[(y - \hat{f}(x))^2] = \text{Bias}[\hat{f}(x)]^2 + \text{Var}[\hat{f}(x)] + \sigma^2$

At the sweet spot, total expected error of $0.27$ splits cleanly into $0.12$ of structural bias squared, $0.10$ of sample variance, and $0.05$ of irreducible data noise $\sigma^2$ generated by inherent label ambiguity.

Sweep Model Capacity

Fitting polynomials of increasing degree across a shared ten-point dataset demonstrates the progression. A linear model (degree 1) underfits with training error $0.38$ and held-out error $0.45$ due to rigid assumptions. A cubic polynomial (degree 3) balances flexibility with restraint, achieving a minimum held-out error of $0.27$. A degree-9 polynomial wiggles wildly through every single training point to reach $0.00$ training error, but explodes to $0.58$ test error when evaluated on held-out samples.

The Sweet Spot Pays

Comparing test performance on a shared scale highlights the reward for stopping at the optimal capacity. The balanced degree-3 model reduces held-out error from $0.45$ down to $0.27$—an outright error reduction of $0.18$. Pushing further into degree 9 imposes an overfitting penalty of $+0.31$, erasing earlier gains.

Add Data or Capacity

The optimal remedial action depends on the available sample size. With $30$ points, high variance dominates: training error is low ($0.05$) while test error remains high ($0.48$). Gathering more training samples narrows the generalization gap. With $300$ points, learning curves converge at a high plateau ($0.38$ test vs $0.35$ train), indicating that the model has reached its representational limit; adding model capacity breaks through the high-bias ceiling.

Where It Breaks

Irreducible noise $\sigma^2 = 0.05$ establishes a fundamental lower bound below which no model can perform. Once structural bias is minimized, allocating additional capacity forces parameter weights to fit stochastic fluctuations in the training sample. Training error flatlines at zero while test error diverges toward $0.58$.

The Quick Version

• Training error monotonically drops to zero; test error forms a U-curve rebounding to 0.58.
• Bias systematically shifts predictions off center; variance disperses predictions widely.
• Total error decomposes additively into bias squared (0.12), variance (0.10), and noise (0.05).
• Degree 9 hits 0.00 training error but degrades to 0.58 generalization error.
• The sweet-spot model cuts held-out error by 0.18 compared to the underfit baseline.
• When sample size is small (N=30), add data; when plateaued on large data (N=300), add capacity.
• The irreducible noise floor (0.05) cannot be modeled; excessive capacity overfits stochastic jitter.