Ridge & Lasso Regularization

The model in this lab is offered eight features — $x, x^2, \dots, x^8$ — to fit sixteen noisy points. The points were generated by a cubic, $y = 2x - 1.5x^3 + \varepsilon$, so six of those eight features are pure decoration. Least squares does not know that, and uses all eight anyway.

Regularization is what you add to the loss to stop it. Instead of minimising the error alone, the model minimises the error plus a penalty on how large its coefficients are:

$$J(w, b) = \frac{1}{2n}\sum_i \left(y_i - b - z_i \cdot w\right)^2 + P(w)$$

Ridge (L2 penalty)

Ridge sets $P(w) = \lambda \sum_j w_j^2$. Differentiate it and the gradient is $2\lambda w_j$ — which shrinks towards nothing as $w_j$ does. So ridge can pull a coefficient arbitrarily close to zero and never arrive. In the lab, every one of the eight bars shortens together, and at the strongest penalty on the control all eight are still standing.

That is often exactly what you want. When a signal is genuinely spread across several correlated features, discarding any one of them throws away information the others cannot fully replace, and shrinking them all is the better trade.

Lasso (L1 penalty)

Lasso sets $P(w) = \lambda \sum_j |w_j|$, and the absolute value has no derivative at zero — it has a corner. Its subgradient there spans $[-\lambda, +\lambda]$, so whenever the unpenalised pull on a coefficient falls inside that interval, zero genuinely solves the stationarity condition and the coefficient is switched off. That is the soft-thresholding operator:

$$S(\rho, \lambda) = \operatorname{sign}(\rho)\,\max(|\rho| - \lambda, 0)$$

Because six of this dataset's eight features really are redundant, removing them beats shrinking them: lasso reaches a held-out error of 0.663 where ridge floors at 0.728. And the two features it keeps hold of hardest are $x$ and $x^3$ — the two the target was built from.

Standardise before you penalise

The penalty is applied to $w$, not to the features, so it only means the same thing in every column if the columns are on the same scale. These are not: the mean squares of $x$ through $x^8$ over this domain span 0.85 to 91. A large column needs a tiny weight to do the same work, so it pays almost no penalty.

This matters more than it sounds. Run lasso on the unstandardised columns and it keeps $x^7$ and $x^8$ and throws away $x$ and $x^3$ — precisely the wrong answer, arrived at confidently. The lab standardises each column using the training set's own mean and standard deviation, which is why $\lambda$ is comparable across features at all.

Where it breaks

Push $\lambda$ past 0.497 with lasso selected and there is nothing left. Every coefficient is zero, the intercept survives untouched — the penalty never applies to it — and the model predicts the training mean, $-0.307$, for every input. The curve is a flat line, and held-out error climbs to 1.172, worse than at any other setting on the control.

That value is not a coincidence, and the lab marks it. It is $\lambda_{\max} = \max_j \left|\tfrac{1}{n} z_j^\top (y - \bar{y})\right|$: the largest correlation between any standardised feature and the centred target. Above it, zero satisfies the stationarity condition for all eight coefficients at once.

Being simple is not the same as being right. A model that is wrong everywhere has not been regularised — it has been deleted.