Core ML

Lasso Regression

Charge the absolute size of each coefficient instead of its square and the penalty stops easing off near zero, so coefficients land on it. Selection, for free.

Grow the error contour out from the unpenalised fit and it touches the L1 diamond at a corner, where one coefficient is exactly zero, while the L2 circle has no corner to catch it

Why Does This Exist?

Sixty delivery vans. Nine hundred telemetry channels each, and one number per van: millimetres of brake pad lost over 40,000 km. The fleet manager doesn't want a prediction. She wants a shortlist, because the next fleet gets ordered with twelve sensors and somebody has to say which twelve.

Ridge can't answer that. It returns nine hundred coefficients, all small, all nonzero, and 0.0004 is not the same statement as "this channel doesn't matter".

One thing first. Least squares picks one coefficient per channel — how much predicted wear moves per unit of that reading — to make squared misses smallest, and a penalty charges a second cost on the size of those coefficients. Ridge charges each coefficient's square; lasso charges its absolute value.

One swapped exponent, and it's the whole difference between shrinking and selecting.

Think of It Like This

A floor with friction, not a spring

Picture one coefficient as a crate on a chalk line, and the evidence in your data as a push trying to slide it off. The penalty resists.

Hook the crate to the line with a spring and the resistance dies away as it gets close: right on the line the spring is slack, so the faintest push nudges it off. That's a squared penalty, and why ridge never hands back a zero.

Now unhook the spring and stand the crate on rough concrete. Friction resists with the same force at every distance, zero included. Push with less than that and nothing moves. Push harder and it slides, but stops short by exactly the friction's worth.

That's the arithmetic, not a metaphor. Evidence weaker than the penalty leaves a coefficient at exactly zero; stronger evidence moves it, minus the penalty.

How It Actually Works

Same squared error as always, one term swapped in:

\sum_{i=1}^{n}\big(y_i - x_i^\top w\big)^2 + \lambda \lVert w \rVert_1

$y_i$ is van $i$ 's measured pad wear, $x_i$ its row of readings, $w$ the coefficients you're solving for. That $\lVert w \rVert_1$ is the L1 norm — add up the absolute values, no squaring — and $\lambda$ sets how hard it pushes back.

Why the corners do the selecting

Every penalty has a twin, and lasso's is the one you can draw: minimising error with total absolute size capped at $t$ gives back the same solutions.

Two coefficients, so it fits on a page. The points where $\lvert w_1 \rvert + \lvert w_2 \rvert \le t$ form a square standing on one corner — a diamond, all four corners on an axis. Sitting on an axis means one coefficient is exactly zero.

Squared error, as a function of the coefficients, is a bowl, and slicing it gives an ellipse centred on the unpenalised fit. Grow that ellipse until it first touches the diamond and you have the picture above.

Where does a growing convex blob first meet a diamond? At a corner, most of the time, and the reason is countable, not mystical. A point on an edge can be touched from exactly one direction, the one square to that edge. A corner can be touched from a whole wedge of them. Corners catch a range of positions; edge points catch a knife-edge. Zeros aren't luck, they're what a region with corners does.

Redraw it with the circle you get from capping squared size and there's nothing to catch. Both coefficients come back nonzero: shrinking, not selecting.

No closed form, and a path made of straight lines

Ridge collapses into one matrix equation because you can differentiate a square. $\lvert w \rvert$ can't: at zero its slope jumps from $-1$ to $+1$ , so there's nothing to set equal to zero.

Coordinate descent handles it instead: freeze every coefficient but one, solve that one alone, sweep until nothing shifts. The one-variable answer is a soft threshold — take the evidence for that channel, slide it toward zero by $\lambda$ , and if that carries it past zero, stop at zero. Friction, written as arithmetic. The objective stays convex, so the sweeps still land on a global optimum.

Sweep $\lambda$ from huge down to nothing and each coefficient traces straight segments, kinking only when a channel enters or leaves the nonzero set. That piecewise-linear path makes every model along it cheap.

One ceiling worth knowing: with sixty vans, lasso can never return more than sixty nonzero coefficients. A property of the solution, not a tuning artefact — if pad wear honestly depends on three hundred channels, lasso can't say so.

Show Me the Code

Five channels instead of nine hundred. Columns 0 and 1 are two pad-temperature sensors bolted to the same axle, so they agree all season.

1import numpy as np2
3
4def lasso(X: np.ndarray, y: np.ndarray, lam: float, sweeps: int = 2000) -> np.ndarray:5    w = np.zeros(X.shape[1])6    for _ in range(sweeps):  # coordinate descent: |w| has no derivative at 0, so no closed form7        for j in range(X.shape[1]):8            g = (y - X @ w + X[:, j] * w[j]) @ X[:, j] / len(y)9            w[j] = (max(g - lam, 0.0) - max(-g - lam, 0.0)) / (X[:, j] @ X[:, j] / len(y))10    return w11
12
13for week in (2, 3, 4):  # one fleet of sixty vans, three different weeks of telemetry14    rng = np.random.default_rng(week)15    X, axle = rng.normal(size=(60, 5)), rng.normal(size=60)16    X[:, 0] = axle + rng.normal(0.0, 0.01, 60)  # left pad sensor17    X[:, 1] = axle + rng.normal(0.0, 0.01, 60)  # right pad sensor, same axle, same reading18    y = 1.5 * (X[:, 0] + X[:, 1]) + rng.normal(0.0, 0.5, 60)19    w = lasso(X, y, 0.1)20    print(f"week {week}: pads {w[0]:+.2f} {w[1]:+.2f}   sum {w[0] + w[1]:.2f}   rest {w[2:].round(2)}")21    # -> week 2: pads +2.75 +0.15   sum 2.90   rest [0.   0.02 0.  ]22    # -> week 3: pads +2.88 +0.00   sum 2.88   rest [0. 0. 0.]23    # -> week 4: pads +1.00 +2.05   sum 3.04   rest [0. 0. 0.]

The noise channels land on exactly zero — selection, working. The two sensors sum to about 2.9 against a true 3.0 every week, so predictions hold. How that 2.9 divides is a coin flip: 2.75 and 0.15, then 2.88 and nothing, then 1.00 and 2.05. Held-out error barely notices; the survivor carries the signal for both. A shortlist notices a lot.

Watch Out For

Reading the surviving coefficients as the important channels

The symptom is a slide claiming twelve key signals, and a nonzero set that reshuffles when you refit on next month's data. Lasso answers "which small set predicts well", and when two channels carry the same information it has no reason to prefer either, so it keeps whichever correlated a hair better. Zero looks like a verdict. It's a tie-break.

Report groups instead. Cluster the features by correlation, then read the shortlist as "one representative from this cluster survived" — a claim about the cluster, and a true one. If you need members ranked inside a cluster, feature selection has tools for it.

Shipping the shortlist without resampling it first

Nine hundred columns, sixty rows, one fit, twelve survivors, and a purchase order for twelve sensors. Refit on a bootstrap resample and eight of the twelve change. One fit gives no hint it was unstable.

Refit on a few hundred resamples and count how often each channel comes back nonzero. That selection frequency is what's worth acting on: 90% is a finding, 30% is noise wearing a coefficient. If whole clusters keep trading places, elastic net holds them together.

The Quick Version

Lasso adds $\lambda \lVert w \rVert_1$ to squared error — absolute values, not squares — and its pull holds constant all the way to zero, so coefficients arrive and stay. Ridge's fades first, which is why it never does.
The constraint region is a diamond with corners on the axes. A growing error contour meets a corner from a wedge of directions, an edge from only one.
No closed form, since $\lvert w \rvert$ isn't differentiable at zero. Coordinate descent handles it, the objective stays convex, and the path it traces is piecewise linear.
At most $n$ nonzero coefficients from $n$ rows, and correlated features get picked between arbitrarily. Predictions survive that; interpretations don't.