L1 vs L2 Regularization

The Short Answer

L1 adds the sum of absolute weights to the loss; L2 adds the sum of squared weights. That small change produces a qualitatively different model rather than a slightly different one: L1 sets coefficients to exactly zero, L2 makes them small but keeps them all.

The reason is the shape of each penalty, not its severity. Picture the set of weight vectors a norm calls length one. L2's is a smooth circle. L1's is a diamond whose corners sit precisely on the axes — and a corner is a point where one coordinate is exactly zero. A penalised optimum is far more likely to land on a spike than on a flat face, so L1 solutions collapse coordinates to true zeros as a consequence of geometry. The full argument is in norms.

Both are also priors in disguise. L2 is a Gaussian prior on the weights and L1 is a Laplace prior, whose sharp spike at zero is the same fact as the diamond's corners. That framing comes out of maximum likelihood estimation.

Where They Differ

	L1 (Lasso)	L2 (Ridge)
Penalty added to the loss	$\lambda \sum_i \lvert w_i \rvert$	$\lambda \sum_i w_i^2$
Effect on weights	Many become exactly 0	All shrink, none vanish
Feature selection	✅ Built in	❌ No
Differentiable everywhere	❌ Corners at 0	✅ Smooth
Closed-form solution (linear case)	❌ Needs iterative solvers	✅ Yes
Correlated features	Picks one arbitrarily, zeroes the rest	Splits weight between them
Stability across resamples	Lower — the chosen set shifts	Higher
Equivalent prior	Laplace	Gaussian
Robustness to outliers as a loss	Higher (linear growth)	Lower (quadratic growth)
Typical framework name	l1, lasso	weight_decay, ridge

Choose A When

Pick L1 when sparsity is something you want as an output, not just a side effect.

• You need to know which features matter. L1 hands you a subset, which is a genuine deliverable when the model has to be explained to someone.
• You have far more features than samples. Text with wide sparse vectors, genomics, any wide tabular problem. L1 cuts the model down to what the data can actually support.
• Inference cost depends on feature count. Zeroed coefficients mean features you never have to compute or fetch at serving time — often the real saving.
• You want a smaller model to ship. Exact zeros compress; small nonzero values do not.

Choose B When

Pick L2 for almost everything else, and specifically:

• You are training a neural network. Weight decay is L2, it is the default in every optimiser, and sparsity in a dense layer buys you nothing without structured pruning to exploit it.
• Your features are correlated. L2 distributes weight across correlated features; L1 picks one and discards the rest, and which one it picks can change when you resample the data.
• You want stable, reproducible coefficients. L2's smoothness makes the optimisation well behaved and the solution unique.
• You need a closed form. Ridge has one; lasso does not.

If you cannot decide, elastic net combines both penalties and exists precisely for the correlated-features case where L1 alone is unstable.

What People Get Wrong

Treating it as a strength dial. L1 and L2 are not weak and strong versions of one idea. They produce different kinds of solution, and the coefficient $\lambda$ is the strength dial within each.

Expecting sparsity to speed up a neural network. Zeros in a dense weight matrix still get multiplied. Without structured sparsity and kernels that exploit it, an L1-regularised network is the same speed as an L2 one.

Confusing the penalty with the measurement. Gradient clipping rescales a gradient by its L2 norm to bound step size while preserving direction. That is unrelated to choosing a regularisation penalty, and the two get conflated constantly.

Forgetting to scale features first. Both penalties treat every coefficient on the same footing, so a feature measured in millimetres gets a systematically different effective penalty from one measured in metres. Standardise before regularising, or the penalty is really a statement about your units.

Regularising the bias. The bias term sets the output's baseline level and shrinking it toward zero is rarely what you want. Most implementations exclude it; hand-rolled ones often do not.