Core ML

Linear Discriminant Analysis

PCA rotates axes toward the direction data spreads most. LDA rotates toward whatever direction best separates classes you know instead, rarely the same axis.

PCA's chosen axis runs along the data's overall spread, missing the classes entirely, while LDA's axis runs across the two class means, along the one direction where the classes actually separate

Why Does This Exist?

PCA picks its axes by one rule only: whichever direction the data spreads the most, then the most remaining spread at right angles, and so on. Nothing about that rule ever looks at a label, because PCA has none to look at — it is explicitly unsupervised. The moment you have classes and want to reduce dimensions for the purpose of separating them, PCA's rule can point somewhere useless, and there's no way to notice from inside PCA that it happened.

Here's the case we'll carry down the page. Two customer classes sit in a two-column space: both spread widely along a "browsing time" axis that has nothing to do with which class they're in, and the classes are cleanly separated along a second, narrower axis — say, a support-ticket count that barely varies within either class but differs sharply between them. PCA's top component is the browsing-time axis, because that's where the overall spread is largest; project onto it and the two classes land almost entirely on top of each other. The axis that actually tells them apart was the second one, which PCA ranks lower because it happens to carry less total variance — variance PCA has no way to know is uninformative.

Linear discriminant analysis exists to ask the question PCA structurally can't: which direction actually separates the classes, not which direction has the most spread.

Think of It Like This

Photographing two flocks of birds from the wrong angle

Two flocks of birds are flying past, each flock tight and compact but the flocks pass each other at a shallow angle, both stretched out lengthwise along nearly the same long axis. Photograph them from directly above, along the direction where the scene stretches most, and the two flocks' long shapes overlap almost entirely in the frame — from that angle they look like one smeared blur.

Rotate the camera ninety degrees to shoot across the flocks instead of along them, and each flock collapses into a tight, separated dot — because that's the direction where each flock is narrow and the gap between the flocks is wide. The first angle captured where the scene spreads out. The second angle captured where the two groups are actually different. LDA always aims the camera the second way.

How It Actually Works

Two kinds of spread, and a ratio between them

LDA needs two quantities computed from the labeled data. Within-class scatter measures how spread out each class is around its own mean, summed across classes — small within-class scatter means each class is tight and predictable. Between-class scatter measures how far apart the class means sit from each other, and from the overall mean — large between-class scatter means the classes sit in different places. LDA chooses the projection direction that maximizes the ratio of between-class scatter to within-class scatter: tight classes that are far apart, which is exactly the separation an actual classifier needs, projected down to however many dimensions you keep.

For two classes, the direction reduces to a clean form: the within-class scatter matrix, inverted, applied to the difference between the two class means. That single vector is the axis PCA has no mechanism to find, because computing it requires the labels PCA never looks at.

Why the two methods can point in genuinely different directions

PCA's ranking depends only on the marginal spread of the data — it would give the identical answer whether or not you'd labeled a single point. LDA's ranking depends entirely on the labels, specifically on where the class means sit relative to the spread within each class. Two variables can be strongly correlated with each other (which drives PCA's ranking up) while being nearly uninformative about class membership, and a variable barely correlated with anything else in the data can be the single best discriminator — a small within-class scatter and a large gap between means, buried in a direction PCA ranked far down the list because that direction happened to carry little overall variance.

The ceiling: at most (classes − 1) informative dimensions

Between-class scatter, for $C$ classes, has rank at most $C - 1$ : with two classes there is exactly one meaningful discriminant direction, since two class means define one line between them, no matter how many original features you started with. Ask for more discriminant dimensions than $C - 1$ and the extra ones carry no separating information at all — a firm ceiling PCA doesn't share, since PCA's component count is bounded only by the number of original features.

Show Me the Code

Two classes separated along an axis that carries almost none of the data's overall spread. PCA's top direction misses it entirely; LDA finds it directly.

1import numpy as np2
3rng = np.random.default_rng(2)4n = 3005# two classes separated along a direction that is NOT the direction of greatest overall spread6class0 = rng.normal([0.0, 0.0], [3.0, 0.3], (n, 2))7class1 = rng.normal([0.0, 2.0], [3.0, 0.3], (n, 2))8x = np.vstack([class0, class1])9y = np.r_[np.zeros(n), np.ones(n)]10
11def pca_direction(x: np.ndarray) -> np.ndarray:12    xc = x - x.mean(0)13    _, _, vt = np.linalg.svd(xc, full_matrices=False)14    return vt[0]15
16def lda_direction(x: np.ndarray, y: np.ndarray) -> np.ndarray:17    mean0, mean1 = x[y == 0].mean(0), x[y == 1].mean(0)18    within_class_scatter = np.cov(x[y == 0].T) + np.cov(x[y == 1].T)19    return np.linalg.solve(within_class_scatter, mean1 - mean0)20
21def separation(direction: np.ndarray) -> float:22    proj = x @ (direction / np.linalg.norm(direction))23    return float(abs(proj[y == 1].mean() - proj[y == 0].mean()) / proj.std())24
25pca_dir, lda_dir = pca_direction(x), lda_direction(x, y)26print(f"PCA's top direction   {np.round(pca_dir, 2)}   separation: {separation(pca_dir):.2f}")27print(f"LDA's chosen direction {np.round(lda_dir / np.linalg.norm(lda_dir), 2)}   separation: {separation(lda_dir):.2f}")28# -> PCA's top direction   [-1.   -0.01]   separation: 0.0229# -> LDA's chosen direction [-0.  1.]   separation: 1.92

PCA's top axis is almost exactly the first coordinate — where the data happens to be widest — and projecting onto it gives a separation score near zero: the classes land on top of each other. LDA's axis is almost exactly the second coordinate, the one PCA had no reason to prefer, and the separation score there is nearly a hundred times larger.

Watch Out For

Using LDA as a general-purpose dimensionality reduction step

LDA optimizes specifically for linear separability between the classes it's given, which makes it excellent preprocessing for a linear classifier and a poor choice whenever the downstream task isn't that classification — visualization of structure you haven't labeled yet, or feeding a different unsupervised method later. Reach for PCA when the goal is capturing spread rather than separating known classes.

Expecting more than (classes − 1) useful dimensions

Asking an LDA implementation for more components than one less than the class count either errors or silently returns dimensions carrying no separating information, because between-class scatter's rank caps there exactly. With only two classes, exactly one discriminant axis exists no matter how many original features were measured — treat additional requested dimensions as a configuration mistake, not a modeling choice.

The Quick Version

LDA finds the projection maximizing the ratio of between-class scatter to within-class scatter — classes far apart, each one tight — using the labels PCA never sees.
PCA ranks directions by overall spread regardless of class; LDA ranks them by class separability, and the two rankings can disagree sharply.
A feature barely correlated with anything else can be the strongest discriminator, buried where PCA ranks it low because it carries little total variance.
At most (number of classes − 1) discriminant directions carry any separating information; the rest are structurally uninformative.
LDA optimizes specifically for linear separability of known classes — a poor fit whenever the actual downstream goal isn't that classification.