Data Concepts

Feature Selection

Three families of method, and which family you pick matters much less than whether the choosing happened inside the fold or on the whole dataset first.

Three routes narrow a wide column set, each blind to something different: filters miss redundancy, wrappers overfit their own search, and embedded selection falls out of the fit

Why Does This Exist?

Forty-two columns go into the model. Nine do the work. The rest add variance, cost money at serving time, and give you thirty-three more pipelines that can break at three in the morning.

So you cut. Three families of method do it, they differ in cost and in what they can see, and choosing between them is what everyone argues about. It's also what matters least.

Here's the part that matters: selection is a fitted step. It reads your labels and decides from them, exactly like a target encoder. Run it once on the whole dataset and then cross-validate, and your score is measuring a choice that already saw the held-out rows. Which family you used stops mattering.

Think of It Like This

Picking five players out of a hundred trialists

Time everyone's sprint and keep the fastest. Cheap, done in an afternoon. It also gets you two identical wingers and no goalkeeper, because a stopwatch can't see how players fit together.

Or play real matches with different line-ups. That answers the actual question, and it takes weeks. Play enough and some line-up wins on a lucky bounce.

Or hand all hundred to a coach who names a formation. Positions get filled, players who fit none stop being picked, and selection came free with a decision you were making anyway.

Then the part that decides whether any of it worked. Pick the squad after watching them win the final and they'll look like the right squad. They always do.

How It Actually Works

The three families

Filter methods score each column against the target on its own: variance threshold, correlation, chi-squared, ANOVA F, mutual information. One pass, no model, cheap on 10,000 columns. Blind to exactly two things. Redundancy: two perfectly correlated columns both score highly, so you keep both. Interaction: a column that only helps alongside another scores zero.

Wrapper methods train the model on candidate subsets and compare. Forward selection adds a column at a time, backward elimination removes, recursive feature elimination refits and drops the weakest each round. Closest to the objective, and the most expensive — a forward search over 40 columns is hundreds of fits. It overfits its own search too, since you're picking among subsets by validation score (multiple comparisons).

Embedded methods get selection out of a fit you wanted anyway. An L1 penalty on the sum of absolute coefficients drives them to exactly zero rather than merely small, so lasso hands you a subset. Tree ensembles report split-based importances, gradient boosting reports total gain. One fit, no outer loop.

Importance is not causation

Impurity-based tree importance is biased toward high-cardinality and continuous columns: more distinct values means more split points to try, so more chances to look useful, and a hashed customer_id can land in your top five. Permutation importance is the honest version — shuffle a column in held-out data and see how far the score falls.

Correlated columns split their credit, so two near-duplicates each take about half and both look weak. None of it is causal either: high importance means the model leans on a column, which also happens when the column is a consequence of the target.

What fewer columns costs

Fewer columns means less variance, cheaper serving, fewer dependencies to monitor, and fewer chances of leakage through a column nobody reviewed. The cost gets skipped, though. Every removed column is one you stop collecting: instrumentation goes, the ETL step gets deleted, the vendor feed lapses. Wanting it back in six months means six months of history you don't have.

Worked example

Four hundred rows, three columns. Column 0 is real signal, column 1 an exact copy, column 2 noise. The target is twice the signal plus a little noise.

A correlation filter scores them 0.974, 0.974 and 0.081, so it drops the noise and keeps both duplicates. It never compared the columns to each other.

L1 at a penalty of 0.2 returns 1.82, 0 and 0, keeping one duplicate and zeroing the other. Not because it noticed they were identical: once the first absorbed the signal, the correlation left in the residual was exactly the penalty, so soft thresholding zeroed the second. Column order decides which one survives.

Show Me the Code

1import numpy as np2
3rng = np.random.default_rng(0)4signal = rng.normal(size=400)5X = np.column_stack([signal, signal, rng.normal(size=400)])  # column 1 copies column 06y = 2.0 * signal + rng.normal(0.0, 0.5, 400)7
8
9def lasso(X: np.ndarray, y: np.ndarray, lam: float, steps: int = 200) -> np.ndarray:10    """Coordinate descent with soft thresholding, which is how L1 reaches exactly zero."""11    w = np.zeros(X.shape[1])12    for _ in range(steps):13        for j in range(X.shape[1]):14            rho = X[:, j] @ (y - X @ w + X[:, j] * w[j]) / len(y)15            w[j] = np.sign(rho) * max(abs(rho) - lam, 0.0) * len(y) / (X[:, j] @ X[:, j])16    return w17
18scores = [abs(np.corrcoef(X[:, j], y)[0, 1]) for j in range(3)]  # each column, alone19print(np.round(scores, 3))            # -> [0.974 0.974 0.081]20print(np.round(lasso(X, y, 0.2), 3))  # -> [1.82 0.   0.  ]

max(abs(rho) - lam, 0.0) is the whole difference. Ridge shrinks a coefficient toward zero and leaves it non-zero forever. L1 clips it flat.

Watch Out For

Selecting on the whole dataset, then cross-validating

It reads innocently. SelectKBest(k=10).fit(X, y), then cross_val_score on the ten survivors. Two lines, each correct, wrong in that order.

Here's the version that settles it. 200 columns of pure standard-normal noise, 100 rows, coin-flip labels, so there's nothing to learn. Pick the top ten by ANOVA F across every row, then five-fold cross-validate a logistic regression on them: 77% accuracy. Move the selector inside the pipeline so each fold picks its own ten and it falls to 48%, the coin flip you started with.

No fold disagrees, because every fold's held-out rows helped choose the columns. The cross-validation is confirming a decision it can't question. Selection goes inside the Pipeline, and tuning k needs its own inner loop (cross-validation schemes).

Trusting default tree importances and dropping a partner column

model.feature_importances_, sorted, cut at the top twenty. Fast, and wrong in two directions at once.

The ranking favours columns with many distinct values, so continuous ones float up while a predictive binary flag sinks. And correlated columns split their credit, so a pair carrying the signal shows up as two mediocre features.

The other half is interactions. is_premium can carry no signal alone and everything alongside discount_pct, so a filter scores it zero. Drop it and the surviving column stops working too.

Use permutation importance on held-out folds, group correlated columns before ranking, and confirm any removal by refitting without it. When the pool is too large for that, you have a generator problem (automated feature engineering).

The Quick Version

Three families. Filters score each column alone: fast, blind to redundancy and interaction. Wrappers train per subset: accurate, expensive, overfits its own search. Embedded gets selection free from the fit.
The family matters less than the placement. Selection is fitted, so it belongs inside the fold.
200 noise columns, 100 rows, random labels: selecting first gives 77% cross-validated accuracy. Selecting inside the pipeline gives 48%.
Impurity importance is biased toward high-cardinality and continuous columns, and it isn't causal. Permutation importance on held-out data is the honest version, and correlated columns split their credit so both look weak.
The filter keeps both duplicates at 0.974 each. L1 keeps one at 1.82 and zeroes the other, and column order decides which.
Fewer columns means less variance and cheaper serving. It also means you stop collecting the data, which is hard to undo.