Feature Engineering

Why Does This Exist?

Two teams take the same forecasting problem. Same gradient-boosted trees, same rows, same tuning budget. One lands at 12% error and the other at 4%. Nothing about the model differed. One team handed it a raw timestamp column; the other handed it hours_since_previous_order, orders_in_prior_7_days, and is_public_holiday.

That gap is feature engineering, and on tabular problems it is routinely wider than the gap between two algorithms.

The reason it exists at all comes down to what a model actually consumes. A model is a function fitted to a fixed set of columns. It does not read your database, infer your business, or notice that two of your columns should be divided by each other. It sees a grid: one row per thing you want to predict about, one column per number you decided to show it. That grid is the design matrix, usually written $X$, and it is the model's entire universe. A pattern that isn't expressed in $X$ cannot be learned from $X$.

So "which model should I use?" is downstream of a bigger question: what does the model get to look at?

The vocabulary, in one paragraph

A feature is one input column — distance_km, is_weekend, word_count. A row is one thing you're predicting about. The target is what you're predicting. Feature engineering is everything that turns your records into that grid: making columns that didn't exist, turning non-numbers into numbers, putting numbers on a comparable footing, and deleting what hurts.

Think of It Like This

How It Actually Works

Four operations, in this order, because each one assumes the last.

1. Construct — create columns that don't exist yet. Ratios, differences, date parts, rolling aggregates, lags. This is where domain knowledge enters, and it has the most headroom.
2. Encode — turn non-numbers into numbers. Categories, text, timestamps, identifiers. Every model consumes floats, so something has to do the conversion, and how decides what the model can infer.
3. Scale — put numeric columns on a comparable footing, and reshape distributions that break a model's assumptions.
4. Select — remove columns that add noise, leak, cost money, or duplicate each other. Fewer features often means a better model, which surprises people.

Every one of those four steps learns from data

This matters more than any individual technique, and it's why this page points at leakage before it points at anything else.

A scaler learns a mean and a standard deviation. An imputer learns a median. A one-hot encoder learns the list of categories. A target encoder learns a per-category average of the thing you're trying to predict. None of these are pure functions — each is fitted, and what it gets fitted to is a choice you make.

Fit them on all your data and you've moved information out of your test set and into your training set. Your validation score improves. Production doesn't. That's data leakage, and preprocessing causes it far more often than anything exotic does.

The rule: fit on training rows, apply everywhere. Every fitted transform, every time, including inside every cross-validation fold.

What the model can and can't work out for itself

Model families need different amounts of help, and knowing which is which saves a lot of pointless work.

• Linear models can't discover interactions or curvature at all. If the effect is price / square_metres, you have to build that column. They also need scaled inputs, because a coefficient's size is entangled with its column's units.
• Trees and gradient boosting get monotone transforms for free — a split on x > 3 and a split on log(x) > 1.1 partition the rows identically, so scaling changes nothing. What they can't do cheaply is arithmetic across columns. A ratio costs you one line and costs them a staircase of dozens of splits, approximated rather than exact.
• Neural networks learn features, given enough data and scaled inputs. That's why feature engineering feels dead in vision and text, where the encoder does it, and very much alive on tabular data, where you rarely have the volume to earn it.

Feature engineering didn't die with deep learning. On raw modalities it moved into curation and representation, which is what most of the rest of this band is about.

Worked example

Predicting a health outcome from weight_kg and height_m. Take two people: 95kg at 1.62m, and 61kg at 1.80m.

On the raw columns they're awkward to separate, because the heavier person is also the shorter one — any rule using weight alone gets the light-and-tall case wrong. Body mass index is $\text{weight} / \text{height}^2$:

One column, one line of code, and a single threshold now separates them cleanly. A gradient-boosted tree given only weight and height would need many axis-aligned splits to trace that curve, and it would still only approximate it. You handed it the exact answer instead.

That's the shape of the whole discipline: you know something about the world that the loss function can't discover from the columns in front of it.

Show Me the Code

Two things worth having in muscle memory — constructing a column, and fitting a transform on training rows only.

1import numpy as np2
3weight_kg = np.array([95.0, 61.0, 88.0, 54.0])4height_m = np.array([1.62, 1.80, 1.91, 1.55])5
6
7def build_bmi(weight: np.ndarray, height: np.ndarray) -> np.ndarray:8    """One constructed column. A tree needs a staircase of splits to fake it."""9    return weight / height**210
11
12def fit_scaler(train: np.ndarray) -> tuple[float, float]:13    """Statistics come from the TRAINING rows only. That is the whole contract."""14    return float(train.mean()), float(train.std())15
16bmi = build_bmi(weight_kg, height_m)17print(np.round(bmi, 1))                    # -> [36.2 18.8 24.1 22.5]18
19mean, std = fit_scaler(bmi[:2])            # fitted on train, applied to everything20print(np.round((bmi - mean) / std, 2))     # -> [ 1.   -1.   -0.39 -0.58]

fit_scaler receives bmi[:2] and nothing else. That deliberate narrowing is the difference between a validation number you can trust and one you can't, and it's why scikit-learn's Pipeline exists — it makes the narrowing structural instead of something you have to remember.

Watch Out For

The Quick Version

• The model only ever sees the design matrix you built. A pattern not expressed in the columns cannot be learned from them.
• Four operations, in order: construct, encode, scale, select.
• On tabular problems, the gap between good and bad features is usually wider than the gap between two algorithms.
• All four steps are fitted — each learns a statistic — so all four can leak. Fit on training rows, apply everywhere.
• Trees ignore monotone scaling but can't do arithmetic across columns. Linear models can't find interactions. Neural nets learn features, given volume.
• One good column replaces dozens of splits: $\text{weight}/\text{height}^2$ instead of a staircase.
• Fewer features frequently beats more.
• For every column, ask whether its value was knowable at prediction time. If not, it's a leak wearing a useful name.

What to Read Next

• Exploratory Data Analysis comes before any of this — you can't engineer features for data you haven't looked at.
• Feature Construction is step one in depth: interactions, ratios, date parts, lags, aggregations.
• Categorical Encoding is step two, where cardinality picks the method.
• Feature Scaling is step three, including which models genuinely don't care.
• Feature Selection is step four, and why importance is not causation.
• Data Leakage is the failure every step above can cause. Read it early rather than late.
• Definitions worth a look: Feature, Feature Matrix, Data Leakage, and Ground Truth.