Data Concepts

Online / Offline Skew

The model is fine and the data is fine. The same input still produces two different feature vectors, depending on which of your two code paths computed it.

One raw event goes down two code paths and comes out as two different feature vectors, so the only way to find the gap is to compute both and diff them

Why Does This Exist?

The model validated at 0.88 and it's running at 0.79. The weights are right, the input distribution hasn't moved, nothing is null, and no alert has fired.

Here's what happened. The training features were computed in SQL against the warehouse. The serving features were rewritten in the API's language six weeks later, by someone else, from the same ticket. One of them treats a missing rating as zero and the other substitutes the global average. One closes a 30-day window exclusively and the other inclusively.

The same input now produces two different feature vectors. The model never sees a bug, because from its point of view it's just being asked about a slightly different customer than the one it was trained to recognise.

I'd call this the most common silent production failure in applied ML, and the reason is structural: it has no error, no null, no shape mismatch, and the only metric that catches it is one almost nobody computes.

Think of It Like This

A recipe scaled by two different cooks

A chef develops a dish and writes it down: 200g flour, 4g salt, 30 minutes.

Someone in the second kitchen scales it for forty covers. They multiply everything by ten, except salt is measured there in teaspoons, so 40g becomes "about seven teaspoons" and seven teaspoons is 42g. The oven runs 15°C hot. Thirty minutes is timed from when the tray goes in rather than from when the oven reaches temperature.

Every one of those is defensible. Nobody made a mistake anyone would write up. And the dish that leaves the second kitchen is not the dish the chef signed off, so the reviews are about a recipe that was never tested.

Nothing in either kitchen is broken. They're just not the same kitchen, and the dish was only ever validated in one of them.

How It Actually Works

Four causes, in rough order of frequency

Two implementations. Features written once for training and again for serving, in a different language, against a different store, by a different person. Everything that can quietly differ does: rounding, null defaults, timezone handling, units, string normalisation, how ties break. This is the actual reason feature stores exist — not caching, but one implementation with two callers.

Different freshness. Training reads a table hours after the fact, so late-arriving events have landed. Serving reads it in the moment, before they have. Same query, same code, different answer — and the training-time value is the one that can't exist at prediction time, which makes it a cousin of data leakage. Streaming is where this gets formalised as watermarks.

Different windows. A nightly batch computes "last 30 days" ending at midnight. Serving computes it ending now. For a customer who ordered this morning, those differ, and the difference is largest for exactly the active customers you care most about.

Transform state. A scaler, encoder or imputer whose parameters get recomputed at serving instead of loaded from the model artefact. Now the same request scores differently depending on which other requests arrived in its batch, which isn't reproducible even in principle. Feature scaling covers this one from the other side.

Skew against leakage

Worth separating cleanly, because the symptom is identical and the fix isn't.

Leakage is training seeing something it shouldn't. Skew is serving computing something differently. Both produce a validation score production never reproduces. Leakage is fixed by changing what the training data contains; skew is fixed by making the two code paths agree.

Diagnose by asking which side is wrong. If the training feature couldn't have been computed at prediction time, that's leakage. If it could have been, but serving computes it another way, that's skew.

The one piece of infrastructure that finds it

Prevention is never complete, so build detection: log the actual feature vector at serving time, replay the same entities through the training pipeline, and diff the two vectors.

That's it. It's unglamorous, most teams don't have it, and it turns a class of invisible bug into a number on a dashboard. Then alert on the diff rate per feature rather than on model accuracy — accuracy tells you weeks later and blames drift, while a diff rate tells you the same afternoon and names the column.

Second-best, if you can't replay: monitor each feature's distribution on both sides and compare. That catches gross mismatches and misses the subtle ones.

Worked example

One order. Three features: amount in pounds, orders in the prior 30 days, and the customer's average rating, where this customer has never rated anything.

The batch path closes the window exclusively and defaults a missing rating to 0.0, giving [42.0, 3, 0.0]. The serving path closes it inclusively — picking up an order placed on the boundary — and defaults a missing rating to the global mean of 3.5, giving [42.0, 4, 3.5].

Two of three columns differ. Push both through the same logistic model with weights [0.01, 0.3, -0.2]: the batch vector scores 0.789 and the serving vector scores 0.715.

Seven points of probability, from one order, on two defaults that each looked reasonable in isolation. Nothing in either path is a bug you could find by reading it alone.

Show Me the Code

1import numpy as np2
3w: np.ndarray = np.array([0.01, 0.3, -0.2])   # amount, prior orders, average rating4
5def score(features: np.ndarray) -> float:6    """The model is identical on both paths. Only its input differs."""7    return float(1 / (1 + np.exp(-(w @ features))))8
9batch: np.ndarray = np.array([42.0, 3.0, 0.0])    # window exclusive, missing rating -> 0.010serving: np.ndarray = np.array([42.0, 4.0, 3.5])  # window inclusive, missing rating -> 3.511
12print((batch != serving).tolist())                # -> [False, True, True]13print(round(score(batch), 3))                     # -> 0.78914print(round(score(serving), 3))                   # -> 0.71515print(round(score(batch) - score(serving), 3))    # -> 0.074

The first line is the whole detection strategy. Everything else is what it costs you not to have run it.

Watch Out For

A feature reimplemented for serving and checked by eye

The ticket says "port orders_prior_30d to the serving path". Someone reads the SQL, writes the equivalent, tests it on a few customers, and the numbers look right.

Unit tests on both sides pass, and they'd pass even if the two disagreed — each test asserts its own path is self-consistent, which is exactly what a skew bug preserves. Reading the two implementations side by side doesn't work either, because the difference is usually a boundary condition or a default that neither file states.

There's only one check that works: run both on the same input and compare the output, on real data, at volume. A few customers won't do it, because the divergence is often conditional — only for customers with a null, only for orders on a window boundary, only across a daylight-saving change.

Make it a test that runs in CI against a sample of production entities, and treat any non-zero diff as a failure rather than as drift.

Transform state recomputed at serving time

StandardScaler().fit_transform(batch) inside the request handler. It looks like the same line that ran in training, and it's a different operation.

Now the scaling depends on the batch. The same customer scores one way when they arrive alongside ordinary customers and another way alongside a whale, because the batch mean moved. The prediction isn't a function of the input any more, which means it isn't reproducible and it isn't debuggable — replaying the request alone gives a third answer.

The categorical version is quieter: an encoder rebuilt from the current batch assigns different integers, so column 4 means "Germany" in training and "Brazil" at serving.

Every fitted parameter is part of the model artefact. Serialise the fitted transformer with the weights, load it, and call transform only. If a fit appears anywhere in your serving code path, that's the bug.

The Quick Version

The model and the data are both fine. Two code paths compute the same feature differently.
No error, no null, no alert, and the only metric that catches it is one most teams don't compute.
Four causes: two implementations, different freshness, different window boundaries, and transform state recomputed at serving.
One implementation with two callers is the fix, and it's the real reason feature stores exist.
Leakage is training seeing what it shouldn't; skew is serving computing differently. Same symptom, different fix.
Log the serving feature vector, replay the entity through the training pipeline, diff them. Alert on the diff rate, not on accuracy.
Unit tests on both paths pass, because each is self-consistent. Only a cross-path diff finds it.
Two defaults and one boundary rule moved a prediction from 0.789 to 0.715.
If fit appears in your serving path, that's the bug.