Data Concepts

Data Drift and Distribution Shift

Three different problems get called drift. The inputs moved, the class balance moved, or the right answer moved, and only the last one needs a new model.

The production distribution has flattened against the training reference, and one bin moving from a tenth of the traffic to a quarter contributes most of the drift score on its own

Why Does This Exist?

Six months after launch, a model that was worth money isn't. The team's monitoring shows eleven features "drifting", and nobody can say which of them matters.

The word is doing too much work. Three genuinely different things get called drift, they need three different responses, and one of them is invisible to the monitoring most teams have.

The three, stated precisely

Write $X$ for the inputs and $Y$ for the thing you're predicting. $P(X)$ is how often each kind of input shows up; $P(Y \mid X)$ is the answer given an input.

Covariate shift — $P(X)$ moves, $P(Y \mid X)$ holds. A new marketing channel brings younger customers. The relationship you learned is still true, so the model is still right where it has data, and may be extrapolating where it doesn't. Often needs nothing at all.

Label shift (also prior shift) — the class balance moves while $P(X \mid Y)$ holds. Fraud rates rise in a recession: fraudsters look the same, there are just more of them. Fixed by recalibrating or moving the decision threshold, not by retraining features.

Concept drift — $P(Y \mid X)$ itself changes. The same input now has a different correct answer, because spammers adapted and what used to be a spam signal is now what everyone writes. No amount of reweighting helps, because what you learned is no longer true.

That third one is the expensive case, and no amount of input monitoring detects it. The inputs can be perfectly stable while the answer changes underneath them.

Think of It Like This

A doctor who moved practice

A GP spends fifteen years in one town and gets very good at it.

Then the town's demographics change — a university opens, and the average patient is 22 instead of 50. Different patients, same medicine. She's still an excellent doctor, and she sees fewer of the conditions she knows best. That's covariate shift.

Or a flu season hits, and the same symptoms now much more often mean flu. Nothing about how flu presents has changed; the base rate has. She should lower her threshold for calling it flu. That's label shift.

Or a new pathogen arrives that presents exactly like flu and needs different treatment. Now the same symptoms mean something else. Fifteen years of pattern recognition is actively wrong, and no adjustment to how often she says "flu" fixes it. That's concept drift, and the only way she finds out is by seeing outcomes — which arrive weeks later.

How It Actually Works

The shapes drift takes

Sudden — a competitor exits, a pricing change ships, a data source is swapped. Gradual — behaviour moves over months. Incremental — small steps, each below your alert threshold, adding up. Recurring — seasonality, which is the one that causes an annual false alarm because December genuinely looks nothing like November and it isn't drift.

Detecting input drift, without labels

This part is cheap, and it's what most monitoring does.

Per feature, compare a production window against a reference window, where the reference should be your training data and not last week. Usable tests: a population stability index over fixed bins, a Kolmogorov-Smirnov test for continuous columns, a chi-squared test on categorical bins, or a divergence measure like KL.

Better than any of them, and underused: train a domain classifier to tell a training row from a production row. If it can't — AUC near 0.5 — nothing has moved. If it hits 0.85, something has, and its feature importances tell you exactly which columns. One model instead of two hundred tests.

Concept drift needs labels, and labels are late

There's no way around this. To know whether $P(Y \mid X)$ changed you need to know $Y$ , and $Y$ arrives when it arrives: a churn label takes 90 days, a loan default takes years, a support-resolution label takes a week. By the time you can measure the thing that actually matters, a quarter has gone.

So monitor proxies in the meantime. The prediction distribution (if the model's output histogram moves and the inputs didn't, something is off), the confidence distribution (mass drifting toward the middle means the model is less sure), and the unseen-category rate, which is a direct signal that the input space is expanding.

Fixing each one

Retrain on recent data; weight recent data more heavily; importance-weight training rows to match the production input distribution, which is the principled correction for covariate shift; recalibrate or move the threshold for label shift; and for concept drift, detect and adapt — a rolling retrain, or an explicit change-point detector feeding a retrain trigger.

The monitoring discipline nobody follows

Set the threshold in advance, per feature, against your training reference. Not "watch the dashboard".

And correct for multiple comparisons, or your alerting is worthless. Two hundred features tested daily at $p < 0.05$ gives you ten false alarms a day by construction — see multiple comparisons. Within a fortnight nobody reads the channel. A drift alerting system that cried wolf is worse than none, because it consumed the attention budget that a real alert needed.

Worked example

One feature, binned four ways. In training the bins hold 40%, 30%, 20% and 10% of the rows. This week they hold 25% each.

The population stability index sums $(\text{actual} - \text{expected}) \times \ln(\text{actual}/\text{expected})$ over the bins:

\text{PSI} = \sum_i (a_i - e_i)\,\ln\!\frac{a_i}{e_i}

Bin by bin that's 0.0705, 0.0091, 0.0112 and 0.1374, summing to 0.228. Against the usual reading — under 0.1 quiet, 0.1 to 0.25 worth investigating, over 0.25 a real shift — that's a "look at it" rather than an emergency.

Notice where it came from. The fourth bin went from 10% to 25% and contributed 0.137, which is 60% of the total from one bin. PSI is dominated by proportional change in small bins, which is usually what you want and occasionally means a rare category tripling is drowning out a genuine move in your main bin.

Show Me the Code

1import numpy as np2
3reference: np.ndarray = np.array([0.40, 0.30, 0.20, 0.10])  # the training window, four bins4current: np.ndarray = np.array([0.25, 0.25, 0.25, 0.25])    # this week, same bin edges5shifted: np.ndarray = np.array([0.20, 0.20, 0.25, 0.35])    # a bigger move into the tail6
7def psi(expected: np.ndarray, actual: np.ndarray) -> float:8    """Population stability index. Under 0.1 is quiet, over 0.25 is a real shift."""9    return float(((actual - expected) * np.log(actual / expected)).sum())10
11print(np.round(current - reference, 2).tolist())  # -> [-0.15, -0.05, 0.05, 0.15]12print(round(psi(reference, current), 4))          # -> 0.2282  worth investigating13print(round(psi(reference, shifted), 4))          # -> 0.5035  a real shift14print(round(psi(reference, reference), 4))        # -> 0.0     nothing moved

The last line is the sanity check worth keeping in a test: a reference compared with itself must score exactly zero.

Watch Out For

Alerting per feature with no correction

Two hundred features, a statistical test each, run nightly at $p < 0.05$ . That's an expected ten alerts every night from a completely stable system, because that's what a 5% false-positive rate means when you run it two hundred times.

Week one, people investigate. Week two, they skim. Week three, there's a filter rule in the mail client. Then a real shift arrives in feature 47 and lands in a folder nobody opens.

Three fixes, and use all of them. Correct for the number of tests, or set thresholds from an effect size like PSI rather than from a p-value, since a p-value shrinks with sample size and will eventually flag a shift too small to matter. Rank alerts by the feature's importance in the model, because drift in your 190th feature is not news. And route one daily summary rather than 200 independent alerts.

Retraining because the inputs moved, and missing the drift that matters

An alert fires, someone retrains, the score comes back a little better, and it feels like the process worked.

Often it didn't. Under pure covariate shift with a stable relationship, the model was already correct — you spent a retrain cycle and a review to move a number that would have moved anyway. Meanwhile the seasonal case is worse: retraining on December to fix "December drift" produces a model that's wrong in January.

And the drift that ends models is invisible from here. Concept drift changes $P(Y \mid X)$ while leaving $P(X)$ untouched, so every input monitor stays green while the model gets steadily wronger. Input monitoring cannot see it, by construction.

So do two things. Before retraining, ask whether the relationship changed or only the input mix, and check the model's error on recent labelled rows rather than its input histograms. And build a labelled feedback path — even a small, slow, sampled one — because that's the only instrument that sees the expensive case.

The Quick Version

Three problems share one word. Covariate shift moves $P(X)$ , label shift moves the class balance, concept drift moves $P(Y \mid X)$ .
Covariate shift often needs nothing. Label shift needs recalibration or a new threshold. Concept drift needs a new model.
Sudden, gradual, incremental, recurring. Seasonality mislabelled as drift is an annual false alarm.
Input drift is detectable without labels: PSI, KS, chi-squared, or a divergence, against your training reference.
A domain classifier is the best single instrument. AUC near 0.5 means nothing moved; its importances name the columns.
Concept drift needs labels, and labels are late. Monitor prediction distribution, confidence and unseen-category rate meanwhile.
PSI over four bins going from 40/30/20/10 to 25/25/25/25 is 0.228, and 60% of it came from the smallest bin.
Set thresholds in advance, per feature, and correct for the number of tests. 200 tests at p < 0.05 is 10 false alarms a night.
An alerting channel people have learned to ignore is worse than no channel.