Core ML

The ML Workflow

Problem, data, model, evaluation, shipping, and monitoring run in a loop, not a line — what monitoring finds reframes the problem, not just the model.

Six stages run in a loop, not a straight line — problem framing, data collection, model training, evaluation, shipping, and monitoring — and monitoring's findings feed back into reframing the problem, not just retraining

Why Does This Exist?

A team builds a churn-prediction model, gets it to 89% accuracy in a notebook, and considers the project done. Three months after shipping it, marketing complains the model flags loyal long-term customers as churn risks constantly, while actual churners slip through unflagged. Nobody changed the model. What changed is everything around it — a new pricing tier launched, a competitor entered the market, and the definition of "a customer likely to churn" quietly shifted underneath a model that has no way of knowing that happened.

The notebook accuracy number was real, and it measured the wrong thing: how well the model fit data from a moment that had already passed by the time anyone was reading dashboards about it. What Is Machine Learning explains fitting a function to examples. The ML workflow is the answer to a harder question: what has to happen before and after that fitting step for the resulting function to still be worth trusting a month later.

Think of It Like This

A restaurant's menu isn't fixed once and forgotten

A restaurant doesn't design a menu once, print it, and never look at it again. It launches with a menu based on a guess about what customers want, watches which dishes actually sell, drops the ones nobody orders, tweaks the ones that sell but get sent back half-finished, and occasionally realizes the whole concept of the restaurant was slightly wrong — not just one dish's seasoning.

That loop — try, watch, adjust, and sometimes rethink the whole premise — never actually ends as long as the restaurant is open. The ML workflow is that same loop, applied to a model instead of a menu: shipping is not the finish line, it's the point where the most important feedback finally starts arriving.

How It Actually Works

The six stages, briefly

Problem framing turns a business question into a prediction task with a target variable, a success metric, and — critically — a decision about what happens with the prediction once it exists. Data collection gathers examples matching that framing, including deciding what counts as a label and how it will be produced going forward, not just for the initial training set. Model training is the step most people mean by "machine learning," and it's genuinely one of the smaller items on this list in terms of time actually spent by a mature team. Evaluation measures performance against the metric chosen in step one, on data the model never trained on — model evaluation covers this in depth. Shipping deploys the model somewhere it actually affects a decision. Monitoring watches the model's real-world behavior after that point, because nothing before shipping tells you what happens once the world starts reacting to the model's own predictions.

Why it's drawn as a loop and not an arrow

A one-way pipeline treats monitoring as the end: ship, watch dashboards, done. What monitoring actually surfaces is frequently a sign that the problem framing itself needs revisiting — not just that the model needs retraining on fresher data. The churn model above didn't need a bigger dataset; it needed someone to notice that "likely to churn" now means something different than it did when the target variable was first defined. That correction belongs in stage one, and the arrow that carries it back there is the entire reason this is drawn as a cycle.

Monitoring is not optional, and it isn't the same job as evaluation

Evaluation happens once, against a held-out set, before shipping. Monitoring happens continuously, against whatever data actually arrives after shipping, and it's watching for a different failure entirely: not "did the model learn the training data well" but "does the training data still resemble what's coming in now." A model that scored perfectly at evaluation time can degrade steadily in production with zero code changes, purely because the input distribution moved — the failure mode monitoring exists to catch before it costs real money, rather than after a dashboard eventually gets checked.

Where time actually goes

Teams that have shipped several models consistently report that data collection and monitoring consume far more of the calendar than model training does. This isn't a sign that training is easy — it's that training is the one stage with the clearest, most well-studied recipe, while data quality and post-deployment drift are the stages with no shortcuts and no fixed endpoint.

Show Me the Code

A minimal monitoring check: comparing a live feature's recent distribution against the distribution it had at training time.

1import numpy as np2
3
4def distribution_shift_flag(train_feature: np.ndarray, live_feature: np.ndarray, threshold: float = 0.5) -> bool:5    """Flag a shift if the live feature's mean has moved more than `threshold`6    standard deviations (measured on the training data) away from the training mean."""7    train_mean, train_std = train_feature.mean(), train_feature.std()8    live_mean = live_feature.mean()9    shift = abs(live_mean - train_mean) / train_std10    return bool(shift > threshold)11
12
13rng = np.random.default_rng(0)14train_feature = rng.normal(50, 5, 1000)  # "average order value" at training time15live_feature_stable = rng.normal(51, 5, 200)  # a few months later, roughly unchanged16live_feature_shifted = rng.normal(62, 5, 200)  # after a pricing change17
18print(f"stable period flagged: {distribution_shift_flag(train_feature, live_feature_stable)}")19print(f"post-pricing-change flagged: {distribution_shift_flag(train_feature, live_feature_shifted)}")20# -> stable period flagged: False21# -> post-pricing-change flagged: True

This check has nothing to do with model accuracy — it never looks at a single prediction. It flags the pricing change purely from the shape of the input data, often before enough fresh labels exist to measure accuracy directly, which is exactly the early-warning role monitoring is meant to play.

Watch Out For

Treating shipping as the finish line

A team that stops paying attention once a model is deployed is betting, implicitly, that the world will stay exactly as it was during data collection. That bet is rarely explicit and almost never true for long. Build the monitoring stage into the project plan and staffing from the start, not as an afterthought added once something visibly breaks.

Retraining on fresh data without re-examining the problem framing

When monitoring flags a degradation, the reflexive fix is "retrain on more recent data." Sometimes that's right. Sometimes the actual issue is that the target variable's real-world meaning shifted — the churn model's real problem wasn't stale data, it was a stale definition of churn — and no amount of fresh data fixes a target that's now measuring the wrong thing. Check whether the problem framing itself still holds before assuming a retrain is the fix.

The Quick Version

The ML workflow runs six stages — problem framing, data collection, model training, evaluation, shipping, monitoring — as a loop, not a one-way pipeline.
Monitoring watches for distribution shift in live data, a different failure mode than evaluation, which only checks performance on a held-out set before shipping.
What monitoring surfaces often belongs back in problem framing, not just in a retraining job.
Data collection and monitoring typically consume more calendar time on a mature team than model training does.