Core ML

Online Evaluation

Offline evals tell you how your model performs on a fixed test set. Online evaluation tells you how it's performing right now, on real user queries — which are messier, weirder, and more important than any test set you'll ever write.

Why Does This Exist?

You ran your eval suite before deploying. Every benchmark passed. Then you shipped, and three days later, support tickets started rolling in: the chatbot was giving vague non-answers, hallucinating company policies, and randomly switching languages mid-response.

Your eval set didn't have those queries because nobody anticipated them. Real users are creative in ways that benchmark datasets aren't. Online evaluation is how you find out about this kind of quality regression before users do.

Think of It Like This

A restaurant kitchen does test-cooking before opening. But the real signal comes from watching actual diners. A great restaurant samples a small number of completed dishes throughout the night — a manager tastes one from table 14, another from table 27. If five in a row are too salty, the chef knows before the review hits Yelp. Online evaluation is the manager's taste-testing, running continuously on your production traffic.

How It Actually Works

You cannot score every production request — the cost would be prohibitive, and the latency of waiting for an evaluator would make it useless for real-time decisions. Instead, you sample: score roughly 1–10% of traffic, chosen randomly or stratified by query type.

Each sampled request-response pair is passed to a scorer. Two types of scorers dominate production systems:

LLM-as-judge — send the original query, the LLM's response, and a rubric to a separate evaluation model (often a large capable one like GPT-4o). Ask it to rate the response on dimensions like helpfulness, accuracy, conciseness, and format compliance. This is flexible and catches nuanced quality problems, but costs more and adds latency to your eval pipeline.

Heuristics — fast, cheap, and deterministic. Check that the response is a valid JSON if it should be; check that it's within an expected length range; check for specific required phrases ("I don't know" for uncertain queries, etc.). Heuristics miss subtle quality issues but catch hard failures instantly.

Layer both: heuristics run on every sampled request in milliseconds; LLM judging runs on a smaller subset (0.5–1%) over longer windows.

Show Me the Code

1import random2from openai import OpenAI3
4client = OpenAI()5
6def maybe_evaluate(query: str, response: str, sample_rate: float = 0.05) -> None:7    """Score ~5% of responses asynchronously."""8    if random.random() > sample_rate:9        return10
11    # Heuristic checks (instant)12    if len(response) < 10:13        record_score("online_eval", query, response, score=1, reason="too_short")14        return15
16    # LLM-as-judge (async, background job)17    judge_prompt = f"""Rate the following AI response on a scale from 1 to 5.18Query: {query}19Response: {response}20
21Criteria:22- 5: Accurate, helpful, appropriately concise23- 3: Mostly correct but vague or incomplete24- 1: Wrong, harmful, or completely off-topic25
26Reply with a JSON object: {{"score": <1-5>, "reason": "<one sentence>"}}"""27
28    result = client.chat.completions.create(29        model="gpt-4o-2024-11-20",30        messages=[{"role": "user", "content": judge_prompt}],31        response_format={"type": "json_object"},32    )33    parsed = json.loads(result.choices[0].message.content)34    record_score("online_eval", query, response, **parsed)

The record_score function writes to your metrics store, where a dashboard aggregates rolling mean scores and triggers alerts when they drop below threshold.

The Alert Threshold

Set two thresholds:

Warning (e.g., mean score < 4.0 for 5 minutes): page the on-call engineer to investigate. Could be a prompt regression, could be a new category of user queries the model handles poorly.
Critical (e.g., mean score < 3.5 sustained 10 minutes): trigger automatic rollback to the previous prompt version.

The 10-minute window prevents false alarms from random variance in a small sample. A single bad response doesn't mean the model is broken.

Watch Out For

Judge bias corrupting your quality signal. LLM judges have known biases: they favour longer responses, prefer responses that agree with the query's framing, and vary in their scoring based on prompt wording. If you change the judge prompt without running it against historical data, your quality score will appear to jump or crash — not because model quality changed, but because your ruler changed. Treat the judge prompt as a versioned artefact (see prompt-and-model-versioning) and run overlap evaluations when updating it.

The Quick Version

Offline evals measure a fixed test set. Online evaluation measures real production traffic.
Sample 1–10% of requests — scoring everything is too expensive and adds latency.
Use fast heuristics on every sample, and LLM judging on a smaller subset for nuanced quality scoring.
Set alert thresholds on rolling mean score; trigger rollback when quality falls critically.
Version your judge prompt — changing it changes your measurement ruler, not just your measurements.