Core ML

Latency Budgets

You can't just build an ML model and hope it's fast enough. You must establish a strict time limit (a budget) that the model is allowed to take before the user gets frustrated and leaves your app.

A typical 500ms Latency Budget isn't just for the ML Model. The network transfer takes 100ms, the API routing takes 50ms, leaving only 350ms for the actual GPU inference.

Why Does This Exist?

In academia, researchers only care about accuracy. If a model takes 5 seconds to process an image, they don't care, as long as it gets the right answer.

In production engineering, speed is arguably more important than accuracy. If you are building a self-driving car, a model that detects a pedestrian with 99.9% accuracy but takes 2 seconds to run is useless—the car has already crashed. If you are building an e-commerce recommendation system, Amazon found that every 100 milliseconds of latency costs them 1% in sales.

To prevent this, engineers establish a Latency Budget before they even start training the model. This is a strict, mathematical limit on exactly how many milliseconds the entire system is allowed to take to return an answer. If a proposed model architecture exceeds the budget, it is rejected, regardless of how accurate it is.

Think of It Like This

Imagine you are a Formula 1 pit crew. Your goal is to change all four tires. You do not just say "Let's change them as fast as we can."

You establish a strict Budget of 2.5 seconds. You then allocate that budget: 0.5 seconds to jack up the car, 1.0 seconds to remove the old tires, 0.5 seconds to attach the new ones, and 0.5 seconds to drop the car. If the team taking off the tires takes 1.5 seconds, they have "blown their budget," and the car will lose the race.

How to Set a Latency Budget

A latency budget is usually defined by the product team based on human psychology.

< 100ms: Feels instantaneous. Required for self-driving cars, high-frequency trading, and video game AI.
100ms - 300ms: A slight, perceptible delay, but acceptable for most web actions (e.g., clicking a button and getting a result).
1,000ms (1 second): The user's flow of thought is interrupted. They feel like they are "waiting."
> 10 seconds: The user assumes the app is broken and abandons the page.

The Breakdown

If your product team says the feature must feel "instantaneous" (100ms budget), you do not get 100ms for your ML model! You must subtract the overhead.

Network Latency (40ms): It takes time for the signal to travel from the user's phone to your AWS server.
API Gateway (10ms): Your web server takes time to route the request and validate the JSON.
Data Pre-processing (15ms): Resizing the image or tokenizing the text.
The ML Model Budget (35ms): This is all you have left. If your model takes 50ms, you have blown the budget.

P99 vs Average Latency

When measuring if you are meeting your budget, you never look at the "Average" (Mean) time. If your average is 90ms, but 10% of your users experience 5-second delays due to network spikes, your average will still look great, but thousands of users will be furious.

Instead, you measure the P99 Latency. P99 means "The 99th percentile." If your P99 latency is 120ms, it means exactly 99% of your requests are faster than 120ms, and only 1% are slower. Engineering teams typically strive to ensure their P99 fits within the Latency Budget.

LLMs: Time to First Token (TTFT)

Large Language Models (LLMs) break traditional latency rules. Because they stream their answers one word at a time, users are willing to wait longer for the entire answer, as long as they see the first word quickly.

For LLMs, the most critical budget is Time to First Token (TTFT). If the TTFT is under 500ms, the user feels like the AI is "typing," and they will happily watch it type for 10 seconds. If the TTFT is 3 seconds, they will assume it is broken.

Watch Out For

The Batching Tradeoff. In dynamic-batching, we learned that waiting a few milliseconds to collect more requests drastically increases GPU efficiency. This is a direct conflict with your Latency Budget! If your budget is 100ms, and your model takes 40ms, you can set your dynamic batching timeout to 60ms. You are intentionally eating up your remaining budget to save money on compute costs.

The Quick Version

A Latency Budget is the strict maximum time a system is allowed to take to return a prediction.
ML Models only get a fraction of the budget; the rest is consumed by network routing and data preprocessing.
Engineers measure success using P99 Latency, ensuring that 99% of users experience fast load times.
For Large Language Models, the most important metric is Time to First Token (TTFT).