ML System Design Interview Questions

Q: How would you design a recommendation system?

Frame it as a funnel: candidate generation narrows millions of items to a few hundred cheaply, then a ranking model scores those precisely for the user, then re-ranking applies business rules and diversity. Combine collaborative filtering (learn from user-item interactions) with content-based signals to handle cold start, serve it with precomputed embeddings and a fast nearest-neighbor lookup, and close the loop by logging feedback to retrain.

Q: How would you serve an LLM at scale?

Serving an LLM is a throughput and latency problem dominated by GPU memory and the autoregressive decode loop. The core techniques are continuous batching to keep GPUs busy, a KV cache (with paged attention) to avoid recomputing past tokens, and model compression like quantization to fit bigger models on fewer GPUs. Add streaming responses, a semantic cache for repeated queries, autoscaling, and fallbacks, and measure time-to-first-token and tokens-per-second separately.

ML System Design

2 interview questions in this topic. Expand any question to read the full answer.

“How would you design a recommendation system?”

Answer

Start by pinning down the problem, because "recommend" means different things: what are we optimizing (clicks, watch time, purchases, retention), how big is the catalog, and what's the latency budget? Assume a large catalog and tight latency.

The core architectural idea is a funnel, because scoring millions of items per request is impossible under a latency budget:

Candidate generation cheaply narrows millions of items to a few hundred, using approximate nearest-neighbor search over embeddings plus collaborative-filtering co-occurrence. Optimized for recall.
Ranking runs a heavier model over those few hundred, scoring each for this specific user with rich features. Optimized for precision — most of the accuracy comes from here.
Re-ranking applies business rules: diversity, freshness, and filtering already-seen items.

For the models, combine collaborative filtering (learns from the user-item interaction matrix — "people who liked what you liked also liked X," often via matrix factorization or a two-tower model) with content-based filtering (uses item features, so it can recommend brand-new items). Most real systems are hybrid, which is also the answer to cold start: new users fall back to popularity and context; new items lean on their content features.

For serving, precompute item embeddings offline and do an approximate nearest-neighbor lookup plus a fast ranking pass within tens of milliseconds. Then close the loop by logging impressions and clicks to retrain — adding some exploration so the model doesn't collapse into a self-reinforcing bubble. Evaluate offline (precision@k, recall@k, NDCG) to pick models cheaply, but trust an online A/B test on the real business metric for the final call.

A recommendation funnel: millions of items are narrowed by candidate generation to hundreds, scored precisely by a ranking model to dozens, then re-ranked with business rules into the final short list shown to the user.

💡 Note Candidate generation is a librarian who, in one second, pulls a cart of a few hundred books you might like from a warehouse of millions. Ranking is you then reading the jackets of just that cart to pick the ten you'll check out. You could never read every book in the warehouse — the two stages exist so you don't have to.

Answer

Begin with why it's hard, because the constraints explain every design choice:

It's autoregressive. The model emits one token, appends it, and runs again — a 500-token answer is 500 sequential forward passes, so you can't parallelize within one response.
It's memory-bound. Each decode step reads the model's billions of weights from GPU memory, so the GPU often waits on memory bandwidth, not math.
Memory is the ceiling. The weights are tens of gigabytes, and the attention KV cache grows with every token of every concurrent request.

Given that, the high-leverage techniques:

Continuous batching. Swap a finished sequence out and a new request in every step so the GPU never idles — usually the single biggest throughput win.
KV cache + paged attention. Cache the keys and values of past tokens so you don't recompute them; paged attention (vLLM) stores that cache in fixed pages like virtual memory, so you fit far more concurrent requests.
Quantization. Serve weights in 8-bit or 4-bit instead of 16-bit to roughly halve or quarter the memory, fitting the model on cheaper GPUs with minimal quality loss. Tensor/model parallelism splits a model too big for one GPU across several.

On the product side: stream tokens so first words appear in ~200 ms (which is why time-to-first-token and tokens-per-second are tracked separately), add a semantic cache for repeated queries to skip the GPU, put a gateway in front for auth and rate limiting, and autoscale on queue depth with a fallback to a smaller model under overload. Since GPUs dominate the bill, the real objective is usually throughput-per-dollar, not just raw latency.

An LLM serving stack: incoming requests hit a gateway with caching and rate limiting, are grouped by a continuous-batching scheduler, and run on GPU workers that reuse a paged KV cache while streaming tokens back.

💡 Note Serving an LLM is running a busy restaurant kitchen. Continuous batching is a chef who starts the next order the instant a plate goes out, instead of cooking in rigid group seatings. The KV cache is keeping each table's prepped ingredients on the counter so you're not chopping from scratch every course. Quantization is using a more compact pantry so you can cook more dishes at once.

ML System Design

Answer

Related Questions

Answer

Related Questions