Ad Click-Through-Rate (CTR) Prediction System

Advanced

Overview

Ad CTR Prediction is the financial engine of digital advertising. When a user loads a page, an ad auction runs in milliseconds. The system must predict the exact probability that the user will click a specific ad. Because advertisers pay per click (CPC) or per impression (CPM), a tiny improvement in model accuracy (even 0.1%) directly translates to millions of dollars in revenue.

High-level architecture of a Real-Time Ad CTR Prediction System

Functional Requirements

Predict the probability of a user clicking an ad (CTR) given the user, context, and ad.
Calculate the Expected Value (eCPM = Bid * CTR) to rank ads for the auction.
Update the model in near real-time based on live click/no-click feedback.
Support strict latency budgets for the entire ad exchange auction.

Non-Functional Requirements

Extreme latency: the prediction must complete in < 10-20ms.
High throughput: handle millions of ad requests per second globally.
Calibrated probabilities: the output must be a true probability (0.0 to 1.0), not just a relative rank, because financial bids are multiplied by this score.
High availability: the ad server cannot fail, as downtime means instant revenue loss.

Capacity Estimation

Assume a global ad network processing 1 Million ad requests per second.

Scoring Load: If each request evaluates 100 candidate ads, the prediction service handles 100 Million predictions per second.
Data Ingestion: 1M requests/sec generates 1M impressions/sec. If the average CTR is 1%, there are 10,000 clicks/sec. The stream processor must join clicks to impressions within a short window to generate training labels.
Model Size: Because ad systems use massive sparse categorical features (e.g., user_id, device_id, site_id), model embedding tables can reach hundreds of Gigabytes and must be distributed across the RAM of multiple inference servers.

High-Level Architecture

The system is split into the Ad Server (Auctioneer), the CTR Prediction Service, and the Real-time Training Pipeline. When the Ad Server receives an ad slot request, it queries the CTR Prediction Service with candidate ads. The service fetches user/context features from a low-latency Feature Store, runs inference using a model like DLRM (Deep Learning Recommendation Model) or FFM (Field-aware Factorization Machine), and returns probabilities. Meanwhile, the Ad Server logs impressions to Kafka. Clicks are logged to a separate Kafka topic. A Flink streaming job joins clicks and impressions using a time window to create labeled training data, which continuously fine-tunes the online model.

Data Model

Entity	Fields / Schema	Storage Choice
user_features	user_id, demographics, browsing_history, recent_clicks	In-Memory Key-Value Store (Aerospike / Redis)
ad_metadata	ad_id, campaign_id, advertiser_id, bid_price, category	In-Memory Cache (Memcached)
impression_log	impression_id, timestamp, features (JSON), ad_id, user_id	Kafka Topic -> S3 Data Lake

Detailed Design

Model Architecture (DLRM)

CTR models must handle both dense features (age, time of day) and sparse categorical features (user ID, specific URL). Modern architectures use Deep Learning Recommendation Models (DLRM). Sparse features are mapped to dense embeddings using massive embedding tables. Because these tables don't fit on a single GPU, they are sharded across multiple GPUs (Model Parallelism), while the neural network layers that process dense features are replicated (Data Parallelism).

Online Learning and Feedback Loops

Ad trends shift in minutes (e.g., breaking news, flash sales). Batch training a model once a day is too slow. The system uses Online Learning: the Flink stream joins impressions and clicks. If an impression doesn't receive a click within 10 minutes, it's labeled as a negative (no-click). These labeled examples are fed continuously to a training worker that updates the weights of the production model via micro-batches.

Probability Calibration

If the model outputs 0.05, it must mean there is exactly a 5% chance of a click. If the model is uncalibrated (e.g., it ranks perfectly but outputs 0.20 for a 5% true probability), the advertiser will be overcharged based on the eCPM formula (Bid * CTR). Techniques like Platt Scaling or Isotonic Regression are applied as a final layer to calibrate the output probabilities.

Bottlenecks & Solutions

The Impression-Click Join is notoriously difficult. Impressions happen instantly, but clicks can arrive minutes or even hours later. Maintaining state in Flink for every single impression across millions of QPS requires massive amounts of RAM. Strategies include using bloom filters, aggressive time-window eviction (e.g., dropping unmatched impressions after 15 minutes and treating them as negatives), and delayed negative sampling.

Interview Follow-up Questions

Q: How do you handle 'Delayed Feedback' where a user clicks an ad hours later?

We typically use a short window (e.g., 10 minutes) to assign a negative label so the model can learn quickly. If a click arrives hours later, it's treated as a 'fake negative'. During nightly batch retraining on the Data Lake, we correct these labels and retrain the model to fix the bias introduced by the fast online learning pipeline.

Q: How do you store and update a 500GB embedding table for online inference?

The embedding table is sharded (partitioned by hash) across the RAM of a cluster of inference nodes. When a request arrives, the aggregator node scatters the feature lookups to the specific shards holding those embeddings, gathers the dense vectors, and runs the top-level neural network. This scatter-gather pattern is crucial for massive CTR models.