Ad Click Aggregation
IntermediateOverview
An ad click aggregation system ingests a massive stream of click events and produces near-real-time aggregated metrics - clicks per ad, per minute, per region - that power advertiser dashboards and billing. Because clicks translate directly into money owed by advertisers, the design must balance real-time freshness with eventually exact, auditable counts, and defend against duplicate and fraudulent clicks.
Functional Requirements
- Ingest click events (ad_id, user, timestamp, context) at very high volume.
- Aggregate counts over time windows (per-minute, per-hour) and dimensions (ad, campaign, region).
- Serve aggregated metrics to advertiser dashboards with low latency.
- Support filtering/grouping (top ads by clicks, clicks by country).
- Produce accurate numbers for billing, correcting real-time approximations.
Non-Functional Requirements
- Scalable: tens of millions of events/sec at peak.
- Correctness for billing: final counts must be exact and auditable (money).
- Freshness: dashboards update within seconds to a couple of minutes.
- Fault tolerant: no data loss; recover from processor crashes by replay.
- Idempotent: duplicate events (retries, at-least-once delivery) must not inflate counts.
Capacity Estimation
Assume 10M clicks/sec peak, each event ~100 bytes.
- Ingest bandwidth: 10M/s x 100 B = 1 GB/s into the pipeline.
- Storage (raw): 10M/s x 100 B x 86,400 s ~ 86 TB/day of raw events - retained for reprocessing/audit, tiered to cheap storage.
- Aggregated storage: counts per (ad, minute, region) are orders of magnitude smaller - millions of rows/day, easily served from a fast store.
- Read QPS: dashboard queries are modest (thousands/sec) and hit pre-aggregated data, not raw events.
The design driver is a write-heavy ingest firehose reduced to compact, queryable aggregates.
High-Level Architecture
The architecture relies on the Lambda Pattern. An API Gateway ingests events, pushing them to a distributed message queue (Kafka). From there, the data forks into two paths:
- Speed Layer: A stream processor (e.g., Flink) reads the queue, dedupes events, and calculates real-time 1-minute aggregates, storing them in a fast OLAP database (e.g., Druid or ClickHouse) for dashboarding.
- Batch Layer: The raw events are dumped into a Data Lake (e.g., S3). A daily MapReduce job (e.g., Spark) runs over this immutable log to compute perfectly exact, deduplicated aggregates for billing, overwriting the approximations from the speed layer.
Data Model
| Entity | Fields / Schema | Storage Choice |
|---|---|---|
| click_event | event_id (dedupe key), ad_id, user_id, ts, country, device | Partitioned log + raw event lake |
| agg_minute | (ad_id, minute, country) -> count | OLAP / columnar store (Druid/ClickHouse) |
| agg_accurate | (ad_id, window) -> exact count | Billing-grade store (batch output) |
| dedupe_set | seen event_ids per window | KV / probabilistic filter |
Detailed Design
Ingest as a Partitioned Log
Clicks land on a partitioned log (Kafka) keyed by ad_id, so all events for an ad go to one partition. This ensures consistent aggregation and parallelizes the firehose. The log is durable and replayable, which is the backbone of fault tolerance - a crashed aggregator resumes from its last committed offset.
Speed Layer - Windowed Stream Aggregation
A stream processor consumes the log and maintains per-window counts (e.g., tumbling 1-minute windows per ad and region), checkpointing state so it can recover exactly where it left off. Results flow to a fast OLAP store that dashboards query, giving sub-minute freshness.
Batch Layer - Exact Recompute
Raw events are also written to a cheap data lake. A periodic Spark job recomputes exact counts for completed windows, correcting any approximation, late-arriving events, or stream bugs. Its output is the billing-grade source of truth.
Deduplication for Correctness
Delivery is at-least-once, so the same click can arrive twice. Each event carries a unique event_id; the aggregator dedupes within the window using a Bloom filter, making counting effectively idempotent.
Click Fraud Filtering
A filtering stage drops obvious invalid traffic (bots, repeated clicks from one user) before counting, since advertisers are not billed for fraudulent clicks. Suspicious patterns are flagged for a separate review pipeline.
Bottlenecks & Solutions
The massive influx of events means the Ingest Gateway and the Message Queue are the primary bottlenecks. If the stream processor falls behind, consumer lag increases, making dashboards stale. Furthermore, keeping exact deduplication state for 10M events/second requires immense RAM. Using probabilistic data structures (Bloom filters) trades a tiny bit of accuracy for massive memory savings in the speed layer, while the batch layer does exact deduplication later.