Video Streaming Service
IntermediateOverview
A video streaming service lets users upload, process, and watch video on demand at global scale. The design centers on transcoding uploads into multiple bitrates and delivering the resulting segments from edge locations close to viewers with minimal buffering.
Functional Requirements
- Upload a video and process it into multiple resolutions/bitrates.
- Stream video on demand with adaptive bitrate based on the viewer's bandwidth.
- Search and browse the video catalog with metadata (title, thumbnails, duration).
- Track playback progress and basic view analytics.
Non-Functional Requirements
- Smooth playback: low startup latency and minimal rebuffering.
- High availability and global reach for viewers.
- Durable storage of source and transcoded assets.
- Cost efficiency for storage and egress bandwidth at scale.
Capacity Estimation
Assume 1M new videos/day (avg 300 MB source), 100M DAU each watching 5 videos/day, avg watch 10 min at 5 Mbps.
- QPS:
- Views: 100M x 5 = 500M/day -> 500M / 86,400 s ~ 5,800 view-starts/sec (metadata/manifest requests; peak ~3x ~ 17,000/sec).
- Uploads: 1M / 86,400 s ~ 12 uploads/sec.
- Storage:
- Source: 1M x 300 MB = 300 TB/day.
- Transcoded copies (say 5 renditions ~ 1.5x source) add ~450 TB/day, so ~ 750 TB/day total -> ~270 PB/year before lifecycle tiering.
- Bandwidth (egress dominates): concurrent streams ~ (500M views x 10 min) / 1,440 min/day ~ 3.47M concurrent; at 5 Mbps -> 3.47M x 5 Mbps ~ 17 Tbps of egress, served overwhelmingly from the CDN edge.
High-Level Architecture
The architecture is fundamentally split into an asynchronous Ingestion & Processing Pipeline (handling uploads, transcoding, and metadata extraction) and a globally distributed Delivery Network (using edge CDNs to serve video chunks directly to players). A metadata database stores video information, while object storage holds the heavy video files.
Data Model
| Entity | Fields / Schema | Storage Choice |
|---|---|---|
| video | video_id (PK), title, uploader_id, status, duration, created_at | Relational / document metadata store |
| rendition | video_id, resolution, bitrate, manifest_url | Metadata store, points into object storage |
| segment | object key per chunk (e.g., video_id/1080p/seg_0001.ts) | Object/blob store (S3), fronted by CDN |
| view_event | video_id, user_id, ts, watch_seconds | Analytics store (column-oriented) |
Detailed Design
Upload and Transcoding Pipeline
An upload lands in raw object storage, then a job is enqueued in Kafka. Transcoding workers split the source video into short segments (e.g., 4 seconds) and encode each segment into multiple renditions (e.g., 240p, 720p, 1080p, 4K). Segmenting lets encoding run in parallel across dozens of servers, vastly speeding up processing. Completed segments are written back to object storage, alongside an adaptive manifest file (HLS or DASH) that indexes all available renditions.
Adaptive Bitrate Streaming (ABR)
Players don't download a single massive MP4 file. They first fetch the manifest, then request individual 4-second segments. The player constantly measures network throughput and buffer health. If the user drives into a tunnel and bandwidth drops, the player requests the next segment at 240p instead of 1080p, avoiding a playback stall. When bandwidth recovers, it steps back up to 1080p.
Content Delivery Networks (CDNs)
The 17 Tbps of egress traffic cannot be served from a central data center. Segments are heavily cached at CDN edge nodes located inside ISPs worldwide. The vast majority of byte delivery never touches origin storage, which is what makes global low-latency playback economically feasible. Without CDNs, a video streaming service simply cannot exist at scale.
Bottlenecks & Solutions
The primary bottleneck is CDN Costs and Egress. Serving Petabytes of video is incredibly expensive. To optimize this, large platforms (like Netflix/YouTube) build their own custom CDNs (e.g., Open Connect) and deploy them directly inside ISP networks (Comcast, Verizon) to bypass internet transit fees completely.