File Storage And Sync Service
IntermediateOverview
A file storage and sync service (think Dropbox or Google Drive) lets users store files in the cloud and keep them synchronized across all their devices. The core challenges are efficient upload/download of large files, detecting changes cheaply, and resolving conflicts when the same file is edited on multiple devices.
Functional Requirements
- Upload and download files, including large files, reliably.
- Automatically sync changes across all of a user's devices.
- Support file versioning and restore of previous versions.
- Share files and folders with other users with access controls.
Non-Functional Requirements
- Durability: uploaded data must never be lost.
- Efficient sync: transfer only what changed, not whole files.
- High availability for the metadata and download paths.
- Sync latency: changes should propagate to other devices within seconds.
Capacity Estimation
Assume 100M users, avg 50 GB stored each, 1M file changes/sec globally at peak, avg file 1 MB with block size 4 MB (files chunked).
- Storage: 100M x 50 GB = 5 EB of raw user data (deduplication and tiering reduce the effective footprint substantially).
- QPS:
- Metadata operations (change notifications, version lookups): ~1M ops/sec at peak.
- Block upload/download requests scale with change volume and chunk count.
- Bandwidth: at 1M changes/sec x avg 1 MB delta ~ 1 TB/s aggregate at peak, dominated by block transfer to and from object storage; block-level delta sync sharply reduces this versus whole-file transfer.
High-Level Architecture
The architecture splits into two distinct paths: Metadata (folder structures, permissions, file versions) and Block Storage (the actual file bytes). Clients communicate with a Metadata Server via WebSockets for real-time change notifications, and upload file chunks directly to a Block Server backed by cloud object storage like S3.
Data Model
| Entity | Fields / Schema | Storage Choice |
|---|---|---|
| file | file_id (PK), owner_id, path, latest_version, updated_at | Relational / metadata store (e.g., PostgreSQL or Spanner) |
| version | file_id, version, ordered block_hash[], size, created_at | Metadata store |
| block | block_hash (PK, content address), bytes | Object/blob store (S3), content-addressed and deduplicated |
| share | file_id, grantee_id, permission | Relational store |
Detailed Design
Block-Level Sync
The client splits each file into fixed-size blocks (e.g., 4 MB) and hashes each block (SHA-256). To sync, it sends only the hashes to the Metadata Service, which replies with the set of blocks the server does not already have. The client uploads only those missing blocks to the Block Service, which stores them in content-addressed object storage. Because blocks are keyed by content hash, identical blocks (across versions, or across users) are stored exactly once. This delta sync is what makes syncing a one-line edit to a large 50 MB file cheap - it only uploads the single 4 MB block that changed.
Change Propagation (The Notification Service)
Clients maintain a persistent WebSocket connection to a Notification Service. When Client A modifies a file and updates the Metadata DB, the DB emits an event (via Kafka) to the Notification Service, which pushes a "changes pending" signal down Client B's WebSocket. Client B then requests the latest metadata, sees the new block hashes, and downloads them. We don't push the actual file bytes over WebSockets, only the notification.
Metadata Consistency
Metadata is the source of truth for the file tree and version history. It requires strong consistency (ACID transactions) so all devices agree on the latest committed version. A relational database like PostgreSQL or a globally consistent NewSQL DB like Spanner is ideal here.
Bottlenecks & Solutions
Client-side operations are a major bottleneck; hashing a 5 GB file to find modified blocks eats battery and CPU on a mobile phone. Clients must maintain an efficient local metadata cache to quickly find changed blocks. On the server side, keeping millions of WebSockets open for the Notification Service requires immense RAM and careful load balancing.