Skip to content
AI360Xpert

Storage Estimation

Storage Estimation architecture
Storage Estimation architecture

Overview

Storage estimation is the process of calculating how much disk space your system will require over a given time frame (usually 1 to 5 years). This dictates what kind of database you use, whether you need sharding, and how you handle data archiving.

🧠 Mental model: If you are building a library, you need to know how many new books arrive every day and how thick they are. Multiply that by 5 years, and you know how many bookshelves you need to buy and how big the building must be.

Key Concepts

The Calculation Process

  1. Define the Data Entities: What are you storing? (Users, Tweets, Photos, Videos).
  2. Estimate Size per Entity:
    • User Profile: ~2 KB
    • Text Post (e.g., Tweet): ~1 KB (ID, text, timestamp, user_id)
    • Photo: ~2 MB
    • Video: ~50 MB
  3. Calculate Daily Growth: (New items per day) x (Size per item).
  4. Project Over Time: Multiply daily growth by 365 days, then by 5 years.

Example: Instagram Clone (Photos only)

Assumptions: 10 million Daily Active Users (DAU). 20% of users upload 1 photo per day. Photo size is 2 MB.

  • Daily Photos: 10M * 0.20 = 2 million photos/day
  • Daily Storage: 2 million * 2 MB = 4,000,000 MB = 4 TB / day
  • Yearly Storage: 4 TB * 365 ~ 1.5 PB / year
  • 5-Year Storage: 1.5 PB * 5 = 7.5 PB

Architectural conclusion: 7.5 PB is way too large for a relational database. We must use Object Storage (like AWS S3) for the photos, and only store the photo metadata/URLs in a database.

Trade-offs

Storage estimates often ignore overhead (indexes, database replication, backups) which can easily double or triple the actual raw disk space needed. However, estimating raw data is sufficient for interview purposes. If the calculated storage is small (e.g., 50 GB/year), you can prioritize simplicity and use a single managed SQL database. If it is massive (Petabytes), you trade simplicity for complexity, requiring sharding, object storage, or NoSQL.

Interview Tips

  • Differentiate between Metadata (text/IDs, goes in a database) and Media (images/videos, goes in object storage). Calculate them separately.
  • Don't forget the Read/Write ratio. Storage estimation is heavily based on the Write volume.
  • If you calculate a massive number (Petabytes), state out loud: "Because of this scale, we will need to shard our database and use tiered object storage."

Summary

  • Storage estimation calculates how much data the system will accumulate over 1-5 years.
  • Multiply daily active users by actions-per-day, then by the average payload size.
  • Separate metadata calculations (KB) from media calculations (MB/GB).
  • Use the final number to decide between a single DB, a sharded DB, or Object Storage.
  • Always clearly state your assumptions for data sizes (e.g., 'Assuming a photo is 2MB').