Lesson 10 / 30

Sizing Stored Data

Estimate record sizes, daily growth and total storage over a retention period.

By · 15 years in web development

Storage = count x size x time

Storage estimates follow one formula: new objects per day x average object size x retention days. Do it per data type, because sizes differ wildly. A text post with metadata is on the order of a kilobyte. A compressed photo is commonly hundreds of kilobytes to a few megabytes. A minute of video can be tens of megabytes or more depending on resolution and bitrate. Metadata rows (IDs, timestamps, foreign keys) are small but numerous; count fields and multiply by their sizes, for example 8 bytes for a 64-bit integer, 16 bytes for a UUID, and roughly one byte per ASCII character of text. Media almost always dominates, which is why it usually lives in object storage while metadata lives in a database. Finally, decide retention: keeping everything forever, keeping 90 days of logs, or moving old data to cheaper tiers changes the answer by orders of magnitude.

Building up a storage estimate

Start with objects per day, multiply by size per object, extend over the retention period, then multiply by replicas and overhead to get raw disk.

Four connected stages: objects per day, size per object, retention, replicas and overhead.
Figure 4.1: objects per day -> size per object -> retention -> replicas and overhead

Photo-sharing app storage

Metadata and media estimated separately; media dominates.

uploads_per_day = 4_000_000
photo_bytes = 500 * 1000          # 500 KB average after compression
meta_bytes = 8 + 8 + 8 + 200 + 100  # ids, timestamp, caption, misc
years = 3
days = 365 * years

media_tb = uploads_per_day * photo_bytes * days / 1e12
meta_tb = uploads_per_day * meta_bytes * days / 1e12
print(f"media    over {years} years: {media_tb:,.0f} TB")
print(f"metadata over {years} years: {meta_tb:,.2f} TB")
print(f"daily media growth: {uploads_per_day * photo_bytes / 1e12:.1f} TB/day")

Output:

media    over 3 years: 2,190 TB
metadata over 3 years: 1.42 TB
daily media growth: 2.0 TB/day

Thumbnails and variants count too

Image and video services usually store several renditions (thumbnail, medium, original, multiple video bitrates). Add a multiplier for variants or you will underestimate media storage.

Quick check: A service stores 1 million 2 KB records per day for 1,000 days. About how much raw data is that before replication?

  • About 2 GB
  • About 2 PB
  • About 200 GB
  • About 2 TB
Answer

About 2 TB — 10^6 x 2 x 10^3 bytes = 2 x 10^9 bytes per day; over 1,000 days that is 2 x 10^12 bytes, about 2 TB.