पाठ 10 / 30
Sizing Stored Data
यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।
Estimate record sizes, daily growth and total storage over a retention period.
लेखक: Rahul Agarwal · 15 साल से वेब डेवलपर
Storage = count x size x time
Storage estimates follow one formula: new objects per day x average object size x retention days. Do it per data type, because sizes differ wildly. A text post with metadata is on the order of a kilobyte. A compressed photo is commonly hundreds of kilobytes to a few megabytes. A minute of video can be tens of megabytes or more depending on resolution and bitrate. Metadata rows (IDs, timestamps, foreign keys) are small but numerous; count fields and multiply by their sizes, for example 8 bytes for a 64-bit integer, 16 bytes for a UUID, and roughly one byte per ASCII character of text. Media almost always dominates, which is why it usually lives in object storage while metadata lives in a database. Finally, decide retention: keeping everything forever, keeping 90 days of logs, or moving old data to cheaper tiers changes the answer by orders of magnitude.
Building up a storage estimate
Start with objects per day, multiply by size per object, extend over the retention period, then multiply by replicas and overhead to get raw disk.
Photo-sharing app storage
Metadata and media estimated separately; media dominates.
uploads_per_day = 4_000_000
photo_bytes = 500 * 1000 # 500 KB average after compression
meta_bytes = 8 + 8 + 8 + 200 + 100 # ids, timestamp, caption, misc
years = 3
days = 365 * years
media_tb = uploads_per_day * photo_bytes * days / 1e12
meta_tb = uploads_per_day * meta_bytes * days / 1e12
print(f"media over {years} years: {media_tb:,.0f} TB")
print(f"metadata over {years} years: {meta_tb:,.2f} TB")
print(f"daily media growth: {uploads_per_day * photo_bytes / 1e12:.1f} TB/day")
Output:
media over 3 years: 2,190 TB metadata over 3 years: 1.42 TB daily media growth: 2.0 TB/day
Thumbnails and variants count too
Image and video services usually store several renditions (thumbnail, medium, original, multiple video bitrates). Add a multiplier for variants or you will underestimate media storage.
त्वरित जाँच: A service stores 1 million 2 KB records per day for 1,000 days. About how much raw data is that before replication?
- About 2 GB
- About 2 PB
- About 200 GB
- About 2 TB
Answer
About 2 TB — 10^6 x 2 x 10^3 bytes = 2 x 10^9 bytes per day; over 1,000 days that is 2 x 10^12 bytes, about 2 TB.