Lesson 16 / 30

Sizing a Cache

Estimate cache memory from the hot working set and predict how hit ratio reduces backend load.

By · 15 years in web development

Cache the working set, not everything

A cache is effective because access is usually skewed: a small fraction of items receives most of the reads. The working set is the data accessed within some window, such as a day. A common starting assumption is that a modest fraction of objects, say 20%, serves most requests, so the cache needs memory for that hot fraction, not the full dataset. Size it as hot items x (value size + key size + per-item overhead); in-memory stores add metadata per entry, so do not assume raw value bytes alone. The payoff is measured by hit ratio. Backend load equals total reads x (1 - hit ratio), so moving from 90% to 99% hits cuts database reads by 10x. That non-linear benefit is why the last few percent of hit ratio are worth fighting for. Also plan for cache failure: if the cache restarts empty, the database briefly sees the full read load.

Where the reads go

Client reads hit the cache first; only misses fall through to the database, so hit ratio directly controls database load.

Three connected stages: client reads, cache, database.
Figure 6.1: client reads -> cache -> database

Memory and backend load

Estimating cache memory and database read load for different hit ratios.

total_items = 200_000_000
hot_fraction = 0.2
value_bytes, key_bytes, overhead_bytes = 1_000, 40, 60
read_qps = 50_000

cache_gb = total_items * hot_fraction * (value_bytes + key_bytes + overhead_bytes) / 1e9
print(f"cache memory needed: {cache_gb:.0f} GB")
for hit in (0.80, 0.90, 0.99):
    print(f"hit {hit:.0%}: DB reads {read_qps * (1 - hit):,.0f}/s")

Output:

cache memory needed: 44 GB
hit 80%: DB reads 10,000/s
hit 90%: DB reads 5,000/s
hit 99%: DB reads 500/s

Plan for a cold cache

After a cache flush or a new deploy, every read misses. Protect the database with request coalescing, gradual warm-up or rate limits, or the cache restart becomes an outage.

Quick check: Reads are 20,000 per second. If the cache hit ratio rises from 90% to 98%, what happens to database reads?

  • They drop from 2,000/s to 400/s
  • They drop from 18,000/s to 19,600/s
  • They stay at 2,000/s
  • They drop from 2,000/s to 1,600/s
Answer

They drop from 2,000/s to 400/s — Misses are 10% then 2% of 20,000: 2,000/s falls to 400/s, a 5x reduction from an 8-point change in hit ratio.