Lesson 4 / 30

Latency Orders of Magnitude

Recall the rough latency of memory, SSD, network and disk access and use the gaps to reason about design.

By · 15 years in web development

The latency ladder

You do not need exact figures, only the order of magnitude and the gaps between rungs. Reading from a CPU cache takes around a nanosecond. Reading main memory takes on the order of 100 nanoseconds. A random read from an NVMe SSD is in the tens to around a hundred microseconds, roughly 100 to 1,000 times slower than memory. A network round trip inside one data centre is commonly in the range of a few hundred microseconds. A spinning-disk seek is several milliseconds. A round trip between continents is tens to a couple of hundred milliseconds, bounded by the speed of light in fibre. Each jump is roughly 10x to 1000x. These gaps explain classic design choices: cache hot data in memory, avoid many sequential network calls per request, and keep users close to servers with regional deployments or CDNs.

Each rung is far slower than the last

Moving from CPU cache to main memory, SSD, the data-centre network and finally a cross-continent link, latency grows by roughly one to three orders of magnitude at each step.

Five connected stages: CPU cache, main memory, SSD, data-centre network, cross-continent network.
Figure 2.1: CPU cache -> main memory -> SSD -> data-centre network -> cross-continent network

How latency budgets add up

A request that makes sequential calls pays every latency in turn. Here we use rough, illustrative values.

ns = 1e-9
us = 1e-6
ms = 1e-3
steps = {
    "10 cache-miss memory reads": 10 * 100 * ns,
    "3 SSD random reads": 3 * 100 * us,
    "5 sequential in-DC RPCs": 5 * 500 * us,
    "1 cross-continent round trip": 150 * ms,
}
for name, sec in steps.items():
    print(f"{name:30s} {sec * 1000:8.3f} ms")
print(f"{'total':30s} {sum(steps.values()) * 1000:8.3f} ms")

Output:

10 cache-miss memory reads        0.001 ms
3 SSD random reads                0.300 ms
5 sequential in-DC RPCs           2.500 ms
1 cross-continent round trip    150.000 ms
total                           152.801 ms

Distance dominates

Notice that one cross-continent round trip outweighs everything else combined. When a user-facing request feels slow, first count how many times it crosses a long network path before tuning code.

Quick check: Which is closest to the latency of a random read from main memory?

  • About 100 nanoseconds
  • About 100 microseconds
  • About 10 milliseconds
  • About 150 milliseconds
Answer

About 100 nanoseconds — Main memory access is on the order of 100 ns. SSD random reads are around tens to a hundred microseconds, disk seeks several milliseconds, and intercontinental round trips over 100 ms.