Lesson 5 / 30
Throughput Rules of Thumb
Use rough per-machine throughput figures for CPU, disk and network as starting points, then verify with benchmarks.
By Rahul Agarwal · 15 years in web development
Throughput is per resource
Latency is how long one operation takes; throughput is how many operations finish per second. They are related but not the same, because systems overlap many operations at once. For estimation, keep a few rough per-machine anchors and always label them as assumptions to be benchmarked. A simple in-memory key-value lookup can be served at tens of thousands of requests per second or more per core. A typical web endpoint that does JSON parsing, business logic and one database call might handle hundreds to a few thousand requests per second per server. An SSD can sustain many thousands to hundreds of thousands of random IOPS depending on the device. A network interface is limited by its rated bits per second. The point is not memorising numbers but asking, for each request, which resource does it use the most of, and dividing that resource's capacity by per-request usage.
Capacity from per-request cost
If each request burns 5 ms of CPU, one core can do at most about 200 requests per second.
cpu_ms_per_request = 5
cores_per_server = 8
target_utilisation = 0.6 # leave headroom
per_core = 1000 / cpu_ms_per_request
per_server = per_core * cores_per_server * target_utilisation
print(f"max per core : {per_core:.0f} req/s")
print(f"usable per server: {per_server:.0f} req/s")
Output:
max per core : 200 req/s usable per server: 960 req/s
Benchmark, then plug in
Rules of thumb get you started; a short load test of your real endpoint gives the per-server number you should actually plan with. Replace the assumption, keep the formula.
Quick check: A request needs 4 ms of CPU time. Ignoring other limits, about how many such requests can one core serve per second?
- About 25
- About 400
- About 250
- About 4,000
Answer
About 250 — One second is 1,000 ms; 1,000 / 4 = 250 requests per second per core at full utilisation.