# Throughput Rules of Thumb — Cost & Capacity Estimation

Source: https://www.skillbyai.com/en/capacity-estimation/ce-throughput-numbers

> Use rough per-machine throughput figures for CPU, disk and network as starting points, then verify with benchmarks.

## Throughput is per resource

Latency is how long one operation takes; **throughput** is how many operations finish per second. They are related but not the same, because systems overlap many operations at once. For estimation, keep a few rough per-machine anchors and always label them as assumptions to be benchmarked. A simple in-memory key-value lookup can be served at tens of thousands of requests per second or more per core. A typical web endpoint that does JSON parsing, business logic and one database call might handle hundreds to a few thousand requests per second per server. An SSD can sustain many thousands to hundreds of thousands of random IOPS depending on the device. A network interface is limited by its rated bits per second. The point is not memorising numbers but asking, for each request, **which resource does it use the most of**, and dividing that resource's capacity by per-request usage.

## Capacity from per-request cost

If each request burns 5 ms of CPU, one core can do at most about 200 requests per second.

```python
cpu_ms_per_request = 5
cores_per_server = 8
target_utilisation = 0.6   # leave headroom

per_core = 1000 / cpu_ms_per_request
per_server = per_core * cores_per_server * target_utilisation
print(f"max per core     : {per_core:.0f} req/s")
print(f"usable per server: {per_server:.0f} req/s")
```

Output:

```
max per core     : 200 req/s
usable per server: 960 req/s
```

## Benchmark, then plug in

Rules of thumb get you started; a short load test of your real endpoint gives the per-server number you should actually plan with. Replace the assumption, keep the formula.

**Quiz:** A request needs 4 ms of CPU time. Ignoring other limits, about how many such requests can one core serve per second?

- [ ] About 25
- [ ] About 400
- [x] About 250
- [ ] About 4,000

*Answer:* About 250. One second is 1,000 ms; 1,000 / 4 = 250 requests per second per core at full utilisation.
