# Utilisation and Queueing Delay — Cost & Capacity Estimation

Source: https://www.skillbyai.com/en/capacity-estimation/ce-utilisation-queueing

> Explain why latency rises steeply as utilisation approaches 100% and choose sensible headroom.

## Why 90% busy feels broken

When requests arrive randomly, they sometimes bunch up and must wait in a queue even if the server is idle on average. Queueing theory captures this. In the simplest textbook model (M/M/1: random arrivals, random service times, one server), average time in the system is **W = S / (1 - rho)**, where S is the service time and rho is utilisation. At 50% utilisation, W = 2S. At 80%, W = 5S. At 90%, W = 10S. At 99%, W = 100S. Real systems do not follow the model exactly, but the shape is general: **latency grows non-linearly and explodes near saturation**, and tail latency (p99) degrades even earlier than the average. This is the mathematical reason behind utilisation targets well below 100% and behind autoscaling that reacts before CPUs are pegged. Multiple servers sharing one queue behave better than separate queues, which is one reason load balancers help.

## The hockey-stick curve

Average response time from the M/M/1 formula for a 10 ms service time.

```python
service_ms = 10
for rho in (0.5, 0.7, 0.8, 0.9, 0.95, 0.99):
    w = service_ms / (1 - rho)
    bar = "#" * int(w / 20)
    print(f"util {rho:4.0%}  avg {w:6.0f} ms  {bar}")
```

Output:

```
util  50%  avg     20 ms  #
util  70%  avg     33 ms  #
util  80%  avg     50 ms  ##
util  90%  avg    100 ms  #####
util  95%  avg    200 ms  #########
util  99%  avg   1000 ms  #################################################
```

## Toll plaza on a holiday weekend

A toll booth that is busy half the time has short queues. Push it to 95% busy and a few cars arriving together create a long line that takes ages to clear, even though the booth only slightly increased its work.

**Quiz:** In the M/M/1 model, if utilisation goes from 80% to 90%, what happens to the average time in system?

- [ ] It increases by about 12%
- [x] It doubles
- [ ] It stays the same
- [ ] It halves

*Answer:* It doubles. W = S / (1 - rho): at 80% it is 5S, at 90% it is 10S, so it doubles even though utilisation rose by only 10 points.
