Lesson 15 / 30
Utilisation and Queueing Delay
Explain why latency rises steeply as utilisation approaches 100% and choose sensible headroom.
By Rahul Agarwal · 15 years in web development
Why 90% busy feels broken
When requests arrive randomly, they sometimes bunch up and must wait in a queue even if the server is idle on average. Queueing theory captures this. In the simplest textbook model (M/M/1: random arrivals, random service times, one server), average time in the system is W = S / (1 - rho), where S is the service time and rho is utilisation. At 50% utilisation, W = 2S. At 80%, W = 5S. At 90%, W = 10S. At 99%, W = 100S. Real systems do not follow the model exactly, but the shape is general: latency grows non-linearly and explodes near saturation, and tail latency (p99) degrades even earlier than the average. This is the mathematical reason behind utilisation targets well below 100% and behind autoscaling that reacts before CPUs are pegged. Multiple servers sharing one queue behave better than separate queues, which is one reason load balancers help.
The hockey-stick curve
Average response time from the M/M/1 formula for a 10 ms service time.
service_ms = 10
for rho in (0.5, 0.7, 0.8, 0.9, 0.95, 0.99):
w = service_ms / (1 - rho)
bar = "#" * int(w / 20)
print(f"util {rho:4.0%} avg {w:6.0f} ms {bar}")
Output:
util 50% avg 20 ms # util 70% avg 33 ms # util 80% avg 50 ms ## util 90% avg 100 ms ##### util 95% avg 200 ms ######### util 99% avg 1000 ms #################################################
Toll plaza on a holiday weekend
A toll booth that is busy half the time has short queues. Push it to 95% busy and a few cars arriving together create a long line that takes ages to clear, even though the booth only slightly increased its work.
Quick check: In the M/M/1 model, if utilisation goes from 80% to 90%, what happens to the average time in system?
- It increases by about 12%
- It doubles
- It stays the same
- It halves
Answer
It doubles — W = S / (1 - rho): at 80% it is 5S, at 90% it is 10S, so it doubles even though utilisation rose by only 10 points.