Lesson 15 / 30

Utilisation and Queueing Delay

Explain why latency rises steeply as utilisation approaches 100% and choose sensible headroom.

By · 15 years in web development

Why 90% busy feels broken

When requests arrive randomly, they sometimes bunch up and must wait in a queue even if the server is idle on average. Queueing theory captures this. In the simplest textbook model (M/M/1: random arrivals, random service times, one server), average time in the system is W = S / (1 - rho), where S is the service time and rho is utilisation. At 50% utilisation, W = 2S. At 80%, W = 5S. At 90%, W = 10S. At 99%, W = 100S. Real systems do not follow the model exactly, but the shape is general: latency grows non-linearly and explodes near saturation, and tail latency (p99) degrades even earlier than the average. This is the mathematical reason behind utilisation targets well below 100% and behind autoscaling that reacts before CPUs are pegged. Multiple servers sharing one queue behave better than separate queues, which is one reason load balancers help.

The hockey-stick curve

Average response time from the M/M/1 formula for a 10 ms service time.

service_ms = 10
for rho in (0.5, 0.7, 0.8, 0.9, 0.95, 0.99):
    w = service_ms / (1 - rho)
    bar = "#" * int(w / 20)
    print(f"util {rho:4.0%}  avg {w:6.0f} ms  {bar}")

Output:

util  50%  avg     20 ms  #
util  70%  avg     33 ms  #
util  80%  avg     50 ms  ##
util  90%  avg    100 ms  #####
util  95%  avg    200 ms  #########
util  99%  avg   1000 ms  #################################################

Toll plaza on a holiday weekend

A toll booth that is busy half the time has short queues. Push it to 95% busy and a few cars arriving together create a long line that takes ages to clear, even though the booth only slightly increased its work.

Quick check: In the M/M/1 model, if utilisation goes from 80% to 90%, what happens to the average time in system?

  • It increases by about 12%
  • It doubles
  • It stays the same
  • It halves
Answer

It doubles — W = S / (1 - rho): at 80% it is 5S, at 90% it is 10S, so it doubles even though utilisation rose by only 10 points.