Little's Law, Amdahl's Law and Queueing

Three formulas worth memorising

Little's law says the average number of requests in a system equals the arrival rate times the average time each spends inside: L = λ × W. At 2,000 requests per second with 50 ms average latency, about 100 requests are in flight, which tells you how many threads or connections you need. Amdahl's law limits speed-ups from parallelism: if a fraction p of the work can be parallelised across n workers, speed-up = 1 / ((1 − p) + p / n). With 90% parallelisable work the maximum speed-up is 10×, however many machines you add, so the serial part (a single database writer, a global lock) caps scalability. Queueing theory explains why latency explodes near full load: for a simple single-server queue, average response time grows roughly with 1 / (1 − utilisation). At 50% busy, responses take about twice the bare service time; at 90%, about ten times, five times worse than at 50%; near 100%, they grow without bound.

Back-of-the-envelope calculations

The same three ideas, applied to a checkout service.

# Little's law: requests in flight
arrival_rate = 2000          # requests per second
latency_s = 0.050            # 50 ms
in_flight = arrival_rate * latency_s          # = 100 concurrent requests

# Amdahl's law: speed-up with n workers when 90% of work is parallel
def amdahl(p, n):
    return 1 / ((1 - p) + p / n)

speedups = {n: round(amdahl(0.9, n), 2) for n in (2, 8, 32, 1024)}
# approaches 10x, never exceeds it

# queueing: response time as a multiple of service time (M/M/1 model)
for rho in (0.5, 0.7, 0.9, 0.95):
    relative_response = 1 / (1 - rho)
    print(rho, round(relative_response, 1))

Plan for headroom, not 100% utilisation

Because latency climbs steeply near saturation, size systems to run at roughly 50 to 70% of capacity at peak. The spare capacity absorbs bursts, failures of a node and slow dependencies.

Quick check: A service receives 500 requests per second and each takes 200 ms on average. How many requests are in flight on average?

• 2.5
• 700
• 100
• 2,500