Lesson 13 / 30

Little's Law

Use L = lambda x W to relate throughput, latency and the number of requests in flight.

By · 15 years in web development

One law, many uses

Little's Law says that in a stable system, the average number of items inside it equals the arrival rate times the average time each item spends inside: L = lambda x W. It holds regardless of the arrival pattern, as long as you measure over a long enough window and the system is not steadily growing. In a web service, lambda is requests per second, W is average response time in seconds, and L is the average number of requests in flight at once. This gives you concurrency, which sizes thread pools, connection pools and worker counts. For example, 2,000 requests per second at 50 ms each means about 100 requests in flight, so a pool of 20 database connections that each request holds for the full 50 ms would become the bottleneck. You can also rearrange it: if you have 64 workers and 200 ms per job, the most you can sustain is 64 / 0.2 = 320 jobs per second.

The three quantities of Little's Law

Arrival rate multiplied by response time gives the number of requests in flight; knowing any two lets you compute the third.

Three connected stages: arrival rate, requests in flight, response time.
Figure 5.1: arrival rate -> requests in flight -> response time

Sizing a connection pool

Little's Law applied to the part of a request that holds a database connection.

qps = 3_000
db_time_s = 0.012          # each request holds a connection for 12 ms
connections_needed = qps * db_time_s
print(f"average connections busy: {connections_needed:.0f}")

workers, job_seconds = 64, 0.2
print(f"max jobs/s with {workers} workers: {workers / job_seconds:.0f}")

Output:

average connections busy: 36
max jobs/s with 64 workers: 320

Queue at a ticket counter

If 2 people join the railway ticket queue every minute and each spends 10 minutes in line and at the window, you will see about 20 people there at any moment. Count any two quantities and you know the third.

Quick check: A service handles 500 requests per second with an average latency of 200 ms. About how many requests are in flight on average?

  • 100
  • 2,500
  • 10
  • 1,000
Answer

100 — L = lambda x W = 500 x 0.2 = 100 requests in flight.