Lesson 26 / 32

Concurrency, Throttling and Downstream Protection

Estimate required concurrency, explain throttling, and use concurrency limits to protect fragile dependencies.

By · 15 years in web development

Scaling your code is easy; scaling everything else is not

In the standard Lambda model each environment handles one request at a time, so the concurrency you need is roughly requests per second multiplied by average duration in seconds (Little's law). Each account has a per-region concurrency quota shared by all its functions; once it is exhausted, new invocations are throttled. Synchronous callers receive a 429 error, while asynchronous and queue-based events are retried later. Two settings help. Reserved concurrency guarantees a function a slice of the quota and also caps it at that number, which is a cheap way to protect a fragile downstream such as a small database or a partner API with rate limits. For SQS triggers, the event source mapping's maximum concurrency setting limits how many concurrent invocations that queue can drive. The broader lesson: serverless scales your code almost instantly, but nothing else in your system automatically scales with it, so put limits and queues in front of the slow parts.

Little's law in three lines

Doubling duration doubles the concurrency you need just as much as doubling traffic does. Slow dependencies are a scaling problem, not only a latency problem.

# Little's law: concurrency = arrival rate x time each request spends in the system
def concurrency_needed(requests_per_sec: float, avg_duration_ms: float) -> float:
    return requests_per_sec * (avg_duration_ms / 1000)

for rps, ms in [(200, 50), (200, 800), (2000, 800)]:
    print(f"{rps:>5} req/s x {ms:>3} ms -> {concurrency_needed(rps, ms):6.0f} concurrent executions")

Output:

  200 req/s x  50 ms ->     10 concurrent executions
  200 req/s x 800 ms ->    160 concurrent executions
 2000 req/s x 800 ms ->   1600 concurrent executions

Toll plaza booths

If each car takes 30 seconds at a booth and 20 cars arrive per minute, you need at least 10 booths open. Cars arriving faster than booths can serve them form a queue, and if the road beyond the plaza is narrow, opening more booths only moves the jam further down.

Quick check: A function receives 300 requests per second and each takes 400 ms on average. Roughly how many concurrent executions does it need?

  • 120
  • 750
  • 300
  • 12
Answer

120 — Concurrency is about rate x duration = 300 x 0.4 s = 120.