Lesson 26 / 30
Tail Latency and Fan-out
Explain why p99 matters and calculate how fan-out across many backends amplifies tail latency.
By Rahul Agarwal · 15 years in web development
Averages hide the slow requests
Tail latency is the latency of the slowest requests, described with percentiles: p99 is the value 99% of requests beat. Averages hide it, yet a 1% slow tail can affect many users, because a single page view may make dozens of requests. Fan-out amplifies the tail: if a request calls N backends in parallel and must wait for all of them, it is slow whenever any one is slow. If each backend independently is slow 1% of the time, the chance that at least one of N is slow is 1 - 0.99^N; with 100 backends that is about 63%. So a service whose components look healthy at p99 can deliver a poor experience at the top level. Mitigations include reducing fan-out, setting timeouts, sending a backup or hedged request after a short delay, and keeping utilisation low enough that queues stay short.
Probability a fan-out request hits the tail
Each backend is slow 1% of the time, independently.
p_slow = 0.01
for n in (1, 10, 50, 100):
p_any = 1 - (1 - p_slow) ** n
print(f"fan-out {n:3d}: {p_any:5.1%} of requests wait on a slow backend")
Output:
fan-out 1: 1.0% of requests wait on a slow backend fan-out 10: 9.6% of requests wait on a slow backend fan-out 50: 39.5% of requests wait on a slow backend fan-out 100: 63.4% of requests wait on a slow backend
Group photo at a function
The group photo is ready only when every relative is in place. With three people it is quick; with fifty, someone is always missing. Fan-out requests wait for their slowest backend the same way.
Quick check: A request fans out to 10 backends in parallel, each slow 5% of the time independently. About what fraction of requests wait on at least one slow backend?
- 5%
- About 40%
- 50%
- About 10%
Answer
About 40% — 1 - 0.95^10 is about 0.40, so roughly 40% of top-level requests see at least one slow call.