पाठ 26 / 30

Tail Latency and Fan-out

यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।

Explain why p99 matters and calculate how fan-out across many backends amplifies tail latency.

लेखक: · 15 साल से वेब डेवलपर

Averages hide the slow requests

Tail latency is the latency of the slowest requests, described with percentiles: p99 is the value 99% of requests beat. Averages hide it, yet a 1% slow tail can affect many users, because a single page view may make dozens of requests. Fan-out amplifies the tail: if a request calls N backends in parallel and must wait for all of them, it is slow whenever any one is slow. If each backend independently is slow 1% of the time, the chance that at least one of N is slow is 1 - 0.99^N; with 100 backends that is about 63%. So a service whose components look healthy at p99 can deliver a poor experience at the top level. Mitigations include reducing fan-out, setting timeouts, sending a backup or hedged request after a short delay, and keeping utilisation low enough that queues stay short.

Probability a fan-out request hits the tail

Each backend is slow 1% of the time, independently.

p_slow = 0.01
for n in (1, 10, 50, 100):
    p_any = 1 - (1 - p_slow) ** n
    print(f"fan-out {n:3d}: {p_any:5.1%} of requests wait on a slow backend")

Output:

fan-out   1:  1.0% of requests wait on a slow backend
fan-out  10:  9.6% of requests wait on a slow backend
fan-out  50: 39.5% of requests wait on a slow backend
fan-out 100: 63.4% of requests wait on a slow backend

Group photo at a function

The group photo is ready only when every relative is in place. With three people it is quick; with fifty, someone is always missing. Fan-out requests wait for their slowest backend the same way.

त्वरित जाँच: A request fans out to 10 backends in parallel, each slow 5% of the time independently. About what fraction of requests wait on at least one slow backend?

  • 5%
  • About 40%
  • 50%
  • About 10%
Answer

About 40% — 1 - 0.95^10 is about 0.40, so roughly 40% of top-level requests see at least one slow call.