# Tail Latency and Fan-out — Cost & Capacity Estimation

Source: https://www.skillbyai.com/en/capacity-estimation/ce-tail-latency

> Explain why p99 matters and calculate how fan-out across many backends amplifies tail latency.

## Averages hide the slow requests

**Tail latency** is the latency of the slowest requests, described with percentiles: p99 is the value 99% of requests beat. Averages hide it, yet a 1% slow tail can affect many users, because a single page view may make dozens of requests. **Fan-out amplifies the tail**: if a request calls N backends in parallel and must wait for all of them, it is slow whenever any one is slow. If each backend independently is slow 1% of the time, the chance that at least one of N is slow is **1 - 0.99^N**; with 100 backends that is about 63%. So a service whose components look healthy at p99 can deliver a poor experience at the top level. Mitigations include reducing fan-out, setting timeouts, sending a backup or **hedged request** after a short delay, and keeping utilisation low enough that queues stay short.

## Probability a fan-out request hits the tail

Each backend is slow 1% of the time, independently.

```python
p_slow = 0.01
for n in (1, 10, 50, 100):
    p_any = 1 - (1 - p_slow) ** n
    print(f"fan-out {n:3d}: {p_any:5.1%} of requests wait on a slow backend")
```

Output:

```
fan-out   1:  1.0% of requests wait on a slow backend
fan-out  10:  9.6% of requests wait on a slow backend
fan-out  50: 39.5% of requests wait on a slow backend
fan-out 100: 63.4% of requests wait on a slow backend
```

## Group photo at a function

The group photo is ready only when every relative is in place. With three people it is quick; with fifty, someone is always missing. Fan-out requests wait for their slowest backend the same way.

**Quiz:** A request fans out to 10 backends in parallel, each slow 5% of the time independently. About what fraction of requests wait on at least one slow backend?

- [ ] 5%
- [x] About 40%
- [ ] 50%
- [ ] About 10%

*Answer:* About 40%. 1 - 0.95^10 is about 0.40, so roughly 40% of top-level requests see at least one slow call.
