# Percentiles and Histogram Queries — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-histograms-percentiles

> Explain why averages hide slow requests, compute percentiles, and query histogram_quantile correctly.

## Averages lie about latency

Latency distributions are usually **skewed**: most requests are fast, and a small tail is very slow. The **mean** mixes them into one number that describes almost no real request. A **percentile** is more honest: p95 = 300 ms means 95% of requests finished within 300 ms and 5% took longer. The **p99** describes the slowest 1%, which on a busy site is still many real users, and often your most valuable ones (large carts, heavy accounts).

Prometheus histograms store cumulative bucket counts. `histogram_quantile(0.99, ...)` finds the bucket where the 99th percentile falls and **linearly interpolates** inside it, so the result is an estimate whose accuracy depends on bucket boundaries. Put boundaries near the values you care about, such as your SLO threshold.

Two rules: always aggregate with `sum by (le)` (the `le` label must survive), and never average percentiles from several instances. The p99 of the fleet is not the mean of each pod's p99; combine the buckets first, then compute the quantile.

## Mean vs percentiles on skewed data

Five slow requests out of 100 drag the mean far above what a typical user saw, while the median stays at 40 ms.

```python
import statistics

# 100 request latencies in ms: most are fast, a few are very slow
lat = [40] * 90 + [60] * 5 + [900, 1200, 1500, 2100, 3000]

print("mean:", statistics.mean(lat), "ms")
q = statistics.quantiles(lat, n=100, method="inclusive")
print("p50 :", q[49], "ms")
print("p95 :", q[94], "ms")
print("p99 :", round(q[98], 1), "ms")
print("max :", max(lat), "ms")
```

Output:

```
mean: 126 ms
p50 : 40.0 ms
p95 : 102.0 ms
p99 : 2109.0 ms
max : 3000 ms
```

## Fleet-wide p99 from a histogram

Buckets from every instance are summed per `le` before the quantile is computed. The second query keeps `route` so you get one p99 per endpoint; the third is the true average from `_sum` and `_count`.

```promql
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

histogram_quantile(0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

sum(rate(http_request_duration_seconds_sum[5m]))
  / sum(rate(http_request_duration_seconds_count[5m]))
```

## Native histograms

Newer Prometheus versions support **native histograms**, which use automatically scaled exponential buckets stored as a single series instead of one series per bucket. They give better accuracy at lower cardinality. Check whether your Prometheus version and client library have the feature enabled before relying on it.

**Quiz:** Three pods report p99 latencies of 100 ms, 120 ms and 900 ms. What is the correct way to get the fleet p99?

- [ ] Average them: about 373 ms
- [ ] Take the maximum: 900 ms
- [x] Sum the histogram buckets of all pods by le, then apply histogram_quantile
- [ ] Take the median: 120 ms

*Answer:* Sum the histogram buckets of all pods by le, then apply histogram_quantile. Percentiles cannot be averaged or combined directly. Summing the underlying bucket counts and computing the quantile from the merged distribution is the correct approach.
