पाठ 15 / 35
Percentiles and Histogram Queries
यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।
Explain why averages hide slow requests, compute percentiles, and query histogram_quantile correctly.
लेखक: Rahul Agarwal · 15 साल से वेब डेवलपर
Averages lie about latency
Latency distributions are usually skewed: most requests are fast, and a small tail is very slow. The mean mixes them into one number that describes almost no real request. A percentile is more honest: p95 = 300 ms means 95% of requests finished within 300 ms and 5% took longer. The p99 describes the slowest 1%, which on a busy site is still many real users, and often your most valuable ones (large carts, heavy accounts).
Prometheus histograms store cumulative bucket counts. histogram_quantile(0.99, ...) finds the bucket where the 99th percentile falls and linearly interpolates inside it, so the result is an estimate whose accuracy depends on bucket boundaries. Put boundaries near the values you care about, such as your SLO threshold.
Two rules: always aggregate with sum by (le) (the le label must survive), and never average percentiles from several instances. The p99 of the fleet is not the mean of each pod's p99; combine the buckets first, then compute the quantile.
Mean vs percentiles on skewed data
Five slow requests out of 100 drag the mean far above what a typical user saw, while the median stays at 40 ms.
import statistics
# 100 request latencies in ms: most are fast, a few are very slow
lat = [40] * 90 + [60] * 5 + [900, 1200, 1500, 2100, 3000]
print("mean:", statistics.mean(lat), "ms")
q = statistics.quantiles(lat, n=100, method="inclusive")
print("p50 :", q[49], "ms")
print("p95 :", q[94], "ms")
print("p99 :", round(q[98], 1), "ms")
print("max :", max(lat), "ms")
Output:
mean: 126 ms p50 : 40.0 ms p95 : 102.0 ms p99 : 2109.0 ms max : 3000 ms
Fleet-wide p99 from a histogram
Buckets from every instance are summed per le before the quantile is computed. The second query keeps route so you get one p99 per endpoint; the third is the true average from _sum and _count.
histogram_quantile(0.99,
sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.99,
sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
sum(rate(http_request_duration_seconds_sum[5m]))
/ sum(rate(http_request_duration_seconds_count[5m]))Native histograms
Newer Prometheus versions support native histograms, which use automatically scaled exponential buckets stored as a single series instead of one series per bucket. They give better accuracy at lower cardinality. Check whether your Prometheus version and client library have the feature enabled before relying on it.
त्वरित जाँच: Three pods report p99 latencies of 100 ms, 120 ms and 900 ms. What is the correct way to get the fleet p99?
- Average them: about 373 ms
- Take the maximum: 900 ms
- Sum the histogram buckets of all pods by le, then apply histogram_quantile
- Take the median: 120 ms
Answer
Sum the histogram buckets of all pods by le, then apply histogram_quantile — Percentiles cannot be averaged or combined directly. Summing the underlying bucket counts and computing the quantile from the merged distribution is the correct approach.