Lesson 14 / 35
PromQL Essentials
Write PromQL with selectors, rate, increase and aggregation, and avoid the classic sum-before-rate mistake.
By Rahul Agarwal · 15 years in web development
Instant vectors, range vectors and rate
A PromQL selector like http_requests_total{job="checkout", status=~"5.."} returns an instant vector: the latest value of each matching series. Matchers can be =, !=, =~ (regex) and !~.
Adding a duration, http_requests_total[5m], gives a range vector: all samples from the last five minutes for each series. You cannot graph a range vector directly; you pass it to a function.
rate(counter[5m]) computes the per-second average increase over the window and automatically handles counter resets after a restart. increase() is the same idea expressed as a total over the window. irate() uses only the last two samples, so it is jumpy and better for fast-moving graphs than for alerts.
Aggregation operators such as sum, avg, max and count collapse series, with by (label) keeping chosen labels or without (label) dropping them. Order matters: always rate first, then sum. Summing raw counters first hides the per-series resets and produces wrong spikes. A window of at least four times the scrape interval is a common rule of thumb.
Everyday queries
Each line is an independent query you could paste into the Prometheus or Grafana query editor.
# per-second request rate over the last 5 minutes, one series per target
rate(http_requests_total[5m])
# the same, summed across targets and grouped by status code
sum by (status) (rate(http_requests_total[5m]))
# approximate number of requests in the last hour
sum(increase(http_requests_total[1h]))
# gauges are read directly: resident memory per instance
process_resident_memory_bytes{job="checkout"}
# compare with one day ago
sum(rate(http_requests_total[5m])) / sum(rate(http_requests_total[5m] offset 1d))rate() then sum(), never the other way
rate(sum(http_requests_total)[5m]) is not even valid without a subquery, and the subquery version gives wrong results: when one pod restarts, its counter drops to zero and the sum appears to fall, which rate then misreads. Write sum(rate(http_requests_total[5m])).
Quick check: Why is `rate()` applied to counters instead of graphing the raw counter value?
- Raw counters are encrypted
- The raw value only grows and resets on restart; rate turns it into a per-second speed that handles resets
- Counters cannot be stored without rate
- rate() converts a counter into a histogram
Answer
The raw value only grows and resets on restart; rate turns it into a per-second speed that handles resets — A counter's absolute value depends on when the process started. Its rate of increase is what you care about, and rate() also corrects for resets.