Lesson 26 / 35
Designing Alerts Worth Waking Up For
Write actionable, symptom-based alert rules with sensible severities, for-durations and runbook links.
By Rahul Agarwal · 15 years in web development
Page on symptoms, investigate causes
An alert interrupts a human, so every alert should pass one test: is there something a person must do now? If not, it should be a ticket, a dashboard panel, or nothing.
Symptom-based alerts describe what users experience: errors are up, checkout is slow, the error budget is burning. Cause-based alerts describe internal conditions: CPU at 90%, one pod restarted, a disk at 70%. Causes are often harmless (high CPU during a healthy traffic peak) and incomplete (many failures have no matching cause alert). So page on symptoms, and keep cause signals on dashboards and in lower-severity tickets that help diagnosis. Exceptions exist for causes that predict a certain outage, such as a disk that will be full in four hours.
In Prometheus, an alerting rule has an expr; while it returns results, the alert is pending, and after the for: duration it becomes firing and is sent to Alertmanager. for: filters out blips. Labels such as severity drive routing, and annotations carry a human summary and a runbook link explaining what to check first.
The life of an alert
A rule is evaluated on a schedule, a matching condition makes the alert pending, after the for duration it is firing, Alertmanager groups and routes it, and the on-call engineer is notified.
Prometheus rules: record, then alert
Recording rules precompute the error ratio over two windows; the alert pages only if both exceed 14.4 times the 0.1% budget. A second alert catches the case where the service cannot be scraped at all.
groups:
- name: checkout-slo
rules:
- record: job:http_errors:ratio_rate5m
expr: |
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (job) (rate(http_requests_total[5m]))
- record: job:http_errors:ratio_rate1h
expr: |
sum by (job) (rate(http_requests_total{status=~"5.."}[1h]))
/ sum by (job) (rate(http_requests_total[1h]))
- alert: CheckoutErrorBudgetFastBurn
expr: |
job:http_errors:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
and
job:http_errors:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
for: 2m
labels:
severity: page
annotations:
summary: "Checkout is burning its 99.9% error budget more than 14x faster than sustainable"
runbook_url: "https://runbooks.example.com/checkout/high-error-rate"
- alert: CheckoutTargetDown
expr: up{job="checkout"} == 0
for: 5m
labels:
severity: pageSigns of alert fatigue
If the team acknowledges alerts without acting, if the same alert fires every night and resolves itself, or if people mute the alert channel, the alerts are training everyone to ignore pages. Review every page in a weekly meeting: fix the cause, tune the rule, downgrade it to a ticket or delete it.
Quick check: Which of these is the best candidate for a page?
- CPU usage on one node is at 85%
- A single pod restarted once and is healthy
- Disk usage reached 60%
- Checkout's error budget is burning 14x faster than planned over the last hour
Answer
Checkout's error budget is burning 14x faster than planned over the last hour — The burn-rate alert is a user-facing symptom that needs action now. The others are causes that are often harmless and belong on dashboards or in tickets.