Lesson 27 / 35
Routing, Grouping and Silencing Alerts
Configure Alertmanager routes, grouping, inhibition and silences so one incident produces one useful notification.
By Rahul Agarwal · 15 years in web development
From many alerts to one notification
Prometheus decides whether an alert is firing; Alertmanager decides who hears about it and how often. Separating the two lets many Prometheus servers share one notification policy.
Deduplication: highly available Prometheus pairs send the same alert twice; Alertmanager sends it once. Grouping (group_by) bundles related alerts, so a database outage that breaks 40 services produces one notification listing them instead of 40 pages. group_wait, group_interval and repeat_interval control timing.
Routing is a tree: each alert walks down routes and stops at the first match unless continue: true is set, then goes to that route's receiver (email, chat, a paging service, a webhook). Inhibition mutes some alerts while another is firing: if the whole cluster is unreachable, there is no point paging for each service inside it. Silences are temporary, time-bounded mutes created by a person, for example during planned maintenance; always give them an end time and a comment.
An Alertmanager configuration
Pages go to a paging service, tickets to a chat channel, everything else to email. While ClusterUnreachable fires, page-level alerts with the same cluster label are suppressed. Global settings such as SMTP details are omitted here.
route:
receiver: team-email
group_by: ["alertname", "job"]
group_wait: 30s # wait to batch alerts that start together
group_interval: 5m # wait before notifying about new alerts in a group
repeat_interval: 4h # re-notify if still firing
routes:
- matchers:
- severity = "page"
receiver: oncall-pager
- matchers:
- severity = "ticket"
receiver: team-chat
inhibit_rules:
- source_matchers:
- alertname = "ClusterUnreachable"
target_matchers:
- severity = "page"
equal: ["cluster"]
receivers:
- name: team-email
email_configs:
- to: "checkout-team@example.com"
- name: oncall-pager
pagerduty_configs:
- routing_key: "<pagerduty-integration-key>"
- name: team-chat
slack_configs:
- api_url: "<slack-webhook-url>"
channel: "#checkout-alerts"Silences are not fixes
A silence with no end date is a deleted alert that nobody reviewed. Expire silences automatically, record why they were created, and open a ticket to fix or retune the underlying rule.
Quick check: A shared database fails and 30 dependent services all fire error alerts within a minute. Which Alertmanager feature most directly turns this into a single notification?
- Silences
- Grouping
- repeat_interval
- Recording rules
Answer
Grouping — Grouping bundles alerts that share the `group_by` labels and arrive close together into one notification. Inhibition could also help if a database-down alert exists, but grouping is the direct mechanism.