पाठ 27 / 35
Routing, Grouping and Silencing Alerts
यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।
Configure Alertmanager routes, grouping, inhibition and silences so one incident produces one useful notification.
लेखक: Rahul Agarwal · 15 साल से वेब डेवलपर
From many alerts to one notification
Prometheus decides whether an alert is firing; Alertmanager decides who hears about it and how often. Separating the two lets many Prometheus servers share one notification policy.
Deduplication: highly available Prometheus pairs send the same alert twice; Alertmanager sends it once. Grouping (group_by) bundles related alerts, so a database outage that breaks 40 services produces one notification listing them instead of 40 pages. group_wait, group_interval and repeat_interval control timing.
Routing is a tree: each alert walks down routes and stops at the first match unless continue: true is set, then goes to that route's receiver (email, chat, a paging service, a webhook). Inhibition mutes some alerts while another is firing: if the whole cluster is unreachable, there is no point paging for each service inside it. Silences are temporary, time-bounded mutes created by a person, for example during planned maintenance; always give them an end time and a comment.
An Alertmanager configuration
Pages go to a paging service, tickets to a chat channel, everything else to email. While ClusterUnreachable fires, page-level alerts with the same cluster label are suppressed. Global settings such as SMTP details are omitted here.
route:
receiver: team-email
group_by: ["alertname", "job"]
group_wait: 30s # wait to batch alerts that start together
group_interval: 5m # wait before notifying about new alerts in a group
repeat_interval: 4h # re-notify if still firing
routes:
- matchers:
- severity = "page"
receiver: oncall-pager
- matchers:
- severity = "ticket"
receiver: team-chat
inhibit_rules:
- source_matchers:
- alertname = "ClusterUnreachable"
target_matchers:
- severity = "page"
equal: ["cluster"]
receivers:
- name: team-email
email_configs:
- to: "checkout-team@example.com"
- name: oncall-pager
pagerduty_configs:
- routing_key: "<pagerduty-integration-key>"
- name: team-chat
slack_configs:
- api_url: "<slack-webhook-url>"
channel: "#checkout-alerts"Silences are not fixes
A silence with no end date is a deleted alert that nobody reviewed. Expire silences automatically, record why they were created, and open a ticket to fix or retune the underlying rule.
त्वरित जाँच: A shared database fails and 30 dependent services all fire error alerts within a minute. Which Alertmanager feature most directly turns this into a single notification?
- Silences
- Grouping
- repeat_interval
- Recording rules
Answer
Grouping — Grouping bundles alerts that share the `group_by` labels and arrive close together into one notification. Inhibition could also help if a database-down alert exists, but grouping is the direct mechanism.