पाठ 27 / 35

Routing, Grouping and Silencing Alerts

यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।

Configure Alertmanager routes, grouping, inhibition and silences so one incident produces one useful notification.

लेखक: · 15 साल से वेब डेवलपर

From many alerts to one notification

Prometheus decides whether an alert is firing; Alertmanager decides who hears about it and how often. Separating the two lets many Prometheus servers share one notification policy.

Deduplication: highly available Prometheus pairs send the same alert twice; Alertmanager sends it once. Grouping (group_by) bundles related alerts, so a database outage that breaks 40 services produces one notification listing them instead of 40 pages. group_wait, group_interval and repeat_interval control timing.

Routing is a tree: each alert walks down routes and stops at the first match unless continue: true is set, then goes to that route's receiver (email, chat, a paging service, a webhook). Inhibition mutes some alerts while another is firing: if the whole cluster is unreachable, there is no point paging for each service inside it. Silences are temporary, time-bounded mutes created by a person, for example during planned maintenance; always give them an end time and a comment.

An Alertmanager configuration

Pages go to a paging service, tickets to a chat channel, everything else to email. While ClusterUnreachable fires, page-level alerts with the same cluster label are suppressed. Global settings such as SMTP details are omitted here.

route:
  receiver: team-email
  group_by: ["alertname", "job"]
  group_wait: 30s        # wait to batch alerts that start together
  group_interval: 5m     # wait before notifying about new alerts in a group
  repeat_interval: 4h    # re-notify if still firing
  routes:
    - matchers:
        - severity = "page"
      receiver: oncall-pager
    - matchers:
        - severity = "ticket"
      receiver: team-chat

inhibit_rules:
  - source_matchers:
      - alertname = "ClusterUnreachable"
    target_matchers:
      - severity = "page"
    equal: ["cluster"]

receivers:
  - name: team-email
    email_configs:
      - to: "checkout-team@example.com"
  - name: oncall-pager
    pagerduty_configs:
      - routing_key: "<pagerduty-integration-key>"
  - name: team-chat
    slack_configs:
      - api_url: "<slack-webhook-url>"
        channel: "#checkout-alerts"

Silences are not fixes

A silence with no end date is a deleted alert that nobody reviewed. Expire silences automatically, record why they were created, and open a ticket to fix or retune the underlying rule.

त्वरित जाँच: A shared database fails and 30 dependent services all fire error alerts within a minute. Which Alertmanager feature most directly turns this into a single notification?

  • Silences
  • Grouping
  • repeat_interval
  • Recording rules
Answer

Grouping — Grouping bundles alerts that share the `group_by` labels and arrive close together into one notification. Inhibition could also help if a database-down alert exists, but grouping is the direct mechanism.