Lesson 27 / 35

Routing, Grouping and Silencing Alerts

Configure Alertmanager routes, grouping, inhibition and silences so one incident produces one useful notification.

By · 15 years in web development

From many alerts to one notification

Prometheus decides whether an alert is firing; Alertmanager decides who hears about it and how often. Separating the two lets many Prometheus servers share one notification policy.

Deduplication: highly available Prometheus pairs send the same alert twice; Alertmanager sends it once. Grouping (group_by) bundles related alerts, so a database outage that breaks 40 services produces one notification listing them instead of 40 pages. group_wait, group_interval and repeat_interval control timing.

Routing is a tree: each alert walks down routes and stops at the first match unless continue: true is set, then goes to that route's receiver (email, chat, a paging service, a webhook). Inhibition mutes some alerts while another is firing: if the whole cluster is unreachable, there is no point paging for each service inside it. Silences are temporary, time-bounded mutes created by a person, for example during planned maintenance; always give them an end time and a comment.

An Alertmanager configuration

Pages go to a paging service, tickets to a chat channel, everything else to email. While ClusterUnreachable fires, page-level alerts with the same cluster label are suppressed. Global settings such as SMTP details are omitted here.

route:
  receiver: team-email
  group_by: ["alertname", "job"]
  group_wait: 30s        # wait to batch alerts that start together
  group_interval: 5m     # wait before notifying about new alerts in a group
  repeat_interval: 4h    # re-notify if still firing
  routes:
    - matchers:
        - severity = "page"
      receiver: oncall-pager
    - matchers:
        - severity = "ticket"
      receiver: team-chat

inhibit_rules:
  - source_matchers:
      - alertname = "ClusterUnreachable"
    target_matchers:
      - severity = "page"
    equal: ["cluster"]

receivers:
  - name: team-email
    email_configs:
      - to: "checkout-team@example.com"
  - name: oncall-pager
    pagerduty_configs:
      - routing_key: "<pagerduty-integration-key>"
  - name: team-chat
    slack_configs:
      - api_url: "<slack-webhook-url>"
        channel: "#checkout-alerts"

Silences are not fixes

A silence with no end date is a deleted alert that nobody reviewed. Expire silences automatically, record why they were created, and open a ticket to fix or retune the underlying rule.

Quick check: A shared database fails and 30 dependent services all fire error alerts within a minute. Which Alertmanager feature most directly turns this into a single notification?

  • Silences
  • Grouping
  • repeat_interval
  • Recording rules
Answer

Grouping — Grouping bundles alerts that share the `group_by` labels and arrive close together into one notification. Inhibition could also help if a database-down alert exists, but grouping is the direct mechanism.