Alertmanager

From alerts to notifications

Prometheus sends firing alerts to Alertmanager, which deduplicates them (including from replicated Prometheus servers), groups related alerts into one notification (group_by), routes them through a tree of matchers to receivers (Slack, email, PagerDuty, Opsgenie, webhooks), applies inhibition (suppress warnings when a related critical alert fires) and honours silences created during maintenance. group_wait, group_interval and repeat_interval control timing.

An alertmanager.yml routing tree

Pages go to on-call, the rest to Slack.

route:
  receiver: slack-default
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: pagerduty-oncall

receivers:
  - name: slack-default
    slack_configs:
      - api_url_file: /etc/alertmanager/slack_url
        channel: "#alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/pd_key

inhibit_rules:
  - source_matchers: [severity="page"]
    target_matchers: [severity="ticket"]
    equal: [job]

Keep secrets in files

Use *_file options for webhook URLs and keys rather than writing them into the configuration.

त्वरित जाँच: What does inhibition do in Alertmanager?

• Speeds up scrapes
• Deletes alert rules
• Suppresses some alerts while a related, more important alert is firing
• Creates dashboards