# Alertmanager — Prometheus + Grafana

Source: https://www.skillbyai.com/en/prometheus-grafana/al-manager

> Routing, grouping, silencing, inhibition.

## From alerts to notifications

Prometheus sends firing alerts to **Alertmanager**, which **deduplicates** them (including from replicated Prometheus servers), **groups** related alerts into one notification (`group_by`), **routes** them through a tree of matchers to receivers (Slack, email, PagerDuty, Opsgenie, webhooks), applies **inhibition** (suppress warnings when a related critical alert fires) and honours **silences** created during maintenance. `group_wait`, `group_interval` and `repeat_interval` control timing.

## An alertmanager.yml routing tree

Pages go to on-call, the rest to Slack.

```yaml
route:
  receiver: slack-default
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: pagerduty-oncall

receivers:
  - name: slack-default
    slack_configs:
      - api_url_file: /etc/alertmanager/slack_url
        channel: "#alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/pd_key

inhibit_rules:
  - source_matchers: [severity="page"]
    target_matchers: [severity="ticket"]
    equal: [job]
```

## Keep secrets in files

Use *_file options for webhook URLs and keys rather than writing them into the configuration.

**Quiz:** What does inhibition do in Alertmanager?

- [ ] Speeds up scrapes
- [ ] Deletes alert rules
- [x] Suppresses some alerts while a related, more important alert is firing
- [ ] Creates dashboards

*Answer:* Suppresses some alerts while a related, more important alert is firing. Reduces noise during incidents.