# Routing, Grouping and Silencing Alerts — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-alertmanager

> Configure Alertmanager routes, grouping, inhibition and silences so one incident produces one useful notification.

## From many alerts to one notification

Prometheus decides **whether** an alert is firing; **Alertmanager** decides **who hears about it and how often**. Separating the two lets many Prometheus servers share one notification policy.

**Deduplication**: highly available Prometheus pairs send the same alert twice; Alertmanager sends it once. **Grouping** (`group_by`) bundles related alerts, so a database outage that breaks 40 services produces one notification listing them instead of 40 pages. `group_wait`, `group_interval` and `repeat_interval` control timing.

**Routing** is a tree: each alert walks down `routes` and stops at the first match unless `continue: true` is set, then goes to that route's **receiver** (email, chat, a paging service, a webhook). **Inhibition** mutes some alerts while another is firing: if the whole cluster is unreachable, there is no point paging for each service inside it. **Silences** are temporary, time-bounded mutes created by a person, for example during planned maintenance; always give them an end time and a comment.

## An Alertmanager configuration

Pages go to a paging service, tickets to a chat channel, everything else to email. While `ClusterUnreachable` fires, page-level alerts with the same `cluster` label are suppressed. Global settings such as SMTP details are omitted here.

```yaml
route:
  receiver: team-email
  group_by: ["alertname", "job"]
  group_wait: 30s        # wait to batch alerts that start together
  group_interval: 5m     # wait before notifying about new alerts in a group
  repeat_interval: 4h    # re-notify if still firing
  routes:
    - matchers:
        - severity = "page"
      receiver: oncall-pager
    - matchers:
        - severity = "ticket"
      receiver: team-chat

inhibit_rules:
  - source_matchers:
      - alertname = "ClusterUnreachable"
    target_matchers:
      - severity = "page"
    equal: ["cluster"]

receivers:
  - name: team-email
    email_configs:
      - to: "checkout-team@example.com"
  - name: oncall-pager
    pagerduty_configs:
      - routing_key: "<pagerduty-integration-key>"
  - name: team-chat
    slack_configs:
      - api_url: "<slack-webhook-url>"
        channel: "#checkout-alerts"
```

## Silences are not fixes

A silence with no end date is a deleted alert that nobody reviewed. Expire silences automatically, record why they were created, and open a ticket to fix or retune the underlying rule.

**Quiz:** A shared database fails and 30 dependent services all fire error alerts within a minute. Which Alertmanager feature most directly turns this into a single notification?

- [ ] Silences
- [x] Grouping
- [ ] repeat_interval
- [ ] Recording rules

*Answer:* Grouping. Grouping bundles alerts that share the `group_by` labels and arrive close together into one notification. Inhibition could also help if a database-down alert exists, but grouping is the direct mechanism.
