# Designing Alerts Worth Waking Up For — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-alert-design

> Write actionable, symptom-based alert rules with sensible severities, for-durations and runbook links.

## Page on symptoms, investigate causes

An alert interrupts a human, so every alert should pass one test: **is there something a person must do now?** If not, it should be a ticket, a dashboard panel, or nothing.

**Symptom-based** alerts describe what users experience: errors are up, checkout is slow, the error budget is burning. **Cause-based** alerts describe internal conditions: CPU at 90%, one pod restarted, a disk at 70%. Causes are often harmless (high CPU during a healthy traffic peak) and incomplete (many failures have no matching cause alert). So page on symptoms, and keep cause signals on dashboards and in lower-severity tickets that help diagnosis. Exceptions exist for causes that predict a certain outage, such as a disk that will be full in four hours.

In Prometheus, an alerting rule has an `expr`; while it returns results, the alert is **pending**, and after the `for:` duration it becomes **firing** and is sent to Alertmanager. `for:` filters out blips. **Labels** such as `severity` drive routing, and **annotations** carry a human summary and a **runbook link** explaining what to check first.

## The life of an alert

A **rule** is evaluated on a schedule, a matching condition makes the alert **pending**, after the `for` duration it is **firing**, **Alertmanager** groups and routes it, and the **on-call** engineer is notified.

![Five stages in order: an alert rule is evaluated, the alert becomes pending, then firing, Alertmanager groups and routes it, the on-call engineer is notified.](assets/figures/observability/section-9-map.svg) — Figure 9.1: rule -> pending -> firing -> Alertmanager -> on-call

## Prometheus rules: record, then alert

Recording rules precompute the error ratio over two windows; the alert pages only if both exceed 14.4 times the 0.1% budget. A second alert catches the case where the service cannot be scraped at all.

```yaml
groups:
  - name: checkout-slo
    rules:
      - record: job:http_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_errors:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[1h]))
            / sum by (job) (rate(http_requests_total[1h]))

      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          job:http_errors:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
          and
          job:http_errors:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Checkout is burning its 99.9% error budget more than 14x faster than sustainable"
          runbook_url: "https://runbooks.example.com/checkout/high-error-rate"

      - alert: CheckoutTargetDown
        expr: up{job="checkout"} == 0
        for: 5m
        labels:
          severity: page
```

## Signs of alert fatigue

If the team acknowledges alerts without acting, if the same alert fires every night and resolves itself, or if people mute the alert channel, the alerts are training everyone to ignore pages. Review every page in a weekly meeting: fix the cause, tune the rule, downgrade it to a ticket or delete it.

**Quiz:** Which of these is the best candidate for a page?

- [ ] CPU usage on one node is at 85%
- [ ] A single pod restarted once and is healthy
- [ ] Disk usage reached 60%
- [x] Checkout's error budget is burning 14x faster than planned over the last hour

*Answer:* Checkout's error budget is burning 14x faster than planned over the last hour. The burn-rate alert is a user-facing symptom that needs action now. The others are causes that are often harmless and belong on dashboards or in tickets.
