पाठ 26 / 35

Designing Alerts Worth Waking Up For

यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।

Write actionable, symptom-based alert rules with sensible severities, for-durations and runbook links.

लेखक: · 15 साल से वेब डेवलपर

Page on symptoms, investigate causes

An alert interrupts a human, so every alert should pass one test: is there something a person must do now? If not, it should be a ticket, a dashboard panel, or nothing.

Symptom-based alerts describe what users experience: errors are up, checkout is slow, the error budget is burning. Cause-based alerts describe internal conditions: CPU at 90%, one pod restarted, a disk at 70%. Causes are often harmless (high CPU during a healthy traffic peak) and incomplete (many failures have no matching cause alert). So page on symptoms, and keep cause signals on dashboards and in lower-severity tickets that help diagnosis. Exceptions exist for causes that predict a certain outage, such as a disk that will be full in four hours.

In Prometheus, an alerting rule has an expr; while it returns results, the alert is pending, and after the for: duration it becomes firing and is sent to Alertmanager. for: filters out blips. Labels such as severity drive routing, and annotations carry a human summary and a runbook link explaining what to check first.

The life of an alert

A rule is evaluated on a schedule, a matching condition makes the alert pending, after the for duration it is firing, Alertmanager groups and routes it, and the on-call engineer is notified.

Five stages in order: an alert rule is evaluated, the alert becomes pending, then firing, Alertmanager groups and routes it, the on-call engineer is notified.
Figure 9.1: rule -> pending -> firing -> Alertmanager -> on-call

Prometheus rules: record, then alert

Recording rules precompute the error ratio over two windows; the alert pages only if both exceed 14.4 times the 0.1% budget. A second alert catches the case where the service cannot be scraped at all.

groups:
  - name: checkout-slo
    rules:
      - record: job:http_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_errors:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[1h]))
            / sum by (job) (rate(http_requests_total[1h]))

      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          job:http_errors:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
          and
          job:http_errors:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Checkout is burning its 99.9% error budget more than 14x faster than sustainable"
          runbook_url: "https://runbooks.example.com/checkout/high-error-rate"

      - alert: CheckoutTargetDown
        expr: up{job="checkout"} == 0
        for: 5m
        labels:
          severity: page

Signs of alert fatigue

If the team acknowledges alerts without acting, if the same alert fires every night and resolves itself, or if people mute the alert channel, the alerts are training everyone to ignore pages. Review every page in a weekly meeting: fix the cause, tune the rule, downgrade it to a ticket or delete it.

त्वरित जाँच: Which of these is the best candidate for a page?

  • CPU usage on one node is at 85%
  • A single pod restarted once and is healthy
  • Disk usage reached 60%
  • Checkout's error budget is burning 14x faster than planned over the last hour
Answer

Checkout's error budget is burning 14x faster than planned over the last hour — The burn-rate alert is a user-facing symptom that needs action now. The others are causes that are often harmless and belong on dashboards or in tickets.