Alerting Rules

expr, for, labels and annotations

An alerting rule has a PromQL expr that returns series when something is wrong, a for duration the condition must hold (to avoid alerts on brief blips; the alert is pending until then and firing after), labels such as severity used for routing, and annotations with a human-readable summary, description and runbook link. Templates like {{ $labels.instance }} and {{ $value }} fill in details.

Wake people only when it matters

Alerting rules detect problems, Alertmanager routes them, and good alert design prevents fatigue.

Two alerting rules

Error ratio and a down target.

groups:
  - name: orders-api-alerts
    rules:
      - alert: OrdersApiHighErrorRatio
        expr: job:http_requests_error_ratio:rate5m{job="orders-api"} > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Orders API error ratio above 5%"
          description: "Error ratio is {{ $value | humanizePercentage }} for 10 minutes."
          runbook_url: "https://runbooks.example.com/orders-api/high-errors"
      - alert: TargetDown
        expr: up == 0
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.job }} target {{ $labels.instance }} is down"

Every paging alert needs a runbook

A link explaining what to check and how to mitigate shortens incidents, especially at night.

Quick check: What does the for clause do in an alerting rule?

• Sets the scrape interval
• Repeats the notification
• Requires the condition to stay true for that long before firing
• Deletes the alert after that time