Golden Signals and SLO-Based Alerting

Know when users are hurting

Google's SRE book recommends four golden signals for user-facing systems: latency (including the latency of errors), traffic, errors and saturation (how full the most constrained resource is). Two popular checklists apply them: RED for request-driven services (Rate, Errors, Duration) and USE for resources (Utilisation, Saturation, Errors). Alerting on raw thresholds such as CPU > 80% produces noise. Better: alert on SLO burn rate, how fast the error budget is being consumed. A burn rate of 1 uses exactly the budget over the SLO window; the SRE workbook suggests paging when, for a 30-day window, 2% of the budget burns in an hour (burn rate 14.4) or 5% in six hours (burn rate 6), and opening a ticket for slower burns. Pair metrics with structured logs and distributed traces so on-call engineers can move from "something is wrong" to "this dependency, this release" quickly.

Signals feeding an alert

Latency, traffic, errors and saturation feed dashboards; SLO burn decides when to page.

A burn-rate alert in PromQL

Fires when the 1-hour error ratio burns a 99.9% SLO 14.4 times faster than sustainable, confirmed by a 5-minute window.

# error ratio over a window
#   sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
# / sum(rate(http_requests_total{job="checkout"}[1h]))

(
  sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
  / sum(rate(http_requests_total{job="checkout"}[1h]))
) > (14.4 * 0.001)
and
(
  sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="checkout"}[5m]))
) > (14.4 * 0.001)

Every page must be actionable

If an alert wakes someone up and the right response is "ignore it", delete or downgrade the alert. Alert fatigue makes people miss the alert that matters.

Quick check: Which set lists the four golden signals?

• Latency, traffic, errors, saturation
• CPU, memory, disk, network
• Rate, errors, duration, uptime
• Throughput, cost, size, age