Lesson 12 / 35

RED, USE and the Four Golden Signals

Apply the RED, USE and golden-signals checklists to decide which metrics a service or resource must expose.

By · 15 years in web development

Checklists instead of guesswork

Teams new to metrics either measure nothing or measure everything. Three well-known checklists help.

RED (for request-driven services): Rate (requests per second), Errors (failed requests per second or as a ratio), Duration (latency distribution, usually as a histogram). If every service exposes RED, any dashboard can compare them.

USE (for resources such as CPU, disks, network links, connection pools): Utilisation (how busy, as a percentage of time or capacity), Saturation (how much work is waiting: run-queue length, pool wait time), Errors (device or resource errors). Saturation is often the earliest sign of trouble: a pool at 100% utilisation with a growing queue means requests are now waiting.

The four golden signals from Google's SRE book combine both: latency, traffic, errors, saturation. One subtle point it highlights: measure the latency of failed requests separately, because a fast error can make average latency look better while users suffer.

A railway ticket counter

For the counter as a service: how many people are served per minute (rate), how many leave without a ticket (errors), how long each takes (duration). For the clerk as a resource: what fraction of time they are busy (utilisation), how long the queue is (saturation), how often the printer jams (errors). A clerk can be 100% busy and still fine; a queue stretching out of the station is the real alarm.

RED in PromQL

Three queries give a RED view of one service, using the counter and histogram from earlier in this course.

# Rate: requests per second
sum(rate(http_requests_total{job="checkout"}[5m]))

# Errors: fraction of requests returning 5xx
sum(rate(http_requests_total{job="checkout", status=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="checkout"}[5m]))

# Duration: 95th percentile latency in seconds
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))

Quick check: In the USE method, which measurement describes saturation of a database connection pool?

  • The number of queries that returned an error
  • The number of requests waiting to obtain a connection
  • The total queries executed since startup
  • The size of the database on disk
Answer

The number of requests waiting to obtain a connection — Saturation is queued or waiting work. Requests waiting for a free connection show the pool cannot keep up, even before errors appear.