Lesson 12 / 35
RED, USE and the Four Golden Signals
Apply the RED, USE and golden-signals checklists to decide which metrics a service or resource must expose.
By Rahul Agarwal · 15 years in web development
Checklists instead of guesswork
Teams new to metrics either measure nothing or measure everything. Three well-known checklists help.
RED (for request-driven services): Rate (requests per second), Errors (failed requests per second or as a ratio), Duration (latency distribution, usually as a histogram). If every service exposes RED, any dashboard can compare them.
USE (for resources such as CPU, disks, network links, connection pools): Utilisation (how busy, as a percentage of time or capacity), Saturation (how much work is waiting: run-queue length, pool wait time), Errors (device or resource errors). Saturation is often the earliest sign of trouble: a pool at 100% utilisation with a growing queue means requests are now waiting.
The four golden signals from Google's SRE book combine both: latency, traffic, errors, saturation. One subtle point it highlights: measure the latency of failed requests separately, because a fast error can make average latency look better while users suffer.
A railway ticket counter
For the counter as a service: how many people are served per minute (rate), how many leave without a ticket (errors), how long each takes (duration). For the clerk as a resource: what fraction of time they are busy (utilisation), how long the queue is (saturation), how often the printer jams (errors). A clerk can be 100% busy and still fine; a queue stretching out of the station is the real alarm.
RED in PromQL
Three queries give a RED view of one service, using the counter and histogram from earlier in this course.
# Rate: requests per second
sum(rate(http_requests_total{job="checkout"}[5m]))
# Errors: fraction of requests returning 5xx
sum(rate(http_requests_total{job="checkout", status=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="checkout"}[5m]))
# Duration: 95th percentile latency in seconds
histogram_quantile(0.95,
sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))Quick check: In the USE method, which measurement describes saturation of a database connection pool?
- The number of queries that returned an error
- The number of requests waiting to obtain a connection
- The total queries executed since startup
- The size of the database on disk
Answer
The number of requests waiting to obtain a connection — Saturation is queued or waiting work. Requests waiting for a free connection show the pool cannot keep up, even before errors appear.