Lesson 32 / 35

Cheat Sheet

Recall the core definitions and the right tool for each observability task in one quick pass before an exam or interview.

By · 15 years in web development

The whole course on one page

  • Observability: answering new questions about internal state from emitted telemetry. Monitoring watches known conditions.
  • Logs: detailed events; make them structured JSON with a correlation or trace ID; never log secrets.
  • Metrics: cheap aggregated numbers. Counter (only up, use rate), gauge (up and down), histogram (buckets, fleet percentiles), summary (client-side quantiles, not aggregatable).
  • Cardinality: series = product of label values; never label with IDs.
  • PromQL: rate before sum; histogram_quantile over sum by (le); up and absent() for blind spots.
  • Traces: tree of spans sharing a trace ID; W3C traceparent carries context; head sampling is cheap, tail sampling keeps errors.
  • OpenTelemetry: API, SDK, instrumentation, OTLP, Collector (receivers, processors, exporters).
  • SLO: target for a good/total SLI ratio; error budget = 1 - SLO; burn rate = error ratio / budget.
  • Alerts: page on symptoms, with runbooks; group and inhibit in Alertmanager; review pages weekly.

The full loop

Revision in one picture: instrument code, collect the signals, correlate them by shared IDs, track SLOs, and alert humans who then improve the instrumentation.

Five stages in order: instrument, collect, correlate signals, track SLOs, alert humans.
Figure 11.1: instrument -> collect -> correlate -> SLOs -> alert

Which tool for which job

A lookup table to rehearse until you can fill it in from memory.

need                                   use
-------------------------------------  ---------------------------------------------
count / rate / ratio over time         counter + rate()
current level that rises and falls     gauge
latency percentiles across replicas    histogram + histogram_quantile(sum by (le))
explain one failure in detail          structured log with trace_id
follow a request across services       trace (spans, traceparent propagation)
reduce trace volume, keep errors       tail sampling in the Collector
user-facing reliability target         SLI ratio + SLO + error budget
page only when it matters              multi-window burn-rate alert
one notification per incident          Alertmanager grouping + inhibition

Quick check: Which pairing is correct?

  • Gauge: total requests since startup
  • Summary: percentiles that can be aggregated across many pods
  • Counter with rate(): requests per second
  • Head sampling: keeps all error traces by design
Answer

Counter with rate(): requests per second — A counter with rate() gives a per-second rate. Totals since startup are counters, summaries cannot be aggregated across pods, and head sampling decides before errors are known.