Lesson 32 / 35
Cheat Sheet
Recall the core definitions and the right tool for each observability task in one quick pass before an exam or interview.
By Rahul Agarwal · 15 years in web development
The whole course on one page
- Observability: answering new questions about internal state from emitted telemetry. Monitoring watches known conditions.
- Logs: detailed events; make them structured JSON with a correlation or trace ID; never log secrets.
- Metrics: cheap aggregated numbers. Counter (only up, use
rate), gauge (up and down), histogram (buckets, fleet percentiles), summary (client-side quantiles, not aggregatable). - Cardinality: series = product of label values; never label with IDs.
- PromQL:
ratebeforesum;histogram_quantileoversum by (le);upandabsent()for blind spots. - Traces: tree of spans sharing a trace ID; W3C
traceparentcarries context; head sampling is cheap, tail sampling keeps errors. - OpenTelemetry: API, SDK, instrumentation, OTLP, Collector (receivers, processors, exporters).
- SLO: target for a good/total SLI ratio; error budget = 1 - SLO; burn rate = error ratio / budget.
- Alerts: page on symptoms, with runbooks; group and inhibit in Alertmanager; review pages weekly.
The full loop
Revision in one picture: instrument code, collect the signals, correlate them by shared IDs, track SLOs, and alert humans who then improve the instrumentation.
Which tool for which job
A lookup table to rehearse until you can fill it in from memory.
need use
------------------------------------- ---------------------------------------------
count / rate / ratio over time counter + rate()
current level that rises and falls gauge
latency percentiles across replicas histogram + histogram_quantile(sum by (le))
explain one failure in detail structured log with trace_id
follow a request across services trace (spans, traceparent propagation)
reduce trace volume, keep errors tail sampling in the Collector
user-facing reliability target SLI ratio + SLO + error budget
page only when it matters multi-window burn-rate alert
one notification per incident Alertmanager grouping + inhibitionQuick check: Which pairing is correct?
- Gauge: total requests since startup
- Summary: percentiles that can be aggregated across many pods
- Counter with rate(): requests per second
- Head sampling: keeps all error traces by design
Answer
Counter with rate(): requests per second — A counter with rate() gives a per-second rate. Totals since startup are counters, summaries cannot be aggregated across pods, and head sampling decides before errors are known.