# Cheat Sheet — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-cheat-sheet

> Recall the core definitions and the right tool for each observability task in one quick pass before an exam or interview.

## The whole course on one page

- **Observability**: answering new questions about internal state from emitted telemetry. Monitoring watches known conditions.
- **Logs**: detailed events; make them structured JSON with a correlation or trace ID; never log secrets.
- **Metrics**: cheap aggregated numbers. Counter (only up, use `rate`), gauge (up and down), histogram (buckets, fleet percentiles), summary (client-side quantiles, not aggregatable).
- **Cardinality**: series = product of label values; never label with IDs.
- **PromQL**: `rate` before `sum`; `histogram_quantile` over `sum by (le)`; `up` and `absent()` for blind spots.
- **Traces**: tree of spans sharing a trace ID; W3C `traceparent` carries context; head sampling is cheap, tail sampling keeps errors.
- **OpenTelemetry**: API, SDK, instrumentation, OTLP, Collector (receivers, processors, exporters).
- **SLO**: target for a good/total SLI ratio; error budget = 1 - SLO; burn rate = error ratio / budget.
- **Alerts**: page on symptoms, with runbooks; group and inhibit in Alertmanager; review pages weekly.

## The full loop

Revision in one picture: **instrument** code, **collect** the signals, **correlate** them by shared IDs, track **SLOs**, and **alert** humans who then improve the instrumentation.

![Five stages in order: instrument, collect, correlate signals, track SLOs, alert humans.](assets/figures/observability/section-11-map.svg) — Figure 11.1: instrument -> collect -> correlate -> SLOs -> alert

## Which tool for which job

A lookup table to rehearse until you can fill it in from memory.

```text
need                                   use
-------------------------------------  ---------------------------------------------
count / rate / ratio over time         counter + rate()
current level that rises and falls     gauge
latency percentiles across replicas    histogram + histogram_quantile(sum by (le))
explain one failure in detail          structured log with trace_id
follow a request across services       trace (spans, traceparent propagation)
reduce trace volume, keep errors       tail sampling in the Collector
user-facing reliability target         SLI ratio + SLO + error budget
page only when it matters              multi-window burn-rate alert
one notification per incident          Alertmanager grouping + inhibition
```

**Quiz:** Which pairing is correct?

- [ ] Gauge: total requests since startup
- [ ] Summary: percentiles that can be aggregated across many pods
- [x] Counter with rate(): requests per second
- [ ] Head sampling: keeps all error traces by design

*Answer:* Counter with rate(): requests per second. A counter with rate() gives a per-second rate. Totals since startup are counters, summaries cannot be aggregated across pods, and head sampling decides before errors are known.
