# Common Observability Pitfalls — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-pitfalls

> Recognise the mistakes teams repeat with telemetry and know the corrective habit for each one.

## Mistakes that look fine until an incident

**Graphing only averages.** The mean hides the slow tail; show p50, p95 and p99. **Alerting on everything.** Too many cause-based pages lead to fatigue and ignored alerts. **No timeouts.** A call without a timeout never produces an error metric; it just hangs, and dashboards look calm while users wait.

**Silent monitoring failures.** When a target stops being scraped, its error-rate alert evaluates to nothing instead of firing. Alert on `up == 0` and use `absent()` for metrics that must exist. Also monitor the observability stack itself: collector queue drops, ingestion errors, rule-evaluation failures.

**Inconsistent naming.** `service`, `svc` and `app` used for the same idea make cross-signal queries impossible; agree on conventions early.

**Clock and time-zone confusion.** Store and display timestamps in UTC, keep hosts synchronised with NTP, and remember that spans from different machines can appear slightly out of order because of clock skew.

**Instrumenting after the outage.** Add RED metrics, structured logs and tracing when a service is created, not during the first incident.

## absent() catches the missing metric

`absent(up{job="checkout"})` returns a value only when no matching series exists at all, for example after a relabelling mistake removes the whole job. Pair it with `up == 0`, which covers targets that exist but fail to scrape.

## The smoke detector with no battery

A smoke detector with a dead battery is silent, and silence looks exactly like safety. Monitoring that has stopped collecting data behaves the same way. That is why detectors chirp when the battery is low, and why your monitoring needs alerts about its own health.

**Quiz:** A service's error-rate alert never fired during an outage because Prometheus could not scrape the service at all. Which addition would have caught it?

- [ ] A lower error-rate threshold
- [ ] A longer `for:` duration
- [x] Alerts on `up == 0` and `absent()` for the job
- [ ] A dashboard with more panels

*Answer:* Alerts on `up == 0` and `absent()` for the job. With no scraped data, the error-rate expression returns nothing. `up == 0` catches failed scrapes and `absent()` catches series that disappear entirely.
