# Logging, Metrics, Tracing & Alerting — System Design: Architecture, Scale and Trade-offs

Source: https://www.skillbyai.com/en/system-design/sd-observability

> Monitor systems with logs, metrics and traces, track latency, traffic, errors and saturation, and alert on user-facing symptoms.

## The three pillars

**Logs** are timestamped discrete events (good for 'what exactly happened here'). **Metrics** are aggregated numbers over time (good for trends and dashboards — request rate, error rate, latency). **Traces** follow one request across every service it touched (good for finding where in a call chain the time went).

## The golden signals

For any service, watch **latency** (p50/p95/p99, not just average), **traffic** (requests/sec), **errors** (rate of failed requests), and **saturation** (how full is CPU, memory, queue depth). These four catch most incidents early.

## A car dashboard

Metrics are the speedometer and fuel gauge — glance and know your state. Logs are the trip computer's detailed event history. Tracing is like a black box recorder that shows the exact sequence of what each part of the car did leading up to a specific trip.

## Alert on symptoms, not causes

Alert on user-facing symptoms (error rate spiked, p99 latency breached SLO) rather than every internal cause (one server's CPU is high). Symptom-based alerts reduce noise and point straight at what users are experiencing.