# Logs, Metrics and Traces — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-three-signals

> Name the three main telemetry signals, what question each answers best, and what each costs to collect.

## Three signals, three shapes of data

**Logs** are timestamped records of discrete events: "payment failed for order A17, gateway timeout". They carry rich detail but are the most expensive signal per event, because every line must be shipped, stored and indexed.

**Metrics** are numbers aggregated over time: requests per second, error count, p99 latency, queue length. A metric is cheap because a counter incremented a million times is still one number per scrape. The trade-off is that aggregation throws away detail: a metric tells you *that* errors rose, not *which* request failed.

**Traces** record the path of a single request across services as a tree of timed operations called **spans**. They answer "where did the time go?" and "which downstream call failed?".

A practical workflow uses all three: a metric alert says something is wrong, a trace shows which service is responsible, and logs from that service explain the exact cause. Newer setups add continuous **profiles** (where CPU and memory go inside a process) as a fourth signal.

## Choosing the right signal

Ask what you need to do with the data:
- **Count or trend** something (rate, ratio, percentile) -> metric.
- **Explain one specific event** with full detail -> log.
- **Follow one request** through many services -> trace.

A common mistake is computing error rates by counting log lines. It works at small scale but becomes slow and costly; emit a counter instead.

## Cricket scorecard, commentary and ball-tracking

The **scorecard** (runs, wickets, run rate) is a metric: compact and great for trends. The **commentary** for each ball is a log: detailed but long. **Hawk-Eye ball tracking** is a trace: it follows one delivery from the bowler's hand to the batter, segment by segment. You need all three to understand why a match was lost.

**Quiz:** An alert fires because the 5xx error rate doubled. Which signal is the best next step to find which downstream service is slow for the failing requests?

- [ ] Another metric with a higher scrape interval
- [ ] Raw CPU usage of the load balancer
- [x] Distributed traces of the failing requests
- [ ] The application's startup logs

*Answer:* Distributed traces of the failing requests. Traces show each hop of a request with timings and error status, so they quickly point to the slow or failing downstream service.
