Lesson 2 / 35

Logs, Metrics and Traces

Name the three main telemetry signals, what question each answers best, and what each costs to collect.

By · 15 years in web development

Three signals, three shapes of data

Logs are timestamped records of discrete events: "payment failed for order A17, gateway timeout". They carry rich detail but are the most expensive signal per event, because every line must be shipped, stored and indexed.

Metrics are numbers aggregated over time: requests per second, error count, p99 latency, queue length. A metric is cheap because a counter incremented a million times is still one number per scrape. The trade-off is that aggregation throws away detail: a metric tells you that errors rose, not which request failed.

Traces record the path of a single request across services as a tree of timed operations called spans. They answer "where did the time go?" and "which downstream call failed?".

A practical workflow uses all three: a metric alert says something is wrong, a trace shows which service is responsible, and logs from that service explain the exact cause. Newer setups add continuous profiles (where CPU and memory go inside a process) as a fourth signal.

Choosing the right signal

Ask what you need to do with the data:

  • Count or trend something (rate, ratio, percentile) -> metric.
  • Explain one specific event with full detail -> log.
  • Follow one request through many services -> trace.

A common mistake is computing error rates by counting log lines. It works at small scale but becomes slow and costly; emit a counter instead.

Cricket scorecard, commentary and ball-tracking

The scorecard (runs, wickets, run rate) is a metric: compact and great for trends. The commentary for each ball is a log: detailed but long. Hawk-Eye ball tracking is a trace: it follows one delivery from the bowler's hand to the batter, segment by segment. You need all three to understand why a match was lost.

Quick check: An alert fires because the 5xx error rate doubled. Which signal is the best next step to find which downstream service is slow for the failing requests?

  • Another metric with a higher scrape interval
  • Raw CPU usage of the load balancer
  • Distributed traces of the failing requests
  • The application's startup logs
Answer

Distributed traces of the failing requests — Traces show each hop of a request with timings and error status, so they quickly point to the slow or failing downstream service.