# Cost and Performance Overhead — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-cost-overhead

> Identify what drives telemetry cost and overhead, and apply sampling, aggregation and retention to control it.

## Where the money and CPU go

Telemetry has two kinds of cost. **Runtime overhead** is the CPU, memory and network your instrumentation consumes inside the application: formatting log lines, recording spans, serialising and exporting. It is usually small when done right (batched, asynchronous, with levels and sampling) but can grow large with debug logging in hot loops or spans around tiny functions.

**Backend cost** is driven by four levers. **Volume**: bytes of logs and number of spans ingested. **Cardinality**: number of active metric series. **Retention**: how long data is kept at full resolution. **Query load**: dashboards that refresh every few seconds with heavy queries across months of data.

Each lever has a fix. Reduce volume with levels, sampling and dropping noise in the agent. Control cardinality with label reviews and by dropping unused series at scrape time with `metric_relabel_configs`. Use **tiered retention**: detailed data for days, downsampled or aggregated data for months. Precompute expensive expressions with **recording rules**. Review cost regularly by service or team, so the people who emit data see what it costs.

## Four cost levers

Telemetry cost grows with **volume**, **cardinality**, **retention** and **query load**. Tune each lever to arrive at an affordable **budget** without losing the signals you need.

![Five stages in order: volume, cardinality, retention and query load, the four levers that together determine the telemetry budget.](assets/figures/observability/section-10-map.svg) — Figure 10.1: volume -> cardinality -> retention -> query load -> budget

## Finding cardinality hot spots

PromQL queries that reveal which metrics and labels create the most series. The first one touches every series, so run it sparingly. Prometheus's own `/tsdb-status` page shows similar statistics.

```promql
# Which metric names have the most series? (expensive: run rarely, on a small range)
topk(10, count by (__name__) ({__name__=~".+"}))

# Which label of one metric has the most distinct values?
count(count by (route) (http_requests_total))
count(count by (instance) (http_requests_total))

# Samples ingested per second by this Prometheus (from its own metrics)
rate(prometheus_tsdb_head_samples_appended_total[5m])
```

## Measure the overhead, do not guess

Benchmark a request path with and without instrumentation, at your production sampling rate. If overhead is noticeable, the usual culprits are synchronous exporters, debug logs left on, string formatting of large objects, and spans or metrics inside very tight loops.

**Quiz:** A team's metrics bill doubled after a release that added a `session_id` label to a request counter. What is the most effective fix?

- [ ] Increase the scrape interval
- [ ] Shorten log retention
- [ ] Switch dashboards to dark mode
- [x] Remove the session_id label and keep session details in logs or traces

*Answer:* Remove the session_id label and keep session details in logs or traces. The cost came from cardinality: every session created new series. Removing the unbounded label fixes the cause; scrape interval changes only reduce samples, not series.
