Lesson 29 / 35
Cost and Performance Overhead
Identify what drives telemetry cost and overhead, and apply sampling, aggregation and retention to control it.
By Rahul Agarwal · 15 years in web development
Where the money and CPU go
Telemetry has two kinds of cost. Runtime overhead is the CPU, memory and network your instrumentation consumes inside the application: formatting log lines, recording spans, serialising and exporting. It is usually small when done right (batched, asynchronous, with levels and sampling) but can grow large with debug logging in hot loops or spans around tiny functions.
Backend cost is driven by four levers. Volume: bytes of logs and number of spans ingested. Cardinality: number of active metric series. Retention: how long data is kept at full resolution. Query load: dashboards that refresh every few seconds with heavy queries across months of data.
Each lever has a fix. Reduce volume with levels, sampling and dropping noise in the agent. Control cardinality with label reviews and by dropping unused series at scrape time with metric_relabel_configs. Use tiered retention: detailed data for days, downsampled or aggregated data for months. Precompute expensive expressions with recording rules. Review cost regularly by service or team, so the people who emit data see what it costs.
Four cost levers
Telemetry cost grows with volume, cardinality, retention and query load. Tune each lever to arrive at an affordable budget without losing the signals you need.
Finding cardinality hot spots
PromQL queries that reveal which metrics and labels create the most series. The first one touches every series, so run it sparingly. Prometheus's own /tsdb-status page shows similar statistics.
# Which metric names have the most series? (expensive: run rarely, on a small range)
topk(10, count by (__name__) ({__name__=~".+"}))
# Which label of one metric has the most distinct values?
count(count by (route) (http_requests_total))
count(count by (instance) (http_requests_total))
# Samples ingested per second by this Prometheus (from its own metrics)
rate(prometheus_tsdb_head_samples_appended_total[5m])Measure the overhead, do not guess
Benchmark a request path with and without instrumentation, at your production sampling rate. If overhead is noticeable, the usual culprits are synchronous exporters, debug logs left on, string formatting of large objects, and spans or metrics inside very tight loops.
Quick check: A team's metrics bill doubled after a release that added a `session_id` label to a request counter. What is the most effective fix?
- Increase the scrape interval
- Shorten log retention
- Switch dashboards to dark mode
- Remove the session_id label and keep session details in logs or traces
Answer
Remove the session_id label and keep session details in logs or traces — The cost came from cardinality: every session created new series. Removing the unbounded label fixes the cause; scrape interval changes only reduce samples, not series.