Linking Metrics to Traces

From a spike to an example trace

Exemplars attach a sample trace ID to metric data points, for example to a histogram bucket, so a dashboard can link a latency spike directly to a trace that experienced it. Backends such as Prometheus (with exemplar storage enabled) and Grafana support them. Another approach derives metrics from spans: the Collector's spanmetrics connector produces request rate, error and duration metrics from traces, giving RED metrics even where no metrics are instrumented.

Generating RED metrics from spans in the Collector

A connector links a traces pipeline to a metrics pipeline (contrib distribution).

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [50ms, 100ms, 250ms, 500ms, 1s, 2s]

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo, spanmetrics]     # the connector acts as an exporter here
    metrics:
      receivers: [spanmetrics]                 # ...and as a receiver here
      processors: [batch]
      exporters: [prometheusremotewrite]

Mind sampling when deriving metrics

Metrics computed from sampled traces undercount unless they are generated before sampling.

Quick check: What is an exemplar?

• A metric type
• A sample trace ID attached to a metric data point
• A log level
• A sampling policy