Lesson 3 / 35

The Telemetry Pipeline

Describe how telemetry travels from application code to storage, and where push and pull models fit.

By · 15 years in web development

From a line of code to a dashboard

Telemetry flows through a pipeline. First, instrumentation: code or a library records an event, increments a counter or starts a span. Second, collection: an agent on the host or a sidecar gathers the data. Logs are often written to stdout and read by an agent such as Fluent Bit or Vector; metrics are commonly pulled by Prometheus scraping an HTTP /metrics endpoint; traces are usually pushed by an SDK to a collector using the OTLP protocol.

Third, processing: the collector can batch, filter, drop noisy data, redact secrets and add labels like env=prod. Fourth, storage: specialised backends such as a time-series database for metrics, a log store, and a trace store. Finally, query and alerting: dashboards and rules read from storage.

Each stage can fail or lose data. A good design buffers between stages, so a slow backend does not block your application, and it watches the pipeline itself.

Push vs pull in one table

How each signal usually travels in a common open-source setup. Real deployments can mix these (Prometheus also accepts remote-write pushes, and collectors can scrape).

signal   produced by           transport            typical backend
-------  --------------------  -------------------  -------------------------
logs     app writes stdout     agent tails files    Loki / Elasticsearch
metrics  app exposes /metrics  Prometheus pulls     Prometheus TSDB
traces   SDK creates spans     SDK pushes OTLP      Tempo / Jaeger

Never let telemetry take the app down

Instrumentation must be non-blocking. If an SDK sends spans synchronously and the collector is down, every request waits. Use batching exporters with bounded queues that drop data when full. Losing some telemetry during an outage is acceptable; losing the service because of telemetry is not.

Quick check: Why do telemetry SDKs use bounded in-memory queues that drop data when full?

  • To keep the application responsive even if the telemetry backend is slow or down
  • Because backends reject batches larger than one item
  • To guarantee every span is stored exactly once
  • Because dropping data improves query accuracy
Answer

To keep the application responsive even if the telemetry backend is slow or down — A bounded queue protects the application: when the backend cannot keep up, telemetry is dropped instead of blocking requests or exhausting memory.