# The Telemetry Pipeline — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-telemetry-pipeline

> Describe how telemetry travels from application code to storage, and where push and pull models fit.

## From a line of code to a dashboard

Telemetry flows through a **pipeline**. First, **instrumentation**: code or a library records an event, increments a counter or starts a span. Second, **collection**: an agent on the host or a sidecar gathers the data. Logs are often written to stdout and read by an agent such as Fluent Bit or Vector; metrics are commonly **pulled** by Prometheus scraping an HTTP `/metrics` endpoint; traces are usually **pushed** by an SDK to a collector using the OTLP protocol.

Third, **processing**: the collector can batch, filter, drop noisy data, redact secrets and add labels like `env=prod`. Fourth, **storage**: specialised backends such as a time-series database for metrics, a log store, and a trace store. Finally, **query and alerting**: dashboards and rules read from storage.

Each stage can fail or lose data. A good design buffers between stages, so a slow backend does not block your application, and it watches the pipeline itself.

## Push vs pull in one table

How each signal usually travels in a common open-source setup. Real deployments can mix these (Prometheus also accepts remote-write pushes, and collectors can scrape).

```text
signal   produced by           transport            typical backend
-------  --------------------  -------------------  -------------------------
logs     app writes stdout     agent tails files    Loki / Elasticsearch
metrics  app exposes /metrics  Prometheus pulls     Prometheus TSDB
traces   SDK creates spans     SDK pushes OTLP      Tempo / Jaeger
```

## Never let telemetry take the app down

Instrumentation must be **non-blocking**. If an SDK sends spans synchronously and the collector is down, every request waits. Use batching exporters with bounded queues that **drop** data when full. Losing some telemetry during an outage is acceptable; losing the service because of telemetry is not.

**Quiz:** Why do telemetry SDKs use bounded in-memory queues that drop data when full?

- [x] To keep the application responsive even if the telemetry backend is slow or down
- [ ] Because backends reject batches larger than one item
- [ ] To guarantee every span is stored exactly once
- [ ] Because dropping data improves query accuracy

*Answer:* To keep the application responsive even if the telemetry backend is slow or down. A bounded queue protects the application: when the backend cannot keep up, telemetry is dropped instead of blocking requests or exhausting memory.
