पाठ 3 / 35
The Telemetry Pipeline
यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।
Describe how telemetry travels from application code to storage, and where push and pull models fit.
लेखक: Rahul Agarwal · 15 साल से वेब डेवलपर
From a line of code to a dashboard
Telemetry flows through a pipeline. First, instrumentation: code or a library records an event, increments a counter or starts a span. Second, collection: an agent on the host or a sidecar gathers the data. Logs are often written to stdout and read by an agent such as Fluent Bit or Vector; metrics are commonly pulled by Prometheus scraping an HTTP /metrics endpoint; traces are usually pushed by an SDK to a collector using the OTLP protocol.
Third, processing: the collector can batch, filter, drop noisy data, redact secrets and add labels like env=prod. Fourth, storage: specialised backends such as a time-series database for metrics, a log store, and a trace store. Finally, query and alerting: dashboards and rules read from storage.
Each stage can fail or lose data. A good design buffers between stages, so a slow backend does not block your application, and it watches the pipeline itself.
Push vs pull in one table
How each signal usually travels in a common open-source setup. Real deployments can mix these (Prometheus also accepts remote-write pushes, and collectors can scrape).
signal produced by transport typical backend
------- -------------------- ------------------- -------------------------
logs app writes stdout agent tails files Loki / Elasticsearch
metrics app exposes /metrics Prometheus pulls Prometheus TSDB
traces SDK creates spans SDK pushes OTLP Tempo / JaegerNever let telemetry take the app down
Instrumentation must be non-blocking. If an SDK sends spans synchronously and the collector is down, every request waits. Use batching exporters with bounded queues that drop data when full. Losing some telemetry during an outage is acceptable; losing the service because of telemetry is not.
त्वरित जाँच: Why do telemetry SDKs use bounded in-memory queues that drop data when full?
- To keep the application responsive even if the telemetry backend is slow or down
- Because backends reject batches larger than one item
- To guarantee every span is stored exactly once
- Because dropping data improves query accuracy
Answer
To keep the application responsive even if the telemetry backend is slow or down — A bounded queue protects the application: when the backend cannot keep up, telemetry is dropped instead of blocking requests or exhausting memory.