Lesson 21 / 35

The OpenTelemetry Collector

Configure Collector pipelines of receivers, processors and exporters, and choose between agent and gateway deployments.

By · 15 years in web development

Receivers, processors, exporters

The OpenTelemetry Collector is a standalone binary that sits between applications and backends. Its configuration has three kinds of components. Receivers accept data: OTLP over gRPC (port 4317 by convention) or HTTP (4318), Prometheus scraping, log files, and many more. Processors transform data in order: memory_limiter protects the Collector from running out of memory, batch groups data for efficient export, attribute processors add or delete fields, and the tail_sampling processor makes outcome-based sampling decisions. Exporters send data onwards: OTLP to a trace backend, Prometheus remote write, a logging backend.

Components only run when they appear in a pipeline under service.pipelines, and each pipeline handles one signal type. Defining a processor without referencing it does nothing.

Two common deployments: an agent Collector on each node or as a sidecar, close to the app, and a gateway Collector pool that centralises processing, credentials and sampling. Many components live in the contrib distribution, so check that your build includes the ones you configure.

A Collector config

Traces pass through the memory limiter, have user.email removed, are batched and sent to Tempo. Metrics are exposed for Prometheus to scrape on port 8889 (that exporter ships in the contrib distribution).

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  attributes/scrub:
    actions:
      - key: user.email
        action: delete
  batch: {}

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/scrub, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

Order processors deliberately

Put memory_limiter first so overload is detected before work is done, and batch near the end so it batches the final data. Scrubbing sensitive attributes in the Collector is a useful safety net, but it is not a substitute for not recording them in the app.

Quick check: A processor is defined under `processors:` but not listed in any pipeline. What happens?

  • It runs on every signal by default
  • The Collector refuses to start
  • It is not used, because only components referenced in a pipeline are active
  • It runs only on traces
Answer

It is not used, because only components referenced in a pipeline are active — Defining a component only configures it. It is enabled only when a pipeline under `service.pipelines` references it.