Lesson 7 / 35

Shipping Logs off the Machine

Explain why apps log to stdout, how agents tail and enrich logs, and what happens when the backend is slow.

By · 15 years in web development

Write to stdout, let the platform ship

A container can be killed and rescheduled at any moment, taking its local files with it. So the common advice (from the Twelve-Factor App) is: the application writes logs as a stream to stdout/stderr and does not manage files or rotation. The container runtime writes that stream to files on the node, and a log agent running on every node (often a Kubernetes DaemonSet) tails those files.

Popular agents include Fluent Bit, Fluentd, Vector, the OpenTelemetry Collector and Grafana Alloy. The agent does three jobs: parse each line (JSON or a regex), enrich it with metadata such as pod name, namespace and node, and forward it to one or more backends with batching and retries.

Agents keep a position file or checkpoint so a restart resumes where it left off. They also buffer in memory or on disk; when the backend is down for longer than the buffer can hold, the oldest data is dropped. Size buffers for the outage you are willing to survive.

Path of a log line

A log line leaves the application on stdout, the container runtime writes it to a file on the node, the agent tails, parses and enriches it, the log store indexes it, and finally an engineer searches it.

Five stages in order: application writes stdout, runtime saves a node log file, an agent tails and enriches it, a log store indexes it, an engineer searches it.
Figure 3.1: application stdout -> node log file -> agent -> log store -> search

A Fluent Bit pipeline

This classic-format config tails container logs (the docker, cri parsers unwrap the runtime's log format), adds Kubernetes metadata, drops health-check noise and sends the rest to Loki. Each [FILTER] runs in order on records whose tag matches.

[SERVICE]
    Flush        5
    Log_Level    info

[INPUT]
    Name         tail
    Path         /var/log/containers/*.log
    multiline.parser docker, cri
    Tag          kube.*

[FILTER]
    Name         kubernetes
    Match        kube.*

[FILTER]
    Name         grep
    Match        kube.*
    Exclude      log healthz

[OUTPUT]
    Name         loki
    Match        kube.*
    Host         loki.monitoring.svc
    Port         3100

Drop noise at the edge

Health checks, readiness probes and successful static-file requests can be a large share of log volume. Filtering them in the agent is far cheaper than storing them and ignoring them later. Do it with care: never drop error lines just because they are frequent.

Quick check: Why do container platforms prefer applications to log to stdout instead of writing their own log files?

  • stdout is encrypted by default
  • Files inside a container can disappear when it is replaced, and the platform can collect stdout uniformly
  • Writing files is not possible inside containers
  • stdout automatically converts logs to JSON
Answer

Files inside a container can disappear when it is replaced, and the platform can collect stdout uniformly — Container filesystems are ephemeral. Logging to stdout lets the runtime and a node-level agent collect every container's logs the same way.