Lesson 7 / 35
Shipping Logs off the Machine
Explain why apps log to stdout, how agents tail and enrich logs, and what happens when the backend is slow.
By Rahul Agarwal · 15 years in web development
Write to stdout, let the platform ship
A container can be killed and rescheduled at any moment, taking its local files with it. So the common advice (from the Twelve-Factor App) is: the application writes logs as a stream to stdout/stderr and does not manage files or rotation. The container runtime writes that stream to files on the node, and a log agent running on every node (often a Kubernetes DaemonSet) tails those files.
Popular agents include Fluent Bit, Fluentd, Vector, the OpenTelemetry Collector and Grafana Alloy. The agent does three jobs: parse each line (JSON or a regex), enrich it with metadata such as pod name, namespace and node, and forward it to one or more backends with batching and retries.
Agents keep a position file or checkpoint so a restart resumes where it left off. They also buffer in memory or on disk; when the backend is down for longer than the buffer can hold, the oldest data is dropped. Size buffers for the outage you are willing to survive.
Path of a log line
A log line leaves the application on stdout, the container runtime writes it to a file on the node, the agent tails, parses and enriches it, the log store indexes it, and finally an engineer searches it.
A Fluent Bit pipeline
This classic-format config tails container logs (the docker, cri parsers unwrap the runtime's log format), adds Kubernetes metadata, drops health-check noise and sends the rest to Loki. Each [FILTER] runs in order on records whose tag matches.
[SERVICE]
Flush 5
Log_Level info
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
[FILTER]
Name kubernetes
Match kube.*
[FILTER]
Name grep
Match kube.*
Exclude log healthz
[OUTPUT]
Name loki
Match kube.*
Host loki.monitoring.svc
Port 3100Drop noise at the edge
Health checks, readiness probes and successful static-file requests can be a large share of log volume. Filtering them in the agent is far cheaper than storing them and ignoring them later. Do it with care: never drop error lines just because they are frequent.
Quick check: Why do container platforms prefer applications to log to stdout instead of writing their own log files?
- stdout is encrypted by default
- Files inside a container can disappear when it is replaced, and the platform can collect stdout uniformly
- Writing files is not possible inside containers
- stdout automatically converts logs to JSON
Answer
Files inside a container can disappear when it is replaced, and the platform can collect stdout uniformly — Container filesystems are ephemeral. Logging to stdout lets the runtime and a node-level agent collect every container's logs the same way.