# Shipping Logs off the Machine — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-log-shipping

> Explain why apps log to stdout, how agents tail and enrich logs, and what happens when the backend is slow.

## Write to stdout, let the platform ship

A container can be killed and rescheduled at any moment, taking its local files with it. So the common advice (from the Twelve-Factor App) is: the application writes logs as a stream to **stdout/stderr** and does not manage files or rotation. The container runtime writes that stream to files on the node, and a **log agent** running on every node (often a Kubernetes DaemonSet) tails those files.

Popular agents include Fluent Bit, Fluentd, Vector, the OpenTelemetry Collector and Grafana Alloy. The agent does three jobs: **parse** each line (JSON or a regex), **enrich** it with metadata such as pod name, namespace and node, and **forward** it to one or more backends with batching and retries.

Agents keep a position file or checkpoint so a restart resumes where it left off. They also buffer in memory or on disk; when the backend is down for longer than the buffer can hold, the oldest data is dropped. Size buffers for the outage you are willing to survive.

## Path of a log line

A log line leaves the **application** on stdout, the **container runtime** writes it to a file on the node, the **agent** tails, parses and enriches it, the **log store** indexes it, and finally an engineer **searches** it.

![Five stages in order: application writes stdout, runtime saves a node log file, an agent tails and enriches it, a log store indexes it, an engineer searches it.](assets/figures/observability/section-3-map.svg) — Figure 3.1: application stdout -> node log file -> agent -> log store -> search

## A Fluent Bit pipeline

This classic-format config tails container logs (the `docker, cri` parsers unwrap the runtime's log format), adds Kubernetes metadata, drops health-check noise and sends the rest to Loki. Each `[FILTER]` runs in order on records whose tag matches.

```text
[SERVICE]
    Flush        5
    Log_Level    info

[INPUT]
    Name         tail
    Path         /var/log/containers/*.log
    multiline.parser docker, cri
    Tag          kube.*

[FILTER]
    Name         kubernetes
    Match        kube.*

[FILTER]
    Name         grep
    Match        kube.*
    Exclude      log healthz

[OUTPUT]
    Name         loki
    Match        kube.*
    Host         loki.monitoring.svc
    Port         3100
```

## Drop noise at the edge

Health checks, readiness probes and successful static-file requests can be a large share of log volume. Filtering them in the agent is far cheaper than storing them and ignoring them later. Do it with care: never drop error lines just because they are frequent.

**Quiz:** Why do container platforms prefer applications to log to stdout instead of writing their own log files?

- [ ] stdout is encrypted by default
- [x] Files inside a container can disappear when it is replaced, and the platform can collect stdout uniformly
- [ ] Writing files is not possible inside containers
- [ ] stdout automatically converts logs to JSON

*Answer:* Files inside a container can disappear when it is replaced, and the platform can collect stdout uniformly. Container filesystems are ephemeral. Logging to stdout lets the runtime and a node-level agent collect every container's logs the same way.
