# Structured Logging — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-structured-logging

> Emit logs as key-value JSON instead of free text so they can be filtered, aggregated and joined reliably.

## Why free-text logs stop scaling

A free-text line like `User 42 paid 499 via upi in 812ms` is easy for a human to read but hard for a machine. To find all payments slower than 500 ms, you would need a regular expression that breaks the moment someone rewords the message.

**Structured logging** writes each event as a set of named fields, typically one JSON object per line:
`{"msg":"payment ok","user_id":42,"amount_inr":499,"method":"upi","duration_ms":812}`.
Now the log backend can filter `duration_ms > 500`, group by `method`, and join on `user_id` without parsing prose.

Good practice: keep the `msg` short and constant (it acts like an event name), and put variable data into fields. Use consistent field names across services (`user_id`, not `userId` in one and `uid` in another), include a timestamp in ISO 8601 UTC, the service name and the environment. Libraries such as `structlog` in Python, `slog` in Go, and Pino in Node.js do this out of the box.

## A minimal JSON formatter

A custom `logging.Formatter` turns each record into one JSON line. The timestamp is left out here so the output is reproducible; a real formatter would add `"ts"` in ISO 8601 UTC.

```python
import json, logging, sys

class JsonFormatter(logging.Formatter):
    def format(self, record):
        entry = {
            "level": record.levelname,
            "logger": record.name,
            "msg": record.getMessage(),
        }
        entry.update(getattr(record, "fields", {}))
        return json.dumps(entry)

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
log = logging.getLogger("orders")
log.addHandler(handler)
log.setLevel(logging.INFO)

log.info("order placed", extra={"fields": {"order_id": "A17", "amount_inr": 499}})
log.warning("slow query", extra={"fields": {"query_ms": 812, "table": "orders"}})
```

Output:

```
{"level": "INFO", "logger": "orders", "msg": "order placed", "order_id": "A17", "amount_inr": 499}
{"level": "WARNING", "logger": "orders", "msg": "slow query", "query_ms": 812, "table": "orders"}
```

## From event to query

An **event** happens in code, its details become named **fields**, the formatter writes one **JSON line**, the backend **parses** it without regexes, and engineers **query** by field.

![Five stages in order: an event in code, named fields, one JSON log line, parsing in the backend, querying by field.](assets/figures/observability/section-2-map.svg) — Figure 2.1: event -> fields -> JSON line -> parse -> query

## Railway reservation chart

A reservation chart on a train door lists coach, berth, name, age and PNR in fixed columns. A TTE can scan for "all passengers in B2" in seconds. If the same information were written as handwritten sentences, every lookup would mean reading every line. Structured logs are the printed chart.

## One event, one line

Multi-line logs (pretty-printed JSON, raw stack traces) get split into separate entries by most shippers. Keep each event on one line and put the stack trace inside a field such as `"exc"`, escaped as a single JSON string.

**Quiz:** What is the main advantage of a constant message plus fields, such as msg="payment ok" with duration_ms=812, over a sentence with values embedded?

- [ ] It makes each log line shorter on disk in all cases
- [x] Backends can filter and aggregate on fields without fragile text parsing
- [ ] It removes the need for log levels
- [ ] It encrypts the values automatically

*Answer:* Backends can filter and aggregate on fields without fragile text parsing. Named fields let the backend query `duration_ms > 500` or group by a field directly. JSON is not always shorter and it does not encrypt anything.
