Lesson 5 / 35
Structured Logging
Emit logs as key-value JSON instead of free text so they can be filtered, aggregated and joined reliably.
By Rahul Agarwal · 15 years in web development
Why free-text logs stop scaling
A free-text line like User 42 paid 499 via upi in 812ms is easy for a human to read but hard for a machine. To find all payments slower than 500 ms, you would need a regular expression that breaks the moment someone rewords the message.
Structured logging writes each event as a set of named fields, typically one JSON object per line:
{"msg":"payment ok","user_id":42,"amount_inr":499,"method":"upi","duration_ms":812}.
Now the log backend can filter duration_ms > 500, group by method, and join on user_id without parsing prose.
Good practice: keep the msg short and constant (it acts like an event name), and put variable data into fields. Use consistent field names across services (user_id, not userId in one and uid in another), include a timestamp in ISO 8601 UTC, the service name and the environment. Libraries such as structlog in Python, slog in Go, and Pino in Node.js do this out of the box.
A minimal JSON formatter
A custom logging.Formatter turns each record into one JSON line. The timestamp is left out here so the output is reproducible; a real formatter would add "ts" in ISO 8601 UTC.
import json, logging, sys
class JsonFormatter(logging.Formatter):
def format(self, record):
entry = {
"level": record.levelname,
"logger": record.name,
"msg": record.getMessage(),
}
entry.update(getattr(record, "fields", {}))
return json.dumps(entry)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
log = logging.getLogger("orders")
log.addHandler(handler)
log.setLevel(logging.INFO)
log.info("order placed", extra={"fields": {"order_id": "A17", "amount_inr": 499}})
log.warning("slow query", extra={"fields": {"query_ms": 812, "table": "orders"}})
Output:
{"level": "INFO", "logger": "orders", "msg": "order placed", "order_id": "A17", "amount_inr": 499}
{"level": "WARNING", "logger": "orders", "msg": "slow query", "query_ms": 812, "table": "orders"}From event to query
An event happens in code, its details become named fields, the formatter writes one JSON line, the backend parses it without regexes, and engineers query by field.
Railway reservation chart
A reservation chart on a train door lists coach, berth, name, age and PNR in fixed columns. A TTE can scan for "all passengers in B2" in seconds. If the same information were written as handwritten sentences, every lookup would mean reading every line. Structured logs are the printed chart.
One event, one line
Multi-line logs (pretty-printed JSON, raw stack traces) get split into separate entries by most shippers. Keep each event on one line and put the stack trace inside a field such as "exc", escaped as a single JSON string.
Quick check: What is the main advantage of a constant message plus fields, such as msg="payment ok" with duration_ms=812, over a sentence with values embedded?
- It makes each log line shorter on disk in all cases
- Backends can filter and aggregate on fields without fragile text parsing
- It removes the need for log levels
- It encrypts the values automatically
Answer
Backends can filter and aggregate on fields without fragile text parsing — Named fields let the backend query `duration_ms > 500` or group by a field directly. JSON is not always shorter and it does not encrypt anything.