Tenant-Aware Observability

Every signal needs a tenant

In a multi-tenant system, "the API is slow" is not actionable; "the API is slow for three tenants on shard 2" is. So tenant context must reach your telemetry. Logs should carry tenant_id as a structured field on every line, added automatically from request context. Traces should carry it as a span attribute so you can filter one tenant's slow requests. Metrics are trickier: tagging every metric with tenant id creates one time series per tenant per metric, which is fine for a few hundred tenants but becomes expensive or impossible with tens of thousands. Common compromises are tenant-level labels only on a few key metrics, labels by plan or shard instead of tenant, and computing per-tenant numbers from logs or traces in a separate analytics store. With this in place you can build per-tenant SLO dashboards, spot noisy neighbours and tell a customer exactly how an incident affected them.

Tenant context flows into telemetry

The request's tenant context is stamped on logs, traces and selected metrics, which then feed per-tenant dashboards and alerts.

Stamping tenant_id on every log line

A logging filter reads the same context variable that the request middleware sets, so no call site has to remember the tenant.

import logging
import sys
from contextvars import ContextVar

current_tenant: ContextVar[str] = ContextVar("current_tenant", default="-")

class TenantFilter(logging.Filter):
    def filter(self, record):
        record.tenant_id = current_tenant.get()   # stamp every record
        return True

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("%(levelname)s tenant=%(tenant_id)s %(message)s"))
handler.addFilter(TenantFilter())
log = logging.getLogger("app")
log.addHandler(handler)
log.setLevel(logging.INFO)

log.info("worker started")
current_tenant.set("acme")
log.info("export finished in 840 ms")
current_tenant.set("zen")
log.warning("payment webhook retried")

Output:

INFO tenant=- worker started
INFO tenant=acme export finished in 840 ms
WARNING tenant=zen payment webhook retried

Mind metric cardinality

Each unique combination of label values is a separate time series. tenant_id x endpoint x status across 20,000 tenants can mean millions of series. Keep high-cardinality ids in logs and traces, and aggregate into metrics deliberately.

Quick check: Why is adding `tenant_id` as a label on every metric risky at large scale?

• Metric systems cannot store strings
• It creates a time series per tenant per label combination, which can explode storage and query cost
• It leaks data to other tenants
• Metrics are not allowed to have labels