Lesson 11 / 35
Labels and Cardinality
Use labels to slice metrics, estimate how many time series they create, and avoid cardinality explosions.
By Rahul Agarwal · 15 years in web development
Every label value is a new time series
In Prometheus-style systems a metric is identified by its name plus its labels (key-value pairs such as method="GET", status="500"). Each unique combination is a separate time series with its own storage and index entries. Cardinality is the number of such series.
Labels are what make metrics powerful: one counter http_requests_total can answer "errors per route" or "traffic per region". But the series count is (roughly) the product of each label's distinct values. Five methods times 40 routes times 8 status codes times 30 instances is already 48,000 series for one counter, and a histogram multiplies that by its bucket count.
The real danger is unbounded labels: user IDs, email addresses, full URLs with IDs inside (/orders/98123), request IDs, timestamps. These create a new series for almost every request, which can exhaust memory in the metrics server and make queries crawl. Keep label values to small, known sets. Put per-user detail in logs or traces, where high cardinality is expected.
Estimating the series count
This is a worst-case upper bound; in practice not every combination occurs. It still shows how quickly multiplication grows.
from math import prod
labels = {"method": 5, "route": 40, "status": 8, "instance": 30}
base = prod(labels.values())
print(f"counter series: {base:,}")
buckets = 12 + 1 # 12 buckets plus le="+Inf"
print(f"histogram series: {base * (buckets + 2):,}") # + _sum and _count
users = 50_000
print(f"counter with user_id label: {base * users:,}")
Output:
counter series: 48,000 histogram series: 720,000 counter with user_id label: 2,400,000,000
Normalise routes before labelling
Label the route template, not the raw path: route="/orders/{id}" instead of /orders/98123. Most web frameworks expose the matched route pattern for exactly this reason. Also cap free-form values, for example map unknown user agents to "other".
Quick check: Which label is most likely to cause a cardinality explosion on an HTTP request counter?
- status="200"
- method="POST"
- region="ap-south-1"
- customer_email="a@b.com"
Answer
customer_email="a@b.com" — Email addresses are effectively unbounded: every new customer creates new series. Status, method and region each have a small, fixed set of values.