Lesson 18 / 35
Head and Tail Sampling
Compare head-based and tail-based trace sampling and pick a strategy that keeps the interesting traces.
By Rahul Agarwal · 15 years in web development
You cannot keep every trace
A busy service can produce millions of spans per minute. Storing all of them is usually too expensive, so traces are sampled.
Head-based sampling decides at the start of the trace, in the first service. A common rule is "keep 10% of traces", made deterministic by hashing the trace ID, and the decision travels downstream in the sampled flag so every service agrees. It is cheap and simple, but blind: the decision is made before anyone knows whether the request will fail or be slow, so 90% of the errors are thrown away too.
Tail-based sampling waits until the trace is complete, then decides: keep every trace with an error, every trace slower than 2 s, and a small percentage of the rest. It keeps the interesting traces, but all spans of a trace must reach the same collector instance, which must buffer them in memory for a decision window. That costs memory and needs trace-ID-aware load balancing.
Many teams combine both: a generous head sample to cap volume, then tail rules in the collector.
Deterministic ratio sampling
Every service computes the same answer from the trace ID alone, so a trace is kept or dropped as a whole. This is the same idea behind OpenTelemetry's trace-ID-ratio sampler.
import random
def keep(trace_id_hex, ratio):
"""Keep a trace if the low 64 bits of its ID fall below ratio * 2**64."""
bound = int(ratio * (1 << 64))
return int(trace_id_hex[-16:], 16) < bound
rng = random.Random(7)
ids = [f"{rng.getrandbits(128):032x}" for _ in range(10_000)]
kept = sum(keep(t, 0.10) for t in ids)
print("kept at 10%:", kept, "of", len(ids))
t = ids[0]
print("service A decides:", keep(t, 0.10))
print("service B decides:", keep(t, 0.10)) # same ID, same answer
Output:
kept at 10%: 1017 of 10000 service A decides: False service B decides: False
Record the sample rate
If you keep 1 in 10 traces, counts derived from traces must be multiplied by 10. Store the sampling probability with the data, and compute request rates and error rates from metrics, which are not sampled, rather than from traces.
Quick check: What is the main advantage of tail-based sampling over head-based sampling?
- It needs no memory in the collector
- It can keep traces based on their outcome, such as errors or high latency
- It works without propagating context
- It reduces the number of spans created by the application
Answer
It can keep traces based on their outcome, such as errors or high latency — Tail sampling decides after the trace completes, so it can keep all errors and slow traces. The cost is buffering spans in the collector until the decision is made.