पाठ 18 / 35

Head and Tail Sampling

यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।

Compare head-based and tail-based trace sampling and pick a strategy that keeps the interesting traces.

लेखक: · 15 साल से वेब डेवलपर

You cannot keep every trace

A busy service can produce millions of spans per minute. Storing all of them is usually too expensive, so traces are sampled.

Head-based sampling decides at the start of the trace, in the first service. A common rule is "keep 10% of traces", made deterministic by hashing the trace ID, and the decision travels downstream in the sampled flag so every service agrees. It is cheap and simple, but blind: the decision is made before anyone knows whether the request will fail or be slow, so 90% of the errors are thrown away too.

Tail-based sampling waits until the trace is complete, then decides: keep every trace with an error, every trace slower than 2 s, and a small percentage of the rest. It keeps the interesting traces, but all spans of a trace must reach the same collector instance, which must buffer them in memory for a decision window. That costs memory and needs trace-ID-aware load balancing.

Many teams combine both: a generous head sample to cap volume, then tail rules in the collector.

Deterministic ratio sampling

Every service computes the same answer from the trace ID alone, so a trace is kept or dropped as a whole. This is the same idea behind OpenTelemetry's trace-ID-ratio sampler.

import random

def keep(trace_id_hex, ratio):
    """Keep a trace if the low 64 bits of its ID fall below ratio * 2**64."""
    bound = int(ratio * (1 << 64))
    return int(trace_id_hex[-16:], 16) < bound

rng = random.Random(7)
ids = [f"{rng.getrandbits(128):032x}" for _ in range(10_000)]

kept = sum(keep(t, 0.10) for t in ids)
print("kept at 10%:", kept, "of", len(ids))

t = ids[0]
print("service A decides:", keep(t, 0.10))
print("service B decides:", keep(t, 0.10))   # same ID, same answer

Output:

kept at 10%: 1017 of 10000
service A decides: False
service B decides: False

Record the sample rate

If you keep 1 in 10 traces, counts derived from traces must be multiplied by 10. Store the sampling probability with the data, and compute request rates and error rates from metrics, which are not sampled, rather than from traces.

त्वरित जाँच: What is the main advantage of tail-based sampling over head-based sampling?

  • It needs no memory in the collector
  • It can keep traces based on their outcome, such as errors or high latency
  • It works without propagating context
  • It reduces the number of spans created by the application
Answer

It can keep traces based on their outcome, such as errors or high latency — Tail sampling decides after the trace completes, so it can keep all errors and slow traces. The cost is buffering spans in the collector until the decision is made.