Head and Tail Sampling

A busy service produces more traces than anyone can store or read, so tracing systems keep only a fraction. Head and tail sampling differ in when that decision is made. Head sampling decides at the start of a trace, before the request has run. Tail sampling decides after the spans of the trace have been collected for a bounded wait, when the outcome is usually known. This is a different idea from statistical sampling of a population for estimation: here the sampler is deciding which complete records to retain, and it can use what it knows about each record to do so.

Head sampling: decide once, from the trace ID

The first service to handle a request creates the trace ID and makes the decision. The most common form keeps a fixed percentage and derives the decision from the trace ID itself, for example by comparing part of the ID, read as a number, with a threshold. Because the decision depends only on the ID, any component can recompute it and get the same answer. The decision is also carried with the request. In the W3C Trace Context header, traceparent, the least significant bit of the trace-flags field is the sampled flag; a value of 01 means the caller may have recorded the trace, and 00 means it did not. Downstream services normally follow that flag, so a trace is kept in full or dropped in full instead of arriving with holes.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |  |                                |                |
             |  trace ID (32 hex)                parent span ID   trace flags: 01 = sampled
             version

Head sampling is cheap and needs no memory: nothing is buffered, and unsampled requests can skip most of the cost of recording spans. Its limit is that the decision is made before the outcome exists. A 5 percent head sample keeps about 5 percent of the failures too, so a rare failure may leave no trace at all.

Tail sampling: decide after collecting the trace

A tail sampler, typically running in a collector, receives spans from many services, groups them by trace ID, waits a bounded time for the trace to look complete, and then applies rules: keep every trace that contains an error, keep traces slower than a latency threshold, keep a small percentage of the rest. That solves the rare-failure problem, and it has real costs. All spans of a trace must reach the same sampler instance, so a fleet of collectors usually routes by trace ID. The sampler must hold spans in memory while it waits, and a span that arrives after the decision is lost or kept inconsistently. OpenTelemetry’s documentation describes the costs as higher implementation complexity, stateful systems with significant storage needs, and, in some cases, vendor-specific tooling. Systems with very high volume often combine the two: a coarse head sample protects the pipeline, and a tail sampler selects within what remains.

What the kept traces can and cannot tell you

A fixed-probability head rule based on uniformly distributed trace IDs gives each request the same inclusion probability when those IDs are independent of the outcome, so proportions estimate the traffic rates. Head sampling can also use attributes; deciding early alone does not make a sample uniform. A tail sample deliberately over-represents errors, so computed directly it gives a distorted rate. The hypothetical population below contains 20,000 traces with about a 2 percent error rate and keeps traces by a head rule (5 percent by trace ID) and by a tail rule (every error, plus the same 5 percent of the rest). Each tail-kept trace records the probability with which it was kept, 1.0 for errors and 0.05 for others, and the weighted estimate divides each trace by that probability.

# Weighting sketch: traces is a supplied collection of (trace_id, is_error) pairs.
def head_keep(tid, p):            # decided from the trace ID alone
    return (tid & 0xFFFFFFFF) < p * 2**32

tail = []
for tid, err in traces:
    if err: tail.append((tid, err, 1.0))
    elif head_keep(tid, 0.05): tail.append((tid, err, 0.05))

naive = sum(e for _, e, _ in tail) / len(tail)
weighted = sum(e / w for _, e, w in tail) / sum(1 / w for _, _, w in tail)

true error traces: 406 of 20000
head 5%: kept 939 errors kept 17
tail rule: kept 1328 errors kept 406
error rate, true 0.0203  head 0.0181  tail unweighted 0.3057  tail weighted 0.0215

The head sample kept 17 of the 406 error traces. The tail rule kept all of them, which is what makes it useful for debugging. Its unweighted error rate, 30.6 percent, describes the kept set and not the traffic. Dividing each trace by its keep probability recovers 2.15 percent against the true 2.03 percent. The head sample’s 1.81 percent is also off by chance, because it rests on only 17 error traces; sampling noise on a rare event is large either way. The counts are a hypothetical illustration of the weighting calculation; the code shows the rule for a supplied trace collection.

The practical rules follow. Use tail sampling when the traces you most need are rare and identifiable by outcome. Keep the sampling probability with each retained trace if anyone will compute rates from them, or compute rates from metrics that count every request and use traces only for inspection. Remember that deciding to retain a trace is also deciding to retain the content in its spans, which matters when spans carry user data.

References

• OpenTelemetry, Sampling
• W3C, Trace Context