Metrics, Logs, and Traces for Services

A checkout page becomes slow at 14:05. A latency chart shows the slowdown but not where it comes from. Error logs from six services show timeouts, with no way to tell which of them belong to the same user request. A trace for one slow request exists, but it ends at the second service because a proxy dropped a header. Each piece of telemetry was collected correctly, and together they still cannot answer the question, because the decisions that make signals answer questions together (shared context, bounded labels, sampling that keeps whole traces, and attributes that do not leak data) were never made.

This article shows how to make those decisions when instrumenting a service. It covers what each of the three signals is good for, how instrumentation and context are attached, how metric cardinality grows and how to bound it, how sampling interacts with traces, how trace context propagates across service and process boundaries, how the signals combine in a diagnosis, and how to apply privacy controls to telemetry. The examples are small programs executed locally with OpenTelemetry’s Python SDK (version 1.45.0) and plain Python simulations. The simulated traffic is generated, and the results illustrate mechanisms, not benchmarks.

The article assumes the previous one in this series, Choosing Kubernetes, Managed Services, or Simpler Runtime Options, in the sense that you know your runtime choice decides where telemetry originates: a managed service may expose its own metrics and logs that you do not instrument, while code you run yourself needs instrumentation you add. The details of each runtime are covered there, and this article concentrates on the service code and the pipeline.

Three signals and the questions each one answers

OpenTelemetry defines traces as the path of a request through your application, metrics as a measurement captured at runtime, and logs as a recording of an event, with baggage as contextual information passed between signals. Telemetry signals fit together by role. Metrics aggregate many events into numbers over time and answer “how many, how slow, how full”. Traces connect the operations of one request across components and answer “where did this request spend its time”. Logs record individual events and answer “what happened here”.

Signal	Shape of the data	Good at	Limited by	Main cost driver
Metrics	Numeric series identified by a name and a set of labels, sampled over time	Alerting, trends, capacity, percentiles across all traffic	Cannot show a single request; detail is limited to the labels chosen in advance	Number of distinct label combinations (series), and histogram buckets
Traces	A tree of spans per request, each with a name, timing, attributes, and parent	Locating where latency or failure arises across services	Needs context carried across every hop; usually sampled, so some requests have no trace	Number of spans stored and attributes per span; sampling rate
Logs	Timestamped event records, ideally with named fields	Explaining what happened in one place, including details not in a metric or span	Volume; free text is hard to query; contents can include sensitive data	Bytes ingested and retained

The Google SRE book gives a way to choose what to measure first: four golden signals for a user-facing system, namely latency, traffic, errors, and saturation. It also distinguishes symptoms from causes: “my responses are slow” is a symptom that should page a person, while “CPUs are overloaded” is a cause that helps diagnose. That distinction guides the signals’ roles. Metrics detect the symptom, traces narrow the location, and logs explain the cause. The same book distinguishes black-box monitoring, which tests external behavior as users experience it, from white-box monitoring, which inspects internals such as logs; the instrumentation in this article is white-box, so pair it with at least one external check on the user-facing path.

Each signal has weaknesses the others cover. A latency metric can show that p95 doubled without saying which requests were slow. A trace identifies the slow request but only if it was sampled. A log line may explain the failure but is useless if you cannot find the one line among millions. The practical design goal is for the three to share identity: the same service name, the same deployment version, and, for requests, the same trace and span identifiers. A collector can move signals but cannot reconstruct context that instrumentation never emitted.

Instrumentation and context

Where instrumentation comes from

OpenTelemetry describes two broad approaches. Zero-code instrumentation requires no application changes and provides telemetry from the libraries you use and the environment you run in, mainly at application boundaries. Code-based instrumentation uses the OpenTelemetry API and SDKs to generate telemetry from the application itself and gives deeper insight; the two can be used at the same time. Libraries that want to be observable should depend only on the OpenTelemetry API, so that applications that do not install the SDK see no effect. This is why a platform can supply the SDK configuration (exporters, sampling, resource attributes, redaction) once, while application and library authors add instrumentation against a stable API.

For a platform team, one consequence is that the golden path for a new service should include the telemetry setup, so every service starts with consistent names and context, and a service author adds code-based instrumentation only for operations that matter to their domain. Another consequence is that you must decide what the defaults are. Zero-code instrumentation of the HTTP server and client gives you inbound and outbound request spans and request-duration metrics without a line of code, which is the right default for most services.

What to instrument, and how to name it

A service’s interactions with the outside world are the most useful places to put spans and metrics: inbound requests, outbound calls to other services and databases, message-queue publish and consume operations, and any significant internal operation such as a cache lookup or a batch job step. Instrument the boundary because that is where latency and errors are attributable to a counterpart. Resist instrumenting every function; each span has a cost, and a trace with hundreds of tiny spans is harder to read than one with a dozen meaningful ones.

Names matter because they become the dimensions you aggregate by. OpenTelemetry’s HTTP semantic conventions say that span names should follow {method} {target} when a low-cardinality target exists, and that instrumentation must not default to using the URI path as the target; for server spans the target is http.route, for example GET /users/{id}. The route template is low-cardinality; the actual path (url.path, such as /users/4) is not. The same reasoning applies to every attribute you attach: an identifier that has millions of values is a poor span name and a dangerous metric attribute.

Resource attributes identify the producer. The OpenTelemetry resource conventions say the SDK must provide service.name, with a default if none is configured. Set it explicitly for every service, along with attributes for the instance and the deployed version, so you can split a latency regression by release. The requirement for service.name comes from the resource conventions; the exact names of the other resource attributes should be checked against the current conventions.

Context: carrying identity inside and between processes

OpenTelemetry defines context as an object containing information that allows services to correlate signals. Inside a process, the SDK keeps the current context so that new spans become children of the active span and so that code such as a log formatter can read the current trace and span IDs. Across processes, context is serialized into a carrier (such as HTTP headers) by a propagator, and extracted on the other side. The next sections return to propagation; here the point is the in-process half. A span is created with start_as_current_span, and anything executed inside it, including logging calls, can read the active span from context.

The following program instruments two “services” (called checkout and payments) in one process. The checkout service handles POST /orders, calls payments, and payments handles POST /charge. A log formatter adds the active trace and span IDs to each JSON log line. The two services exchange a traceparent header through the propagator, as separate processes would over HTTP. The code is lightly condensed from the script; imports and the metrics section are omitted.

exporter = InMemorySpanExporter()
tp = TracerProvider(); tp.add_span_processor(SimpleSpanProcessor(exporter))
checkout = tp.get_tracer("checkout"); payments = tp.get_tracer("payments")
prop = TraceContextTextMapPropagator()

class JsonFmt(logging.Formatter):          # adds the current span's IDs to every log line
    def format(self, r):
        sc = trace.get_current_span().get_span_context()
        d = {"level": r.levelname, "msg": r.getMessage(), "service": r.name}
        if sc.is_valid:
            d["trace_id"] = format(sc.trace_id, "032x"); d["span_id"] = format(sc.span_id, "016x")
        return json.dumps(d)

def payments_handler(headers):             # "server" side of the call
    ctx = prop.extract(headers)            # read traceparent from the incoming request
    with payments.start_as_current_span("POST /charge", context=ctx, kind=trace.SpanKind.SERVER) as s:
        s.set_attribute("http.request.method","POST"); s.set_attribute("http.route","/charge")
        logging.getLogger("payments").info("charge authorized")

with checkout.start_as_current_span("POST /orders", kind=trace.SpanKind.SERVER):
    with checkout.start_as_current_span("POST payments", kind=trace.SpanKind.CLIENT):
        headers = {}
        prop.inject(headers)               # write traceparent into the outgoing request headers
        print("outgoing headers:", headers)
        logging.getLogger("checkout").info("calling payments")
        payments_handler(headers)

{"level": "INFO", "msg": "calling payments", "service": "checkout", "trace_id": "d571d9fd7154960513aba3ac9c76e2f0", "span_id": "fdd617ac8307f003"}
{"level": "INFO", "msg": "charge authorized", "service": "payments", "trace_id": "d571d9fd7154960513aba3ac9c76e2f0", "span_id": "f617a5bf6f22fe48"}
outgoing headers: {'traceparent': '00-d571d9fd7154960513aba3ac9c76e2f0-fdd617ac8307f003-03'}
POST /charge trace d571d9fd span f617a5bf parent fdd617ac remote_parent True
POST payments trace d571d9fd span fdd617ac parent fd9e527d remote_parent False
POST /orders trace d571d9fd span fd9e527d parent None remote_parent None
same trace id: True

Three things are visible in the output. First, all three spans share one trace ID (d571d9fd...), so a backend can assemble them into a single tree. Second, the payments span’s parent is the checkout client span (fdd617ac), and the SDK marks that parent as remote, which is how it records that the parent lives in another process. Third, the two log lines carry the same trace ID as the spans, so a search for that trace ID in the log store finds the lines that belong to this request; the OpenTelemetry log data model has the fields TraceId, SpanId, and TraceFlags for this purpose, and a collector that receives logs with those fields can store them for correlation. That correlation only works if the application writes the IDs, which is why the formatter exists. Without it the logs and traces would be two unrelated collections.

The traceparent value ends in -03. The W3C Recommendation defines the trace-flags field as 8 bits and describes only the least significant bit, the sampled flag; this SDK version emitted a second bit set as well, which is not described in that text, and its meaning is not covered here. Nothing below depends on it, but it is an example of why you should treat a propagated header as data to be validated, not a stable format you can interpret beyond its documented fields.

Structured logging makes the logs side practical: record events as named fields with defined meanings and stable names, so a collector can filter by field without parsing prose. Define the identity fields (service, version, trace ID, span ID) once in the formatter and leave application code to supply event-specific fields. Note also that JSON encoding does not remove sensitive data, which the privacy section returns to.

Cardinality

A Prometheus time series is identified by its metric name and its complete set of labels, and the documentation states that changing any label value, including adding or removing labels, creates a new time series. Consequently, every distinct combination of label values stored for a metric is a separate series that costs storage, memory for indexing, and query time. The Prometheus naming guidance says not to use labels for dimensions with high cardinality, such as user IDs or email addresses, or other unbounded sets of values. Metric cardinality is the count of those combinations, and it multiplies.

The arithmetic of multiplication

Take an HTTP request counter with labels for status code (5 values), method (4 values), and endpoint (20 values). The maximum number of series is 5 × 4 × 20 = 400 per exporting instance, though only combinations that actually occur are created. Run 50 instances and the ceiling is 20,000. Add a label with 10 values (say, a region or customer tier) and it becomes 4,000 per instance. Nothing in this arithmetic requires an unreasonable label; each looked sensible alone.

Histograms multiply further. A classic Prometheus histogram creates a series for each bucket, plus a sum and a count series, per label combination. A histogram with 10 buckets over the 400 combinations above is on the order of 400 × (10 + 2) = 4,800 series per instance, before counting the implicit overflow bucket. Native histograms in Prometheus compress this into one composite sample per series, according to the Prometheus documentation, so the cost depends on which histogram type and which backend you use. The principle holds regardless: the cost of a metric is the number of label combinations times the size of the value structure per combination.

The expensive case is unbounded labels. A label holding a user ID, a request ID, a full URL path, or an error message creates a new series for every distinct value, and those values keep arriving. Series also churn over time: if instances are replaced often and the instance name is a label, each new name starts another series even though the number of live instances is constant. That follows from the identity rule and is something to check in your own system rather than assume.

A demonstration with the OpenTelemetry SDK

The following code records a duration histogram for 5,000 requests, each to a different user path such as /users/1, /users/2, and so on. In the first run the path is an attribute on the metric. In the second, a View keeps only the method and the route template.

def run(view_kwargs, label_fn, n=5000):
    reader = InMemoryMetricReader()
    mp = MeterProvider(metric_readers=[reader],
                       views=[View(instrument_name="http.server.request.duration", **view_kwargs)])
    hist = mp.get_meter("demo").create_histogram("http.server.request.duration", unit="s")
    for i in range(n):
        hist.record(0.05, label_fn(i))                 # n requests, each to a different user id
    pts = reader.get_metrics_data().resource_metrics[0].scope_metrics[0].metrics[0].data.data_points
    return len(pts), any(p.attributes.get("otel.metric.overflow") for p in pts)

raw   = lambda i: {"http.request.method":"GET","url.path":f"/users/{i}"}
route = lambda i: {"http.request.method":"GET","http.route":"/users/{id}","url.path":f"/users/{i}"}
print("raw path attr, default limit  ->", run({}, raw))
print("route attr only (allow-list)  ->", run({"attribute_keys":{"http.request.method","http.route"}}, route))

raw path attr, default limit  -> (5000, False)
route attr only (allow-list)  -> (1, False)

With the raw path as an attribute, the SDK exported 5,000 distinct data points, one for each user, for a single metric. With the route template and an attribute allow-list in a View, it exported 1. The views feature is the mechanism: OpenTelemetry’s metrics SDK specification describes views that can specify an allow-list of attribute keys, so attributes you did not intend to aggregate by can be dropped before export. The more robust fix is not to put the raw path in the metric in the first place, since the allow-list is a second line of defense.

The specification also describes cardinality limits: the SDK should enforce a default limit of 2,000 metric points per metric per collection cycle, and when it is exceeded, further combinations are aggregated into a point carrying the attribute otel.metric.overflow=true. This run did not behave that way. The 5,000 points were exported with no overflow point, and a text search of the installed Python SDK’s metrics package for the word cardinality found no match. This indicates that the Python SDK at version 1.45.0 did not apply a default limit in this test; other language SDKs, other versions, or a collector in the pipeline may differ. The practical lesson is to not assume that the SDK or backend will protect you, and to check the limits of the specific components in your pipeline.

Choosing and bounding dimensions

• Start from the question. Each label should enable a question you will ask: by route, by status class, by region. If no one will query by a dimension, do not add it.
• Use templates and classes, not raw values. Route templates instead of paths, status class (2xx, 4xx, 5xx) instead of every code when the distinction does not matter, a small set of customer tiers instead of customer IDs.
• Put high-cardinality detail in traces and logs. A user ID is fine as a span attribute or log field, subject to the privacy section, because those signals store individual events; it is not fine as a metric label. This is the main reason to have three signals.
• Allow-list attributes in a View or collector. Define the permitted attribute set per metric and drop the rest.
• Budget series per service and monitor them. Track the number of series each service produces; review increases at code review. A new label is a change that multiplies cost and deserves the same scrutiny as a new dependency.
• Mind histogram bucket counts. Each bucket multiplies series; choose boundaries around the thresholds you care about, such as an objective, instead of a long list.

Percentiles from histograms

Choosing buckets affects accuracy as well as cost. The Prometheus documentation explains that a histogram stores counts in cumulative buckets with an le (less than or equal) label, that quantiles are estimated from bucket counts, and that the estimation error depends on bucket width. It also explains that precomputed quantiles from a summary generally cannot be aggregated across instances, whereas histogram buckets can be summed across instances before estimating a quantile. The glossary entry on the percentile gives the same warning about averaging per-server values.

The following script generates synthetic latencies, 99,000 from a lognormal body and 1,000 slow requests spread between 1 and 3 seconds, and estimates quantiles from bucket counts alone using linear interpolation inside the bucket, which is the approach described in the documentation. It compares a coarse set of six boundaries with a finer geometric set of 22.

# synthetic latency (seconds): lognormal body plus a 1% slow tail, generated in the script
xs=[random.lognormvariate(-2.2,0.5) for _ in range(99000)]+[random.uniform(1.0,3.0) for _ in range(1000)]
def hist_q(bounds,q):                 # estimate a quantile from bucket counts only, interpolating inside the bucket
    counts=[0]*(len(bounds)+1)
    for x in xs: counts[bisect.bisect_left(bounds,x)]+=1
    cum=0; target=q*n
    for i,c in enumerate(counts):
        if cum+c>=target:
            lo=0 if i==0 else bounds[i-1]; hi=bounds[i] if i<len(bounds) else bounds[-1]
            return lo+(hi-lo)*(target-cum)/c
        cum+=c
coarse=[0.1,0.25,0.5,1,2.5,5]
fine=[0.025*1.25**k for k in range(0,22)]

coarse(6 bounds) {0.5: (0.111, 0.124), 0.95: (0.265, 0.307), 0.99: (1.005, 1.0)}
fine(22 geometric bounds) {0.5: (0.111, 0.112), 0.95: (0.265, 0.273), 0.99: (1.005, 0.899)}
p95 A 0.217 p95 B 2.767 mean of p95s 1.492 pooled p95 1.002

In each pair, the first number is the exact quantile of the sorted data and the second is the bucket estimate. With coarse buckets the p95 estimate is 0.307 s against a true 0.265 s, about 16% too high, because the bucket from 0.25 to 0.5 s is wide and the interpolation assumes values are spread evenly across it. With 22 geometric bounds the estimate is 0.273 s, about 3% high. The p99 case shows a different effect: the finer buckets estimate 0.899 s against a true 1.005 s, because the 99th percentile sits at the edge of the slow tail where the data change abruptly, and linear interpolation inside a bucket spanning that edge misplaces it. Bucket resolution matters most near the percentile and thresholds you care about, and an estimate is an estimate, not a measurement. The coarse set happens to have a boundary at 1 s, exactly where the slow tail begins, so its p99 estimate of 1.0 s is accurate here because of where the boundary fell, not because coarse buckets are better.

The last line demonstrates the aggregation point. Instance A has a p95 of 0.217 s and instance B (half of its requests slow) has 2.767 s. Their average is 1.492 s, while the p95 of all the requests pooled together is 1.002 s. Averaging per-instance percentiles gave an answer about 49% larger than the pooled value in this example, and neither number would be right in general. Aggregate buckets or raw counts, then compute the percentile.

Sampling

Traces cost more than metrics per event, since each request produces many spans with attributes. Sampling reduces what is stored or exported. OpenTelemetry’s documentation describes two approaches. Head sampling decides early, at or near the start of a trace, without seeing the whole trace; it is easy to understand and configure and efficient, but it cannot guarantee that traces with errors are kept, since the decision precedes the outcome. Tail sampling decides after seeing the complete trace and can keep traces by error, latency, or attribute; it is harder to implement and operate, needing stateful, resource-intensive infrastructure, and is often vendor specific. The documentation suggests sampling when you generate high volume (it gives 1,000 or more traces per second as an example) and have clear criteria for quality, and advises against it when you produce little data, use only aggregate analytics, or face regulatory data-retention requirements.

Why the decision must be consistent across services

A trace spans services, and each service makes its own decision unless told otherwise. If every service independently keeps 10% of its spans, a trace crossing four services keeps all four spans with probability 0.1⁴ = 0.0001, which makes complete traces nearly nonexistent. The alternative is to make the decision once and propagate it, or to make every service’s decision a deterministic function of the same input, the trace ID. OpenTelemetry’s trace SDK specification describes a ParentBased sampler that delegates by parent state: a root span uses a root sampler, a remote parent that was sampled leads to sampling by default (AlwaysOn), and an unsampled remote parent leads to dropping by default (AlwaysOff), so the decision made at the root is honored downstream. It also describes TraceIdRatioBased, which must be deterministic and use a hash of the trace ID, so a sampler with a lower probability keeps a subset of what a higher-probability sampler keeps. The specification notes that this sampler is being phased out in favor of a probability-based sampler and will not be removed before 1 January 2027; check the current text when you configure it.

The simulation below compares the two approaches over 20,000 synthetic traces, each crossing four services, at a 10% rate. About 2% of the traces contain an error, which is an assumption.

def ratio_decision(trace_id, rate):   # deterministic function of the trace ID only
    return (trace_id & ((1<<64)-1)) < int(rate*(1<<64))
# 20,000 traces, each crossing 4 services; about 2% contain an error (all synthetic)
indep      = lambda t,s: random.random()<RATE          # each service flips its own 10% coin
consistent = lambda t,s: ratio_decision(t,RATE)        # every service applies the same rule to the same trace ID

independent per-service coin flips  (complete, partial, none): [0.0001, 0.3432, 0.6567]
deterministic on trace ID           (complete, partial, none): [0.0984, 0.0, 0.9015]
error traces: 406 | kept by 10% head sampling: 51 (12.6%)
tail policy 'keep errors + 10% of the rest': traces stored = 2324 of 20000 (11.6%); head stores 1969 (9.8%)
5% set subset of 10% set: True 989 1969
estimated total from kept/rate: 19690 actual 20000

With independent coin flips, only 0.01% of traces are complete (all four spans kept), 34.3% are partial, and 65.7% are dropped everywhere. Partial traces waste storage and mislead, since a trace with a missing middle span looks like a request that skipped a service. Deciding on the trace ID gives 9.84% complete traces, no partial ones, and the remainder dropped, close to the 10% target. The deterministic rule also gave the subset property: the traces kept at a 5% rate (989) were all among the traces kept at 10% (1,969).

Head sampling and rare events

The same simulation shows the limit of head sampling. There were 406 error traces, and a 10% head sampler kept 51 of them (12.6%, near the 10% target given the small number of errors). A decision made before the outcome is known keeps errors at the same rate as everything else, so 90% of the evidence you most need is discarded. A tail policy that keeps all error traces plus 10% of the rest stored 2,324 traces (11.6% of traffic) against 1,969 (9.8%) for the head sampler, so keeping every error trace cost 355 more traces in this synthetic mix, about 1.8 percentage points of traffic. The cost depends on your error rate; at a higher error rate the policy stores proportionally more. What tail sampling costs is operational: spans from one trace must reach a place where the whole trace can be assembled before the decision, which is the stateful infrastructure the documentation warns about.

Sampling and what you can still count

Sampled traces cannot answer questions about totals unless you account for the rate. The last line of the output estimated the total request count by dividing kept traces by the rate (19,690 against the true 20,000); the estimate carries sampling noise and needs the rate to be known and constant. For rates, error ratios, and latency percentiles across all traffic, use metrics, which count every request, and keep traces for investigation. Do not compute an error rate from sampled traces if the sampling depends on errors, since that policy over-represents them by design. A tail sampler’s keep decisions should therefore be treated as a way to store evidence, not as a statistically representative sample.

Trace propagation

Propagation is what makes a trace span processes. The W3C Trace Context Recommendation (published 23 November 2021 as version 1.0) defines the traceparent header as four hyphen-separated fields: a 2-hex-digit version, a 32-hex-digit trace ID (16 bytes, not all zeros), a 16-hex-digit parent ID (8 bytes, not all zeros), and 2 hex digits of trace flags. An example from the specification is 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. The least significant bit of the flags is the sampled flag; when set to 1 it means the caller may have recorded trace data, which lets downstream services honor an upstream decision. The companion tracestate header carries vendor-specific key-value pairs, with at most 32 list members, and implementations are expected to propagate at least 512 characters of it. OpenTelemetry’s default propagator uses this format, and baggage is a separate mechanism for propagating arbitrary key-value pairs.

On the receiving side, the service extracts the remote context from the carrier, which makes the incoming trace ID and parent span ID the parent of the span it creates. The program above shows this exactly: prop.inject(headers) wrote the header, and prop.extract(headers) produced the context passed to the payments span. In real services, the HTTP instrumentation libraries do both steps for you, which is why zero-code instrumentation covers most traffic.

Where propagation breaks

• Header stripping. A proxy, gateway, or custom HTTP client that does not forward unknown headers cuts the trace in two. The downstream service starts a new trace, and both halves look complete. Check each intermediary, and test a request across the real path, not just between two services.
• Lost in-process context. Work handed to a thread pool, a callback, or an async task can run without the current context unless the runtime carries it. Spans created there appear as new roots. Use the instrumentation your language provides for the concurrency mechanism you use, and test it.
• Messaging and batch work. A message queue has no request-response structure, so the producer must write the context into the message’s headers or attributes, and the consumer must read it. When one consumer handles a batch built from many producers, a single parent is wrong; OpenTelemetry’s span links associate a span with one or more other spans for such causal relationships, particularly for asynchronous operations.
• Unsampled but propagating. A service that receives a context with the sampled flag unset should still forward the context, so the downstream services see the same decision. Dropping the context makes downstream services start new roots and sample on their own.
• Clock differences. Span timestamps come from the clocks of the hosts that produce them. When clocks differ, a child can appear to start before its parent. Interpret small negative gaps as possible clock skew before concluding the trace is wrong.

Trust boundaries

A traceparent header arriving from outside your system is input from an untrusted party. The security and privacy section of the Trace Context specification warns that naively continuing a trace with the sampled flag set lets an attacker overwhelm the application with tracing overhead, forge trace ID collisions that make monitoring data unusable, or increase a tracing bill. It suggests defining trust boundaries on upstream and downstream connections and restarting traces at security perimeters, along with rate limiting, treating authenticated and unauthenticated requests differently, and validating the randomness of incoming IDs. It also says random number generators must not rely on user-identifiable information. The simplest policy follows from this: at the edge where external clients enter, start a new trace and record the incoming identifier, if you keep it at all, as an attribute, not as the parent. Inside the system, continue the trace. Baggage deserves the same caution: OpenTelemetry’s guidance says to avoid sensitive data such as credentials or personal information in baggage, since it crosses service boundaries and could be logged or sent to untrusted downstream services.

Diagnosis with the three signals

The following scenario is hypothetical, constructed to show how the signals are used together; none of the numbers are measurements. A service “checkout” calls “payments” and “inventory”. Instrumentation follows the practices above: route-templated spans, service.name and version on every signal, a duration histogram with route and status class labels, trace IDs in logs.

Step	Signal	What you do	What it tells you	What it does not
1. Detect	Metric	An alert fires on a rise in p95 of the request-duration histogram for POST /orders. Compare against the previous release’s version label	The symptom is real, started at a time, and affects one route	Which requests are slow or why
2. Narrow	Metric	Split by the labels available: status class, version, region	The slowdown is on the new version and not tied to errors	Which dependency is involved
3. Locate	Trace	Find sampled traces for slow POST /orders requests on the new version and read their span timing	In these traces the call from checkout to payments takes most of the time, while payments’ own server span is short	Whether sampled traces are representative
4. Explain	Log	Search logs by the trace IDs of those traces, in checkout and payments	Checkout logs show repeated retries before the call succeeded; the retry setting changed in the new version	How common this pattern is across all requests
5. Verify	Metric	Compare retry-count metric or histogram before and after, by version	Whether the pattern holds across all traffic	Causation by itself; confirm by reverting or testing the setting

Three details in step 3 are easy to get wrong. A client span in checkout covering the payments call includes time spent in network transit, queuing, and the payments server span; if the client span is long and the server span is short, the time is elsewhere (connection setup, retries, queuing), which is what step 4 then explains. Spans can contain child operations or overlap with each other, so summing every span duration double counts time; examine the timeline and the parallelism. And a technically successful span does not establish that the business outcome occurred, so check the outcome separately. The glossary entry on request tracing makes the same points.

This workflow depends on instrumentation decisions made earlier: the version attribute that made step 2 possible, the route template that kept the metric affordable and the span name readable, the propagation that kept the trace whole across the call, the trace ID in the log lines that made step 4 a search, and a sampling policy that kept slow or error traces for step 3. If any one is missing, the investigation falls back to guessing from one signal.

The next article in this series, Service Objectives, Alerting, and Error Budgets in Practice, builds on this: it takes the user-visible measurements you now have and shows how to define objectives, alert on them, and decide when to stop releasing. Paging on a cause such as CPU in addition to the symptom it produces is one of the duplicated-notification patterns the entry on alert fatigue describes, and symptom-level signals like the duration histogram above are what make a cleaner paging rule possible.

Privacy controls

Telemetry collects data about what services do, and what services do involves people. OpenTelemetry’s guidance on handling sensitive data lists examples such as personal information, authentication credentials, session tokens, financial data, health information, and user behavior patterns, says the implementer is responsible for compliance with privacy regulations and for reviewing the telemetry that instrumentation libraries produce, and states a principle: only collect data that serves an observability purpose. Treat these as engineering controls, not legal advice; which regulations apply to your data is a question for the people responsible for compliance.

Where sensitive data enters telemetry

• URLs and query strings. A full URL attribute can contain an email address or a token in the query string.
• Identifiers. User IDs, session IDs, and account numbers are attached to spans or logs for convenience.
• Log message bodies. Free-text messages interpolate request fields. Exception messages and stack traces can include values from the failing input.
• Headers and request or response bodies captured by instrumentation for debugging.
• Baggage that propagates a value to every downstream service.
• Metric labels, which combine a privacy problem with a cardinality problem.

Layered controls

Prevent at the source first. The most reliable way to keep a value out of the telemetry store is for the application not to emit it. The log redaction entry describes the limit of later stages: collector redaction does not erase copies already written elsewhere, and an input spool may hold the original before a filter runs. Use an allow-list of attributes, not a deny-list. A deny-list protects against the fields you thought of; an allow-list drops everything you did not approve, including fields added by a library upgrade.

The code below applies an allow-list to a set of span attributes and then tests the common idea of anonymizing an ID by hashing it. Everything is made up: the host is an invalid example domain, and the email and ID are fake.

ALLOWED={"http.request.method","http.route","http.response.status_code","order.status"}
span_attrs={"http.request.method":"POST","http.route":"/orders/{id}","http.response.status_code":201,
            "url.full":"https://api.example.test/orders/1042?email=user@example.test",
            "enduser.id":"184223","order.status":"created"}
safe={k:v for k,v in span_attrs.items() if k in ALLOWED}      # allow-list, not deny-list

h=hashlib.sha256(b"184223").hexdigest()                        # "anonymize" a numeric ID by hashing it
for i in range(0,1_000_000):
    if hashlib.sha256(str(i).encode()).hexdigest()==h: found=i; break

allow-list result: {"http.request.method": "POST", "http.route": "/orders/{id}", "http.response.status_code": 201, "order.status": "created"}
recovered ID from unsalted hash: 184223 in 0.26 s

The allow-list removed the full URL (with its email query parameter) and the user identifier, and kept the four attributes that serve an operational purpose. The hashing test shows why hashing is not a full answer: for an ID drawn from a small, predictable space, a plain SHA-256 hash can be reversed by enumerating candidate values. Here it took 0.26 seconds on the machine used to recover a six-digit ID from its hash by trying numbers in order. OpenTelemetry’s guidance makes the same caution, noting that hashing has limits for small predictable inputs like numeric IDs. A keyed hash with a secret held outside the telemetry system resists enumeration better, though anyone who holds the key can still link records, so it is pseudonymization and not anonymization.

The OpenTelemetry Collector offers processors to apply these controls centrally. Its documentation names an attributes processor to remove or modify fields, a filter processor to exclude whole spans or metrics, a redaction processor to delete attributes outside an allowed list, and a transform processor to modify data using patterns such as regular expressions. Collector-side controls are a useful backstop, particularly for libraries you do not control, and they protect everything downstream of the collector only; data may already exist in application logs and local files.

Other controls belong in the design. Limit access to each signal by purpose, since traces and logs tend to hold more detail than metrics. Set retention per signal and for sensitive stores, and make sure deletion obligations reach the telemetry backend. Sample with care: sampling reduces volume but does not make retained data less sensitive. Test the controls with realistic inputs, and treat the result as evidence about the patterns tested; a test that finds no sample tokens shows that those patterns were handled, not that no sensitive data is present.

Operating the telemetry pipeline

• Monitor the monitors. Track dropped spans, exporter queue sizes, collector health, and the volume of data per service. A stalled pipeline produces silence that resembles health.
• Own the defaults. The platform team owns SDK configuration, resource attributes, sampling defaults, redaction rules, and the allowed attribute lists, and publishes how service teams extend them.
• Review cost by signal. Series count for metrics, spans stored for traces, and bytes for logs. Attribute spending to services, so teams see the cost of their choices.
• Version and test propagation. Include a request that crosses every proxy and queue in a test, and fail the test if the trace ID changes.
• Keep a data dictionary. For each attribute and label: meaning, allowed values or cardinality bound, sensitivity classification, and owner.

Failure modes and limits

• Broken traces from missing propagation look like separate healthy requests.
• Logs without trace IDs cannot be tied to a request, which removes the cheapest correlation.
• Unbounded labels exhaust the metrics backend or its budget, sometimes at once, sometimes gradually as churn adds series.
• Independent sampling yields fragments; head sampling discards most rare errors; tail sampling needs infrastructure that can fail under load and needs capacity planning.
• Sampled data used as totals produces wrong rates unless the rate is known and the sampling is independent of the outcome.
• Percentiles from coarse buckets misplace the tail, and averaged percentiles are wrong.
• Over-trust of incoming headers lets outside clients influence your sampling and trace data.
• Sensitive data in telemetry persists in several stores with different retention and access rules.
• Instrumentation overhead and gaps. Instrumentation costs CPU, memory, and network; that cost was not measured in these experiments. Libraries that are not instrumented, or are instrumented differently, leave gaps in traces.
• Evidence limits. The experiments here use synthetic data and one SDK version, run in a single process, with the two services simulated by function calls. They show mechanisms and do not measure real network behavior, collector throughput, or backend cost.

References

• OpenTelemetry: Signals
• OpenTelemetry: Context propagation
• OpenTelemetry: Traces (spans, span context, links)
• OpenTelemetry: Instrumentation
• OpenTelemetry: Sampling
• OpenTelemetry specification: Trace SDK (samplers)
• OpenTelemetry specification: Metrics SDK (cardinality limits, views)
• OpenTelemetry specification: Logs data model
• OpenTelemetry semantic conventions: HTTP spans
• OpenTelemetry semantic conventions: Resource
• OpenTelemetry: Handling sensitive data
• W3C Recommendation: Trace Context
• Prometheus: Data model
• Prometheus: Metric and label naming
• Prometheus: Histograms and summaries
• Google SRE Book: Monitoring Distributed Systems