AI Observability and Sensitive Data

Consider a hypothetical support assistant. A customer writes in, the assistant retrieves a few account notes, calls a customer-lookup tool, and answers. One afternoon the p95 latency doubles and a manager asks why. The on-call engineer opens a trace and finds a gap: the lookup tool took ten seconds. That is a useful finding, and it needed no customer data at all. A week later a different complaint arrives: the assistant told a customer a refund was approved when it was not. Now the engineer wants the exact prompt, the retrieved notes, and the tool result. All of them contain names, email addresses, and, in some requests, a card number someone pasted into the chat.

That tension is the subject of this post. An AI system’s behavior is determined by its content (the prompt, the retrieved text, the tool results), so the most useful debugging record is the one that contains the most sensitive data. Telemetry systems are built to be broadly readable by engineers, retained for weeks, copied to vendors and dashboards, and searched by anyone with access. Writing content into them moves customer data into a place with weaker controls than the place it came from.

The approach in this post has three parts. First, decide what each request leaves behind by default, and make that mostly structure and measurements rather than content. Second, where content must be kept, remove sensitive values before they leave the process, and know how well that removal works; this post measures it on synthetic data and shows where it fails. Third, control who can read what remains and for how long. It assumes you have read Prompt and Model Promotion with Evaluation Gates, which covers versioned evaluation and release thresholds, and Agent Runtime Isolation, Tools, and Execution Policies, which covers tool calls, execution policies, and action audit. It also assumes familiarity with logs, metrics, and traces as the three telemetry signals.

What one AI request looks like to a tracing system

A distributed trace records one request as a tree of spans. A span is a timed operation with a name, a start and end, a status, and key-value attributes. The parent-child links show which operation caused which. For an ordinary web service the tree is fairly stable: a request handler, a database call, a cache lookup. For an AI request the tree is shaped by the model’s decisions. An agent may call a tool once or six times, retrieve before and after a tool result, or retry a model call with a different prompt. Two requests to the same endpoint can produce differently shaped traces, and the shape itself is information.

A typical request in the support assistant has this structure:

request (root span, one per user request)
|-- retrieval            vector search over account notes     k=5, returned 4, top score 0.81
|-- chat small-v1        first model call                      input 1,820 tokens, output 94, finish: tool_calls
|-- execute_tool lookup_customer                               status ok, 389 ms
|-- chat small-v1        second model call, with tool result   input 2,104 tokens, output 212, finish: stop
`-- response             rendered to the user                  status ok

Alongside the trace, metrics aggregate the same operations across requests: histograms of model-call duration, tool-call duration, and token counts, each broken down by a small number of labels such as model and tool name. Logs record discrete events, such as a policy decision to deny a tool call. A trace explains one request; metrics tell you whether many requests share the problem; logs carry detail that does not fit a span.

The GenAI semantic conventions

OpenTelemetry is the vendor-neutral standard for these signals, and it defines semantic conventions, which are agreed names for spans and attributes so that different tools and libraries describe the same thing the same way. Its conventions for generative AI live in a dedicated repository, open-telemetry/semantic-conventions-genai, and their status in that repository is Development, which in OpenTelemetry’s vocabulary means they may still change in incompatible ways. In practice this means attribute names you build dashboards on can be renamed. Pin the instrumentation library versions you use, and keep your own constants for the attribute names you query, so a rename is a one-line change instead of a hunt through dashboards.

The conventions define the operations you would expect. gen_ai.operation.name takes values such as chat, embeddings, retrieval, execute_tool, and invoke_agent. A model-call span is named {operation} {model}, for example chat small-v1; a tool span is named execute_tool {tool name}; an agent span is named invoke_agent {agent name}. Metadata attributes such as gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons are recommended or conditionally required. Metrics include gen_ai.client.operation.duration, gen_ai.server.time_to_first_token, and gen_ai.execute_tool.duration. All of these are measurements and identifiers. None contains the text of a conversation.

Content is separate and opt-in. The attributes that carry it are gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, gen_ai.tool.call.result, gen_ai.retrieval.query.text, and gen_ai.retrieval.documents, each marked Opt-In, and the specification notes that several of them are likely to contain sensitive information including personal data. The specification’s own guidance is that instrumentations should not capture instructions, inputs, or outputs by default, because content is large, may be regulated, and may need different access rules from operational data. It names three usage patterns: record no content (the default); record content on the spans, which it suggests for volumes that are manageable and for environments such as pre-production where privacy rules do not bite; or store content externally and record only references on the spans, which it recommends for production because the external store can have its own access controls.

The third pattern is the architecture the rest of this post builds toward. The trace carries a reference to content; the content lives in a different store with a different audience and a different retention clock.

Where sensitive values actually appear

Teams that decide to redact the prompt and the response often discover that those are not the only places. Sensitive values enter telemetry through at least six paths.

• User prompts and model responses. Users paste what they have. A response often echoes it back.
• Retrieved documents. A retrieval span can hold chunks of other people’s records: account notes, tickets, emails. The requesting user may not even be entitled to see them directly; the retriever fetched them on the model’s behalf.
• Tool arguments and results. A lookup tool receives an email address and returns a record. Both sides are content.
• Error messages and exceptions. NotFound: no customer with email jane@example.com is a natural thing for a service to say, and exception text is routinely attached to spans and logs.
• Identifiers. User IDs, session IDs, conversation IDs, and IP addresses are personal data in many regimes even when they are not names.
• Infrastructure instrumentation. HTTP client instrumentation can record URLs with query strings and, if configured, request bodies, which for a model API call is the prompt again.

The experiment below makes this concrete. It generates 1,000 synthetic support-assistant records, each with a user prompt, retrieved notes, a tool call with arguments and a result, and a response, and plants fabricated values in them. The values are fake: names drawn from a short invented list, example.com addresses, phone numbers in the 555-01xx fictional range, checksum-valid card numbers in test prefixes, and social-security-style numbers in the 900 range that is not issued. The record layout and the number formats were constructed for this experiment, so the results measure these formats and this mix of easy and hard cases, not real traffic. Treat them as a demonstration of mechanisms and of the shape of the failures rather than as expected detection rates.

Environment for all experiments in this post: Python 3.12.14 (standard library only) on macOS (Darwin 25.6, x86_64). Counting only email addresses, phone numbers, card numbers, social-security-style numbers, API-key-style tokens, and IPv4 addresses, the planted values by field are:

values per field:
  {'user_prompt': 1736, 'retrieved': 2517, 'tool_args': 712, 'tool_result': 712, 'response': 200, 'exception': 112}

The retrieved notes hold more sensitive values than the user’s own prompt, which is a fact about how this data is laid out, but the pattern is general: the model’s context contains many people’s data, and the person who typed the prompt is only one of them. A policy that scrubs only the user’s message misses the largest source.

Choosing what to record: four levels

Before choosing a redaction technique, decide what a request leaves behind. It helps to think in levels, ordered by how much they reveal.

Level	What is recorded	Answers	Main risk
0. Metadata	Operation names, model and template versions, token counts, durations, status and finish reason, counts (documents retrieved, tool calls), scores, error class, keyed pseudonym of the user	Is it slow, failing, truncated, over budget, retrieving nothing, or calling a broken tool?	Low, if every field is constrained to numbers, enumerations, or short validated codes
1. Derived content features	Length buckets, language, whether a tool was called, evaluation scores computed offline, flags from classifiers	Which kinds of requests behave differently; whether quality dropped	Low to moderate; a feature computed from content can still be revealing in a small population
2. Redacted content	Prompt, response, retrieved text, and tool data, with sensitive values masked at collection	What did the model see and say, approximately?	Whatever the redaction misses; measured below
3. Raw content in a restricted store	Full content, referenced by ID from the trace	Exactly what happened, for incident review and evaluation data	A concentrated store of customer data; needs strict access, short retention, and deletion support

Level 0 is the default for every request. Most operational questions are answered there, as the diagnosis experiment later in this post shows. Levels 2 and 3 are chosen per use: for a sampled fraction of traffic, for requests that failed, for requests from tenants that opted in, or for internal and synthetic traffic where no customer data is present. Level 1 deserves attention because it is where quality monitoring usually lives. An offline evaluator can read content in its own controlled environment and emit only a number, for example a groundedness score, back to the telemetry system. That number is metadata. This is how the evaluation work from the previous module connects to production monitoring without copying content into the telemetry path.

Redaction at collection time

Where redaction can run

Redaction can run in the application process before the SDK exports data, in a collector between the application and the backend, or in the backend after ingestion. The later it runs, the more places the raw value has already been: process memory, a local buffer, the network, a collector’s queue, a disk spool. Log redaction makes the same point for logs: a filter in a pipeline does not erase copies that were written before the filter ran. In-process redaction (or, better, not putting the value on the span in the first place) gives the strongest guarantee, at the cost of running inside every service and shipping with each release. A collector applies one policy to many services and can be changed without redeploying them, but sees raw data on the way in. Backend-only redaction is the weakest and should be a last line, not the design.

The OpenTelemetry Collector has a redaction processor in its contrib distribution that illustrates the allowlist-then-blocklist design. Its documentation describes it as removing span, log, and metric attributes that are not on an allowlist of keys and masking attribute values that match blocked patterns, with an option to replace matches with a hash instead of a fixed mask. The same documentation says the processor is meant as one line of defense rather than the only compliance measure. A configuration sketch (not run here; option names are from the processor’s README, so check the README for the version you deploy):

processors:
  redaction:
    allow_all_keys: false        # drop any attribute key not listed below
    allowed_keys:
      - gen_ai.operation.name
      - gen_ai.request.model
      - gen_ai.response.finish_reasons
      - gen_ai.usage.input_tokens
      - gen_ai.usage.output_tokens
      - gen_ai.tool.name
      - error.type
      - content.ref              # reference into the restricted content store
    blocked_values:              # mask matching values in the keys that remain
      - "\\b\\d{3}-\\d{2}-\\d{4}\\b"
    summary: info                # log counts of what was removed or masked

The allowlist half of this design is structural: a content attribute that was never listed is dropped no matter what it contains. The blocklist half is pattern matching on what remains, and that is the half whose limits need measuring.

An experiment: how well do patterns find sensitive values?

Three pattern-based redactors were written and run against the 1,000 synthetic records. The first, v1 strict, matches only well-formed formats: an email address, a phone number with separators, a card number in four groups of four, a dashed social-security-style number, a token with a known prefix, and an IPv4 address.

# ---- v1: strict formats only
V1 = [
    ("email", re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")),
    ("phone", re.compile(r"(?:\+1[ -]?)?(?:\(\d{3}\)\s?|\d{3}[-. ])\d{3}[-. ]\d{4}")),
    ("card",  re.compile(r"\b(?:\d{4}[ -]){3}\d{4}\b")),
    ("ssn",   re.compile(r"\b\d{3}-\d{2}-\d{4}\b")),
    ("api_key", re.compile(r"\btk_live_[A-Za-z0-9]{24}\b")),
    ("ip",    re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b")),
]

The second, v2 loose, adds the formats people actually type: bare 10-digit runs as phone numbers, bare 9-digit runs as social-security-style numbers, bare 16-digit runs as cards, and 40-character hex strings as secret tokens. The third, v3 gated, tries to buy back precision. A bare digit run only counts as a phone number or identifier when a keyword such as call, phone, social, or token appears just before it. A card-shaped number must pass the Luhn checksum, the check-digit rule used by payment card numbers: counting from the right, double every second digit, subtract 9 from any result above 9, and require the sum to be divisible by 10. Four-octet strings after the word build are treated as version numbers, and IPv4 octets are limited to 255.

def luhn_ok(s):
    d = [int(c) for c in re.sub(r"\D", "", s)]
    if not 13 <= len(d) <= 19: return False
    t = 0
    for i, x in enumerate(reversed(d)):
        if i % 2 == 1:
            x *= 2
            if x > 9: x -= 9
        t += x
    return t % 10 == 0

# ---- v3: bare forms need a keyword nearby (or a checksum); IPv4 octets must be <= 255
KW_PHONE = re.compile(r"(?:call|phone|reach|tel)[^.]{0,20}?(\b\d{10}\b)", re.I)
KW_SSN = re.compile(r"(?:social|ssn)[^.]{0,20}?(\b\d{9}\b)", re.I)
KW_KEY = re.compile(r"(?:token|key|secret)[^.]{0,12}?(\b[0-9a-f]{40}\b)", re.I)
IP_STRICT = re.compile(r"\b(?:(?:25[0-5]|2[0-4]\d|1?\d?\d)\.){3}(?:25[0-5]|2[0-4]\d|1?\d?\d)\b")

def spans_v1(t): return [(m.start(), m.end(), c) for c, r in V1 for m in r.finditer(t)]
def spans_v2(t): return [(m.start(), m.end(), c) for c, r in V2 for m in r.finditer(t)]

def spans_v3(t):
    out = []
    for c, r in V1:
        if c == "ip":
            continue
        out += [(m.start(), m.end(), c) for m in r.finditer(t)]
    out += [(m.start(), m.end(), "ip") for m in IP_STRICT.finditer(t)]
    # four-octet strings preceded by 'build' are version numbers
    out = [s for s in out if not (s[2] == "ip" and re.search(r"build\s*$", t[max(0, s[0] - 7):s[0]]))]
    for r, c in ((KW_PHONE, "phone"), (KW_SSN, "ssn"), (KW_KEY, "api_key")):
        out += [(m.start(1), m.end(1), c) for m in r.finditer(t)]
    # bare 16-digit runs only with a valid Luhn checksum
    out += [(m.start(), m.end(), "card") for m in re.finditer(r"\b\d{16}\b", t) if luhn_ok(m.group())]
    # grouped card-shaped strings must pass the checksum too
    out = [s for s in out if not (s[2] == "card" and not luhn_ok(t[s[0]:s[1]]))]
    return out

Scoring is strict. A planted value counts as caught only if every character of it falls inside a detected span, so a detector that masks the area code but leaves the rest of a phone number is scored as a miss. The benign set contains strings that look sensitive but are not: ten-digit order numbers, sixteen-digit tracking numbers, nine-digit part numbers, build versions that look like IP addresses, and 40-character commit hashes. A good redactor leaves these alone, because they are exactly the identifiers an engineer needs to keep when debugging. The results with every string field scanned:

recall by category, all fields scanned (percent of planted values fully covered)
detector      email    phone     card      ssn  api_key       ip  address     name six-pattern
v1 strict      90.1     69.3     76.7     76.4     70.6    100.0      0.0      0.0     84.8
v2 loose       90.1     90.5    100.0    100.0    100.0    100.0      0.0      0.0     92.1
v3 gated       90.1     86.1    100.0     88.2     79.2    100.0      0.0      0.0     89.7

benign look-alikes flagged (percent of benign strings that were masked)
detector        order_no   tracking_no build_version       git_sha           sku
v1 strict            0.0           0.0         100.0           0.0           0.0
v2 loose           100.0         100.0         100.0         100.0         100.0
v3 gated             0.0           6.9           0.0           0.0           0.0
benign counts: {'order_no': 3104, 'tracking_no': 291, 'build_version': 287, 'git_sha': 203, 'sku': 303}

Several things are visible in these numbers.

• Loosening the patterns raises recall and destroys the benign identifiers. v2 loose catches 92.1 percent of the six pattern-addressable categories, up from 84.8 percent for v1 strict, and it masks every order number, tracking number, part number, and commit hash in the data set. A trace whose order IDs are all [PHONE] cannot be joined to the order system. Whether that matters depends on the debugging you need, but it is a cost, and it is paid on every request rather than only on the sensitive ones.
• Gating recovers the identifiers but loses recall that depends on context. v3 gated masks none of the order numbers, part numbers, build versions, or commit hashes. It also lets through bare phone numbers, tokens, and social-security-style numbers that appear without a trigger word nearby, which is why its recall sits between the other two (89.7 percent). Its 6.9 percent false-positive rate on tracking numbers is the checksum at work: about one random 16-digit number in ten passes the Luhn test by chance. The gating rules also encode assumptions about English phrasing that a different language or template would break.
• No pattern reaches names or addresses. Recall is zero for both because none of these detectors tries. Detecting a person’s name needs a model or a dictionary rather than a regular expression, and those have their own misses and false positives; they were not measured here. Names are among the most common values in a support conversation.
• The email column does not move. All three detectors catch 90.1 percent. The remaining 9.9 percent are the obfuscated forms in the data (name at example dot org), which no email pattern matches. Phone numbers written out as words fail the same way. Obfuscation is not only an adversary’s tool; people write this way in chat. Whether it appears in your traffic at the rate planted here is an empirical question about your traffic.

The results above assume the redactor sees every field. Real deployments often scan the fields they think of as content. The next comparison runs the best detector, v3 gated, on only the prompt and the response, and then on every string field including retrieved text, tool arguments, tool results, and exception messages:

leaked values by field (v3 gated; names excluded as no pattern targets them)
prompt+response only   leaked non-name values: 4550 of 6277
all string fields      leaked non-name values: 906 of 6277
by field with prompt+response-only scope: {'retrieved': 2517, 'tool_args': 712, 'tool_result': 712, 'user_prompt': 481, 'response': 16, 'exception': 112}

records with >=1 leaked email/phone/card/ssn/key/ip value after v3 on all fields: 432 of 1000

Scanning only the prompt and the response leaves 4,550 of 6,277 non-name values exposed, because most of them sit in retrieved text, tool arguments and results, and exception messages. Scanning everything leaves 906. That difference, 3,644 values, comes from scope alone and has nothing to do with how clever the patterns are. The remaining 906 are the format misses described above, plus the 288 street addresses that no detector targets. Counted per record, 432 of the 1,000 records still hold at least one email, phone, card, social-security-style number, token, or IP address after the best detector ran over every field.

Two records show what the surviving values look like:

IN : Hi, this is Priya Moreau. Ship the replacement to 636 Willow Lane, Oakridge. The failing client was 151.151.44.16. My number: four one eight five five five zero one two six. The tracking page says tracking 5459465973168979. The app is build 39.135.131.5.
OUT: Hi, this is Priya Moreau. Ship the replacement to 636 Willow Lane, Oakridge. The failing client was [IP]. My number: four one eight five five five zero one two six. The tracking page says tracking 5459465973168979. The app is build 39.135.131.5.
IN : Hi, this is a customer. mariaduarte at example dot net is my work address. Auth header carried 48d33296c87009e8a7f770d9106fd287db7f1adb. Ship the replacement to 415 Harbor Street, Oakridge. It concerns order 3433734419. The app is build 58.125.107.234.
OUT: Hi, this is a customer. mariaduarte at example dot net is my work address. Auth header carried 48d33296c87009e8a7f770d9106fd287db7f1adb. Ship the replacement to 415 Harbor Street, Oakridge. It concerns order 3433734419. The app is build 58.125.107.234.

The first record has its IP address masked and a phone number written as words left in place. The second keeps an email written as mariaduarte at example dot net and a 40-character hex token that follows Auth header carried, a phrase the keyword gate does not include. In both records the commit hash and build version, which are not sensitive, are intact. This is the behavior of the gated detector doing what it was designed to do. The design simply cannot be complete.

What this means for how redaction is used

Pattern-based redaction is a risk reducer, and its failures are silent: a value that slips through looks like any other text. Four practices follow.

1. Do not use redaction as the reason content is safe to retain broadly. The residual rate above is not small, and any given record can contain what the detector missed. Treat redacted content as still sensitive, with access and retention set accordingly.
2. Scan every field that can carry text, and keep the list of fields explicit. Add exception messages, tool arguments, and retrieved text to the review checklist for each new span type.
3. Prefer not to record a value over finding it afterward. If the retrieved documents are only needed to compute a count and a top score, record the count and the score. The OpenTelemetry guidance on sensitive data makes the same recommendation: prevent collection rather than clean up afterward.
4. Test the redactor against planted values continuously, using synthetic traffic with known canary values (described below), so a change in template or format that defeats a pattern shows up as a failing test instead of as an incident.

Hashing and tokenization

When engineers need to correlate the same user or the same customer across spans without seeing who it is, the usual substitute is a hash or a token. A plain cryptographic hash is not a safe substitute for a value with a small space of possibilities, because an attacker can compute the hash of every possible value and look for a match. Phone numbers, social-security-style numbers, and email addresses with known domains all have limited structure. The OpenTelemetry guidance on sensitive data warns specifically that hashing a user’s ID or name may not provide the anonymity you need when the input space is small and predictable.

The following script hashes a fabricated phone number with SHA-256, then recovers it by enumerating every number in one area code (200 to 999 for the exchange, 0000 to 9999 for the line). It also builds a keyed digest of the same number with HMAC-SHA-256 and shows that the same enumeration cannot match it without the key.

import hashlib, hmac, time

# ---------- 1. hashing a low-entropy identifier
secret = b"lab-only-key-not-a-real-secret"
target = "4155550134"                       # fabricated number
plain_digest = hashlib.sha256(target.encode()).hexdigest()
keyed_digest = hmac.new(secret, target.encode(), hashlib.sha256).hexdigest()
print("plain sha256 :", plain_digest[:16], "...")
print("hmac-sha256  :", keyed_digest[:16], "...")

t0 = time.perf_counter(); tried = 0; found = None
for nxx in range(200, 1000):                 # exchange codes 200-999
    for line in range(10000):
        cand = f"415{nxx}{line:04d}"
        tried += 1
        if hashlib.sha256(cand.encode()).hexdigest() == plain_digest:
            found = cand; break
    if found: break
dt = time.perf_counter() - t0
print(f"enumerating one area code: found {found} after {tried:,} candidates in {dt:.1f}s")
# the same enumeration against the keyed digest, without the key, over the whole space
miss = 0; t0 = time.perf_counter()
for nxx in range(200, 1000):
    for line in range(0, 10000, 97):       # a 1% stride is enough to show the digest never matches
        if hashlib.sha256(f"415{nxx}{line:04d}".encode()).hexdigest() == keyed_digest: miss = -1
print("unkeyed enumeration against the keyed digest matched:", miss == -1)

plain sha256 : 86a907bbd527b362 ...
hmac-sha256  : e63aa8663453f75b ...
enumerating one area code: found 4155550134 after 3,550,135 candidates in 7.3s
unkeyed enumeration against the keyed digest matched: False

The plain hash fell after 3,550,135 guesses in about 7 seconds in unoptimized Python on the machine described above. The attacker also needs to know the area code, which is rarely secret, and optimized or parallel implementations would be much faster. The keyed digest was not matched, because the enumeration lacks the key. The stride of 97 (about 1 percent of the line numbers) in the second loop is only there to keep the script short, because every guess against a keyed digest is expected to fail; it is not a measure of strength.

Keyed pseudonyms have costs of their own. The key becomes a secret to manage and rotate. The mapping is deterministic, so anyone who holds both the key and a candidate value can confirm the identity, and rotating the key breaks correlation across the rotation boundary. A pseudonym that stays stable for years still lets a person with access to telemetry build a behavioral profile of one user. If a request must be tied to a person only occasionally, for support or deletion, a lookup table kept in a restricted store (a token vault) avoids exposing even a keyed derivation.

Metadata-only export with an allowlist

The alternative to finding sensitive values inside content is to never export content. Define the exported record as a fixed set of named fields, each with a type constraint, and drop everything else. This is the allowlist idea from the collector, applied in the application. The function below builds the exported form of each synthetic record. Strings are only allowed where they come from a closed set (the tool name is checked against a registry, the error class is matched against a fixed list of prefixes, and the exception message itself is dropped), sizes are reduced to order-of-magnitude buckets, and the user is represented by a keyed pseudonym.

import hashlib, hmac, math, re

# ---------- 2. allowlist export
TOOL_ENUM = {"lookup_customer", "refund_order", "search_orders"}
ERR_CLASS = re.compile(r"^([A-Za-z]+Error|NotFound|Timeout|Forbidden)\b")

def bucket(n):                               # coarse size class: 0, 1-9, 10-99, ...
    return 0 if n == 0 else 10 ** int(math.log10(n))

def pseud(v):                                # keyed pseudonym, stable within a deployment
    return hmac.new(secret, v.encode(), hashlib.sha256).hexdigest()[:12]

def export_allowlisted(rec):
    tools = []
    for t in rec["tool_calls"]:
        name = t["name"] if t["name"] in TOOL_ENUM else "unknown_tool"
        m = ERR_CLASS.match(t["error_message"]) if t["error_message"] else None
        tools.append({"tool": name, "status": t["status"], "latency_ms": int(t["latency_ms"]),
                      "error_class": m.group(1) if m else ("other" if t["error_message"] else None)})
    return {
        "request_id": rec["request_id"], "tenant": rec["tenant"], "user_key": pseud(rec["user_id"]),
        "model": rec["model"], "template": rec["prompt_template_version"],
        "in_tokens": rec["in_tokens"], "out_tokens": rec["out_tokens"], "latency_ms": rec["latency_ms"],
        "finish_reason": rec["finish_reason"], "status": rec["status"],
        "prompt_chars_bucket": bucket(len(rec["user_prompt"])), "response_chars_bucket": bucket(len(rec["response"])),
        "retrieved_count": len(rec["retrieved"]),
        "retrieval_top_score": max((r["score"] for r in rec["retrieved"]), default=None),
        "tool_calls": tools,
    }

Run over the 1,000 records, searching the exported JSON for every planted value of five characters or more, the result is below, followed by one exported record and the tool-name test described afterward:

allowlisted export: planted values (len>=5) found in exported records: 0
example exported record:
{
 "request_id": "r00001",
 "tenant": "t-b",
 "user_key": "1e90b25ea31a",
 "model": "large-v2",
 "template": "support-v7",
 "in_tokens": 1822,
 "out_tokens": 83,
 "latency_ms": 1140,
 "finish_reason": "stop",
 "status": "ok",
 "prompt_chars_bucket": 100,
 "response_chars_bucket": 10,
 "retrieved_count": 3,
 "retrieval_top_score": 0.94,
 "tool_calls": [
  {
   "tool": "lookup_customer",
   "status": "ok",
   "latency_ms": 389,
   "error_class": null
  }
 ]
}

model-chosen tool name exported as: unknown_tool

Zero planted values appear in the export. This is a structural result, not a detection result: the exporter has no path by which a free-text field can enter, so there is nothing for a format to evade. The guarantee holds only as long as each field stays constrained. The last test shows one way it can erode. A model chooses tool names, and a model can be induced to choose a name that embeds user data; the exporter checks the name against the registry and emits unknown_tool instead of the string. Three other changes would silently reintroduce content: a developer adding a note or error_message field, an error_class derived from message text with a permissive pattern, and a metric label that includes a user or conversation ID. The last also has a cost problem apart from privacy, because each distinct label value creates a new time series; see metric cardinality. A review gate on changes to the exporter schema, with a test that feeds canary values through and fails if any appear, turns these from latent risks into build failures.

Compared with pattern redaction, the allowlist fails in the opposite direction. A redactor passes through anything it does not recognize; an allowlist drops anything it was not told about. The price is diagnostic power: the exported record cannot tell you what the customer wrote. The next section measures how much of the operational picture survives.

Diagnosing quality and operations without content

Most operational questions about an AI system can be framed as questions about metadata. The table maps common symptoms to the fields that answer them.

Symptom	Field or span that shows it	Reading
Slow responses	Duration of model spans, tool spans, retrieval span; time to first token	Which stage owns the extra time; whether it is one model or one tool
Cut-off answers	Finish reason length; output tokens at the configured maximum	The output limit is too low or the prompt is steering toward long output
Cost jump	Input and output token counts per template version and model	A template change, an added tool schema, or retrieval returning more text
Ungrounded answers	Retrieved count zero; top retrieval score low	The retriever found nothing relevant, so the model answered from its training data
Tool trouble	Tool span status, error class, duration, retry count	A failing or timing-out dependency; a rise in retries
Behavior shift after a release	Template and model version attributes joined with all of the above	Whether the shift lines up with a deploy

The experiment injects known faults into 20,000 synthetic requests, using only the metadata fields from the exporter, and applies a small rule-based diagnoser: tool timeout first, then truncation, then empty retrieval, then low retrieval score, then a latency spike on the larger model. The seven causes are healthy traffic (70 percent) and six faults at 4 to 6 percent each. One of them, wrong_answer_healthy_looking, is a request where the model gave a wrong answer while every measured quantity is drawn from the healthy distribution. It is constructed that way on purpose to isolate the question of what content-free telemetry can see. Real wrong answers are sometimes correlated with metadata (they tend to follow poor retrieval, for example), so this is the hard case rather than the average case.

diagnosis from allowlisted fields only (rows: injected cause, columns: assigned diagnosis)
injected cause                     n         tool_timeout            truncated      retrieval_empty  retrieval_low_score        provider_slow      healthy_looking
healthy                        13885                    0                    0                    0                    0                   12                13873
retrieval_empty                 1167                    0                    0                 1167                    0                    0                    0
retrieval_low_score             1248                    0                    0                    0                 1248                    0                    0
tool_timeout                    1080                 1080                    0                    0                    0                    0                    0
truncated                        825                    0                  825                    0                    0                    0                    0
provider_slow                    803                    0                    0                    0                    0                  784                   19
wrong_answer_healthy_looking     992                    0                    0                    0                    0                    0                  992
latency_ms   healthy p50/p95 = (989, 1630)   wrong-answer p50/p95 = (989, 1606)
in_tokens    healthy p50/p95 = (1398, 2390)   wrong-answer p50/p95 = (1345, 2364)
out_tokens   healthy p50/p95 = (220, 383)   wrong-answer p50/p95 = (222, 385)

Every injected operational fault is assigned the right label, with two kinds of error at the edges: 12 healthy requests were flagged as slow-provider because the latency tail on the large model crossed the threshold, and 19 slow-provider requests were not caught. The perfect diagnosis is a property of how the faults were injected (each one writes a distinct, clean signature into one field), so it says the metadata fields are sufficient for these failure shapes, not that real faults will be this clean. The informative row is the last one. All 992 wrong-answer requests were labeled healthy-looking, and the three summary lines show why: their latency and token distributions are indistinguishable from healthy traffic (median latency 989 ms for both). The gap in median input tokens (1,345 against 1,398) is within what random variation produces at that sample size, since the generator draws both groups from the same distribution.

Content-free telemetry therefore covers the failures that leave a footprint in structure and timing, and it cannot see a fluent, well-formed, wrong answer. For that, three mechanisms put quality into the telemetry path without leaking content:

• Scores computed where the content already is. An evaluator that runs in a controlled environment (the same one used for release gates) reads a sample of requests and writes back a number per request: groundedness, correctness against a reference, policy compliance. Dashboards and alerts then work on those numbers, with the template and model version as labels. The sample size and the quality of the scorer limit what the numbers mean, so report them alongside the scores.
• User and downstream signals. A thumbs-down, a retry, an escalation to a human, or a refund reversal is a boolean fact about the request. It is metadata, and it often correlates with quality.
• Synthetic probes. A scheduled job sends known prompts through the production path and compares answers to expectations. Because the traffic is yours, probe traces can be recorded at level 3 with full content, which gives engineers a place to see exactly what the system did without touching customer data.

Sampling: how much to keep, and how it interacts with privacy

Keeping every trace is expensive, and keeping every trace’s content multiplies the exposure. OpenTelemetry describes two sampling styles. Head sampling decides at the start of a trace, usually from the trace ID and a fixed percentage, which is cheap and stateless but cannot know yet whether the request will fail. Tail sampling decides after the spans of a trace have arrived and can keep traces by outcome: errors, high latency, specific attributes. Its documentation lists the costs: tail sampling needs stateful components that handle large data volumes, the logic grows more complicated as systems evolve, and implementations are often vendor-specific. The two can be combined.

Using the same 20,000 requests, 5,123 had an operational fault. A 5 percent head sample and a simple tail rule (keep every faulty trace, keep 5 percent of the rest) compare as follows:

requests: 20000, requests with an operational fault: 5123
5% head sampling : kept 1037 traces, 280 of the faulty ones
tail rule        : kept 5825 traces, 5123 of the faulty ones

The head sample kept 1,037 traces and only 280 of the faulty ones, about 5 percent of them. The tail rule kept all 5,123 faulty traces plus a 5 percent sample of the rest. For metadata, that is the better trade, because the rare failures are what you will search for. For content the privacy considerations point the other way: content recorded for the failing requests is content about the people who had a bad experience, and retaining it by default, without a purpose check, is exactly what a data-minimization policy is meant to prevent. A reasonable pattern is to keep metadata for the failures and keep content only when a defined trigger fires (an explicit incident, a user-reported problem, a tenant that opted in to debug capture), with the trigger logged. The sampling rates in the experiment are arbitrary choices; the right rates depend on volume, storage cost, and how rare the failures you care about are.

Controlling who reads telemetry

A telemetry backend is a database of everything the system did, usually with a broader audience than the production data stores it summarizes. The access model should be designed, not inherited from a default that every engineer can read every trace. Separate by data level and by purpose:

• Operational access (on-call and service owners) covers level 0 and 1 data: traces and metrics with no content, plus scores. This is the default for engineering.
• Content access covers levels 2 and 3. It is granted for a stated purpose, to a named person or role, for a limited time, and every read is recorded in an audit log that records who read which content and why. The audit log is itself sensitive, because it reveals which customers were looked at, and needs its own access control.
• Tenant scoping. On a shared platform, the people operating the platform and the people owning one tenant’s application should not both see all tenants’ content. Tag telemetry with the tenant, and restrict content queries by tenant. A later post in this module returns to tenant isolation for the data plane.
• Export and vendor paths. Dashboards, notebooks, support tickets, and third-party backends are copies. A rule that content never leaves the restricted store is only real if the export paths are inventoried and the content store is not reachable from them.

Retention should follow the level, and each content-bearing store needs a way to delete by subject. The sketch below is an illustrative policy, not a recommendation of specific durations; the right values come from your legal and contractual obligations and from how long your incident reviews need the data.

telemetry_policy:
  level_0_metadata:
    readers: [oncall, service-owner]
    retention_days: 90          # example value
  level_1_scores:
    readers: [oncall, service-owner, evaluation-team]
    retention_days: 180         # example value
  level_2_redacted_content:
    readers: [service-owner]
    purpose_required: true
    retention_days: 14          # example value
    sampling: {rate: 0.01, triggers: [incident, user_report, tenant_opt_in]}
  level_3_raw_content:
    readers: [named-reviewers]
    purpose_required: true
    access_logged: true
    retention_days: 7           # example value
    delete_by: [user_key, tenant]
    stored_in: restricted-content-store

Deletion is easier when content is kept out of the trace backend. Trace stores are typically optimized for append and time-ordered query and are awkward at deleting one person’s records; a content store keyed by pseudonym or request ID can support it directly. This is a further argument for the reference-based pattern, and it is a concrete case of the data privacy principle that collection should match purpose and that retention is part of the design.

Verifying the pipeline with canaries

Everything above can be tested the way the experiments were, with fabricated values whose locations are known. A canary test sends synthetic requests through the real production path (instrumentation, collector, backend) whose prompts, retrieved documents, tool arguments, and error paths contain unique fabricated values, such as canary-7f3a@example.com. After a delay, the test searches every telemetry sink (trace store, log store, metric labels, dead-letter queues, spools, exports) for those strings and fails if any appears at a level where it should not. The test checks the actual configuration, including the parts nobody remembers, such as a debug exporter left on or an HTTP instrumentation that records bodies. Run it on every change to instrumentation, collector configuration, or the exporter schema.

A short acceptance list for a pipeline that follows this design:

1. With default settings, no content attribute from the semantic conventions appears on any production span.
2. Every field in the exported record has a type or enumeration constraint, and a test fails when a new unconstrained string field is added.
3. Canary values planted in prompts, retrieved text, tool arguments, and exception messages are absent from level 0 and level 1 stores.
4. Content reads require a stated purpose and are recorded; the audit log is queryable by reader and by tenant.
5. Each content store has a retention limit and a deletion path keyed by user or tenant.
6. Residual redaction misses are measured on synthetic data after each change to patterns or templates, and the result is compared with the previous run.

With telemetry in place that can be shared widely without exposing customers, the next question is what to promise and measure with it: latency, availability, and the harder problem of a service-level indicator for whether the assistant actually completed the task. That is the subject of AI Service Objectives and Reliability Engineering.

References

• OpenTelemetry, Semantic conventions for generative AI (repository and documentation)
• OpenTelemetry documentation, Handling sensitive data
• OpenTelemetry documentation, Sampling
• OpenTelemetry Collector contrib, Redaction processor
• OWASP GenAI Security Project, LLM02:2025 Sensitive Information Disclosure