Canary Token

A canary token is a unique, meaningless string planted somewhere it should never be read or reproduced, so that its unexpected appearance elsewhere signals that data from the place it was planted may have leaked or been used without authorization. The name comes from the canary in the coal mine: the token carries no value of its own, and its only job is to signal a condition. Thinkst's Canarytokens popularized the idea for files, URLs and credentials that raise an alert when opened. In LLM applications the same idea is used as a test instrument: plant a marker, run the system, and search its outputs and storage for the marker.

Why a random marker works

A leak is hard to detect when the leaked text looks like ordinary text. A budget paragraph that appears in the wrong user's answer may be indistinguishable from the model's own words. A marker removes that ambiguity. If the string cnr-f830be6544137ba2 appears in a reply, it almost certainly came from the document that contained it, because the string has no other legitimate source (the exception is a marker that someone has copied; see Limits). Detection becomes an exact substring search, with no similarity threshold and almost no false positives.

How few false positives depends on how guessable the marker is. A marker built from 16 hexadecimal characters carries 64 bits, so the chance that unrelated text contains a specific marker is about \( 2^{-64} \approx 5.4\times10^{-20} \) per position. Markers should therefore be random or keyed (for example a truncated HMAC of the location name) and not a readable word such as "secret", which real text can contain. One marker per location is what turns a hit into a lead: the marker points to the source it was planted in, and the investigation then checks whether that source was exposed or misused, and by which path.

A traced example

The program below is a hypothetical leak test. Each document gets its own marker, and a model reply is scanned for all of them.

import hmac, hashlib
KEY = b"demo-key"
def canary(tag):
    return "cnr-" + hmac.new(KEY, tag.encode(), hashlib.sha256).hexdigest()[:16]

docs = {"hr_policy": "Leave policy text. ", "finance_wiki": "Budget notes. "}
marks = {k: canary(k) for k in docs}
corpus = {k: v + marks[k] for k, v in docs.items()}
reply = "Sure. The budget notes say: " + marks["finance_wiki"] + " and nothing else."
print([k for k, m in marks.items() if m in reply])

Running it prints ['finance_wiki']: the reply contains the finance marker and not the HR marker, which points to the finance document as the source. Whether that is a violation depends on whether the requester was allowed to read that document and on how the marker reached the reply. The same scan applies to any sink: logs, traces, caches, evaluation sets, vendor-side retention, or the text a tool sends to an external service.

Uses in LLM applications

• Access-control tests. Plant a marker in a document the test user is not allowed to read, ask questions that would surface it, and assert that no response, citation, rewritten query or trace contains it. This catches leaks that happen between components, when each component passes its own test.
• Retention and deletion tests. Put a marker in a probe subject's data, request deletion, and scan every store on later days. A store that still holds the marker after the deletion deadline needs explaining: the data inventory may have missed it, the deletion may have failed, or the copy may be retention that is allowed. First establish which copies must be deleted and by when, then sort the remaining hits into those three cases. A scan that reads the stores directly can also find copies nobody registered.
• Prompt and context leakage. A marker inside a system prompt or a restricted instruction shows whether an attacker's request made the model repeat it.
• Memorization. Carlini et al. insert random canary sequences into training data and measure how much more likely the trained model makes them than other random sequences, to quantify unintended memorization.

Limits

A canary only detects verbatim reproduction. A model can leak the substance of a document, such as a paraphrase or a number derived from it, without ever emitting the marker, so a clean scan does not prove the absence of leakage. A canary also detects only the paths that were exercised, and a marker that is stripped by normalization or truncation before the sink goes unseen. If a marker becomes known to an attacker, the attacker can copy it and create a false alarm. Keep the generation key outside the protected data; the marker itself must be planted in that data for the test to detect it.

Related terms

A canary token differs from a honeypot, which is a decoy system meant to be attacked, and from a watermark, which is embedded in generated content to provide evidence of origin under a detector's stated conditions. It also differs from a stack canary, a memory-safety guard unrelated to data leakage.

References

• Carlini et al., The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks