Cross-Tenant Leakage Test

A cross-tenant leakage test checks tenant isolation by experiment. Each tenant gets a canary, a value that exists only in that tenant’s data. A test then acts as another tenant through every channel that shares state and asserts that the canary never appears in what the attacker receives. Reading the code and the access rules shows what the designers intended; the test shows what the running system does. The test looks for a planted value in the wrong place, not for sensitive content in general, so it works with synthetic tenants and needs no real customer data.

The test matrix

With \( n \) tenants there are \( n(n-1) \) ordered (attacker, victim) pairs, because leakage from A to B does not imply leakage from B to A: one tenant’s cached answer may be reachable while the reverse is not. For each pair and each shared channel, such as the retrieval index, a response cache, a tool service, logs, and an export or batch job, the test seeds the victim’s canary, performs the attacker’s request, and searches the response. Every new shared store or code path adds a column to the matrix. Two companion checks keep the result meaningful. The positive check confirms that each tenant can still retrieve its own canary through the same path. A mutation check removes one isolation control at a time and confirms that some test fails; a control whose removal changes no result is not covered by the suite.

Designing the canary and the matching rule

Two weak points make a test pass or fail for the wrong reason. A canary that is a plain word, such as the tenant’s name, also occurs in innocent text and raises false alarms. A canary found only by exact substring search is missed when the model lowercases, spaces, or encodes it. The script uses a random canary and compares a naive substring check with one that normalizes case and punctuation and decodes long base64-looking tokens.

import base64, re, secrets
def make_canary(tenant): return f"CNRY-{tenant}-{secrets.token_hex(6)}"
def norm(s): return re.sub(r"[^a-z0-9]", "", s.lower())

victim = make_canary("birch")
leaks = {
 "verbatim":      f"The code is {victim}.",
 "lowercased":    f"the code is {victim.lower()}",
 "spaced/hyphen": "the code is " + " ".join(victim),
 "base64":        "the code is " + base64.b64encode(victim.encode()).decode(),
 "no leak":       "I could not find that code.",
}
def naive(text):  return victim in text
def robust(text):
    t = norm(text); c = norm(victim)
    if c in t: return True
    for tok in re.findall(r"[A-Za-z0-9+/=]{16,}", text):
        try:
            decoded = base64.b64decode(tok + "=" * (-len(tok) % 4), validate=False).decode("latin1")
        except (ValueError, base64.binascii.Error):
            continue
        if c in norm(decoded): return True
    return False

print(f"{'case':14s} naive robust")
for k, v in leaks.items(): print(f"{k:14s} {str(naive(v)):5s} {robust(v)}")
print("word canary 'birch' in unrelated text:", "birch" in "A birch tree costs $40 to plant")
print("silent system passes leak test:", not robust(""), "| own-data check passes:", victim in "")

case           naive robust
verbatim       True  True
lowercased     False True
spaced/hyphen  False True
base64         False True
no leak        False False
word canary 'birch' in unrelated text: True
silent system passes leak test: True | own-data check passes: False

The naive check catches only the verbatim leak and misses the three transformed ones, which the normalizing check catches without flagging the clean answer. The word canary matches an unrelated sentence, which is why the canary carries a fixed marker and 48 random bits. The last line shows the vacuous pass: a system that returns nothing never leaks, so the leak test passes while the positive check fails. Always run both. The run used Python 3.12.14 on macOS and tests the matching rule only, not a real platform. A matching rule cannot cover every encoding a model might produce, such as translation or paraphrase of the surrounding text. Make the canary a short, random token that a model copies rather than rewrites, and treat the test as evidence of what it covers.

Running it continuously

Run the matrix in continuous integration against the real code paths, so a configuration change, a new endpoint, or a library upgrade fails a test before it leaks customer data. In production, keep dedicated canary tenants that no customer uses, seeded with their own canaries, and run the same attacker-side requests on a schedule. Keep canary tenants out of billing and analytics, and make their alerts route to security. OWASP’s guidance on vector and embedding weaknesses names cross-context leakage between groups sharing a vector database as a risk and recommends permission-aware stores with strict logical partitioning; the test is how a team confirms that the partitioning holds on the paths it actually runs.

References

• OWASP GenAI Security Project, LLM08:2025 Vector and Embedding Weaknesses