Case: Enterprise Knowledge Search

Consider an employee who asks the company assistant what the nightly hotel limit is. Three things can go wrong that no error rate will show. The answer can come from a board-only travel policy the employee was never allowed to open. It can quote last quarter’s limit because the edit that changed it is still on its way to the index. Or it can cite a document that was deleted on Monday. In each case the service is healthy, the search is fast, and the answer reads well. What failed is the relationship between the index and the systems it copies, and that relationship has to be designed, measured, and recovered like any other production dependency.

This article builds one such system end to end and runs it. Quillfeather Logistics is a fictional company invented for this article, and so is everything about it: its 32 employees and contractors, its 177 documents in three repositories, its folder permissions, and its policy values. It is not a real case. The knowledge search service is a small program written for this article, about 400 lines of Python plus a simulation harness. Its embedding is a hashed bag of words standing in for a model, its answers are extractive, and time is simulated. Every number below is the output of running that code on synthetic data. None of it describes the performance of a real product or organization.

The article assumes you have read the two data articles on retrieval: Document Pipelines for Retrieval: Parsing, Versions, and Deletion (provenance, chunk keys, the registry, tombstones, ingestion checks) and Preserving Access Controls in Retrieval Data (permission copies, identity propagation, filtering, revocation, leakage tests). It uses the as-of reasoning from Feature Pipelines and Point-in-Time Correctness, the containment and recovery vocabulary from AI Platform Incident Response and Recovery, the release identity ideas from Versioning Models, Data, Prompts, and Evaluation Artifacts, the isolation paths from AI Platform Security and Tenant Isolation, and the experiment design cautions from Measuring an AI Development Tool Rollout. Each is recalled in a sentence where it is used. Online evaluation and controlled rollout get their own treatment here, in the section on releasing a new index, because the case needs them now.

After reading, you should be able to do five things: state the confidentiality, freshness, and deletion requirements of an enterprise search system as invariants and service objectives that a test can falsify; choose among the main designs for carrying permissions into an index and name what each gives up; measure how long a permission change takes to reach retrieval; evaluate citations, abstention, and leakage on a labeled set with honest uncertainty; and recover from a bad index, a bad connector release, and an erasure request without reintroducing revoked access.

The code ran with Python 3.12.14, NumPy 2.2.3, and SQLite 3.53.4 on macOS 26.7.1, using only those and the standard library. No network, model, or GPU is involved, and every run is seeded, so repeating a script reproduces its output exactly.

Requirements, and the invariants they become

Quillfeather keeps policies in three repositories: a wiki, a shared drive, and an HR portal. People belong to teams (engineering, support, finance, HR, legal, operations), and a group called g-all contains the team groups, so reading a document shared with all employees requires resolving nested membership. The finance group in turn contains a smaller group of finance leads. Four contractors belong to none of these and can read only the handbook. Permissions live on folders and are inherited by the nearest ancestor that defines them; a document can override its folder with its own list, carry a deny entry for one person, or be link-shared with everyone in the company. One document sits in a folder that has no access policy at all. Three folders hold material the company treats as restricted: board-level finance, legal, and HR cases. Legal also keeps a hold archive of 96 earlier revisions of popular policies, readable only by legal. Documents can be edited, deleted, and given effective dates, so a revision can be published now and take effect next month.

The business asks sound simple: show people only what they may read, keep the answers current, cite sources, and be able to undo mistakes. To engineer them, each ask needs a statement that a test can fail. Let \( V(u,t) \) be the set of chunks whose source documents the source system lets user \( u \) read at time \( t \), and \( R(u,q,t) \) the chunks used to answer query \( q \). The strict requirement is \( R(u,q,t) \subseteq V(u,t) \). Copies cannot satisfy it exactly, because a permission change at the source needs time to reach the index. The requirement the system can honor is a bounded one:

\[ R(u,q,t) \subseteq \bigcup_{s \in [t – W_c,\, t]} V(u,s) \]

A returned chunk is acceptable if the user could read its document at some moment within the last \( W_c \) seconds, where \( W_c \) is the permitted propagation window for the document’s class \( c \). A violation is a chunk the source has denied for longer than the window. For standard documents the window is a stated service objective. For restricted documents the window is zero, which a design can meet only by asking the source at serving time.

Requirement in business words	Testable statement	Objective chosen for this system
Never show text the user cannot open	No returned chunk is outside the bounded visible set; no side channel (titles, counts, suggestions, caches) carries unreadable content	Restricted class: zero violations, window 0. Standard class: window 300 s for ACL narrowing and deletion; zero violations beyond it
People who lose access lose it quickly	Group removal reaches retrieval within the membership cache lifetime	60 s
Edits show up soon and old text goes away	A new version is retrievable within a stated time; a superseded version is never served after its replacement is published	99% of edits retrievable within 5 minutes; no superseded version served after the pointer flip
Answers say where they came from, and say nothing when they cannot	The cited chunk supports the answer; the system abstains when no authorized chunk scores above a calibrated floor	Citation correctness and abstention measured on a labeled set with intervals
Mistakes can be undone	A retained index can be reactivated without reviving revoked access; a deletion request reaches every copy	Rollback and erasure drills pass with zero violations

The objectives are choices, not discoveries. A five-minute window for standard documents means the company accepts that for up to five minutes after a folder is locked down, a person who was entitled a moment ago can still find its contents through search, in exchange for not calling the source system on every query. Sizing and defending that number is part of the design, and the measurements later show what the pipeline actually achieves against it. The method of defining indicators and objectives from user-visible outcomes is the one in AI Service Objectives and Reliability Engineering.

Design decisions and what each one gives up

The decisions below are the ones the rest of the article implements and tests. Each row names the chosen design, the main alternative, and the price of the choice.

Decision	Chosen	Alternative	What the choice gives up
Representing group grants	Group identifiers on each chunk; resolve the user’s group closure at query time with a 60 s cache	Expand group members onto every chunk at index time	A runtime dependency on the directory and up to 60 s of membership lag. In return, a membership change rewrites no chunks
Where the permission predicate runs	Inside the search, before the top-k is chosen	Search the whole index, then drop disallowed chunks	Needs an index that supports filtered vector search, and a very selective filter can make the search slower
Restricted documents	Ask the source before text reaches the answer (recheck)	Trust the index copy everywhere	An extra source call per request that touches restricted chunks, and a dependency on source availability. Fails closed
ACL changes	A metadata-only fast lane that never re-embeds and runs ahead of content changes	Treat an ACL change as a content change	Two ingestion paths to keep consistent
What is current	A registry decides which version of which document is servable; chunks only carry a version	Rely on deleting old chunks when a new version arrives	One registry lookup per query. In return, a failed cleanup cannot expose an old version
Noticing changes	Webhook events plus a polling safety net every 120 s	Webhooks only	Polling load on source APIs
Retained indexes	Keep previous index versions and an append-only revocation log; replay it on activation	Roll back by repointing an alias	Log storage and replay code. In return, a rollback cannot resurrect revoked access
Answer cache	Key by principal set and a change epoch	Key by normalized query	A lower hit rate
No good evidence	Abstain below a score floor calibrated on held-out questions	Always answer with the best chunk	Some answerable questions go unanswered

The first row deserves a number. In Preserving Access Controls in Retrieval Data the expansion cost was computed for a 5,000-person group over 200,000 chunks. Quillfeather is far smaller, so expansion would be affordable here. Group identifiers are chosen anyway because membership is the quantity that changes most often and because removals are the changes that must not lag. At Quillfeather’s size the choice costs little. At the size where it matters, it is the only workable one, and the system should not be built differently at each size.

Implementation

The registry, and what a chunk carries

The registry has one row per document and one table for revocations. A chunk carries a document identifier, a version, an ordinal, the allow and deny principals copied from the source, a classification, effective dates, and a publication time. The registry holds the current version pointer, a pending version, a content hash, a status (current, quarantined, or deleted), the version of the ACL copy, and when it was fetched. The revocation log records every narrowing, quarantine, and deletion with a sequence number; it is what a retained index must catch up on when reactivated. The audit table records which chunk identifiers each request returned, never the text.

CREATE TABLE doc_registry(doc_id TEXT PRIMARY KEY, source TEXT, version_id INT, pending_version INT, content_hash TEXT,
  status TEXT, acl_version INT, acl_fetched_at REAL, recorded_at REAL, deleted_at REAL, chunker_v TEXT);
CREATE TABLE revocation_log(seq INTEGER PRIMARY KEY AUTOINCREMENT, t REAL, kind TEXT, doc_id TEXT);
CREATE TABLE audit_log(req INTEGER PRIMARY KEY AUTOINCREMENT, t REAL, user TEXT, principals TEXT, index_id TEXT,
  docs TEXT, chunks TEXT, acl_versions TEXT, abstained INT);

The corpus is generated from 42 policy topics, each with a stated limit such as The nightly hotel limit for domestic travel is 180 dollars., plus regional variants of 12 topics with different values, the legal-hold archive, and three special documents (a link-shared page, a page with a deny entry, and the page without an ACL). That yields 78 questions with known answers. Because documents and questions come from the same templates, retrieval is easier than it would be on real text. The experiments therefore test mechanisms, not retrieval quality, and the evaluation section says so again where it matters.

Ingestion: content, then permission, then publication

Ingestion follows the three phases from Document Pipelines for Retrieval: Parsing, Versions, and Deletion: write the new version’s chunks as pending, flip the registry pointer, and remove the superseded chunks. Two details come from Preserving Access Controls in Retrieval Data. The ACL is fetched when the content is ingested, so new text can never be published with an older permission copy. And a document whose ACL cannot be determined is quarantined, not treated as public.

def sync_content(self, doc_id, t, index=None, force=False):
    """Classify (new/changed/unchanged/deleted), validate, write pending chunks. Returns an outcome string."""
    index = index or self.live
    reg, rec = self.reg(doc_id), self.src.rec(doc_id, t)
    if reg and reg["status"] == "deleted": return "refused-tombstoned"
    if rec is None or rec["deleted"]: return self.tombstone(doc_id, t)
    chash = sha(rec["text"] + f"|{rec['eff_from']}|{rec['eff_to']}")      # text and effective dates both shape the chunks
    if reg and reg["content_hash"] == chash and reg["status"] != "quarantined" and not force: return "unchanged"
    if reg and reg["pending_version"] == rec["version"] and not force: return "pending"
    acl = self.fetch_acl(doc_id, t) if (self.cfg.acl_with_content or not reg) else self._stored_acl(doc_id, index)
    if acl is None or not parse(rec["text"]):
        self.db.execute("INSERT INTO doc_registry(doc_id,source,status,acl_version) VALUES(?,?,?,0) ON CONFLICT(doc_id) DO UPDATE SET status='quarantined'",
                        (doc_id, rec["folder"].split("/")[0], "quarantined"))
        self.epoch += 1
        return "quarantined"
    index.add(self.make_chunks(doc_id, rec, acl, t, rec["version"]))
    if reg is None:
        self.db.execute("INSERT INTO doc_registry VALUES(?,?,?,?,?,?,?,?,?,?,?)",
                        (doc_id, rec["folder"].split("/")[0], None, rec["version"], chash, "current", 1, t, None, None, CHUNKER_V))
    else:
        self.db.execute("UPDATE doc_registry SET pending_version=?, content_hash=?, status='current', acl_fetched_at=? WHERE doc_id=?",
                        (rec["version"], chash, t, doc_id))
    return "new" if reg is None else "changed"

The content hash covers the text together with the effective dates. An earlier version of this function hashed only the text, and ending a revision’s validity period, which changes no text, was classified as unchanged and never reached the index. The metadata that shapes chunks belongs in the hash.

def publish(self, t, index=None):
    """Phase 2 and 3: flip the registry pointer for pending versions, then remove superseded chunks."""
    index = index or self.live; n = 0
    for (doc_id, pend) in self.db.execute("SELECT doc_id,pending_version FROM doc_registry WHERE pending_version IS NOT NULL").fetchall():
        for c in index.chunks:
            if c["doc_id"] == doc_id and c["version"] == pend: c["live"], c["recorded_at"] = True, t
        self.db.execute("UPDATE doc_registry SET version_id=?, pending_version=NULL, recorded_at=? WHERE doc_id=?", (pend, t, doc_id))
        if not self.cfg.fail_cleanup: index.drop(lambda c: c["doc_id"] == doc_id and c["version"] != pend)
        n += 1
    if n: self.epoch += 1
    return n

The ACL fast lane copies the source’s current ACL onto the document’s chunks without touching vectors. It also appends a revocation log entry when anything changed, and it quarantines a document whose ACL has become undeterminable.

def sync_acl(self, doc_id, t, index=None):
    """Metadata-only update: copy the source's current ACL onto the document's chunks."""
    index = index or self.live
    reg, rec = self.reg(doc_id), self.src.rec(doc_id, t)
    if reg is None or reg["status"] == "deleted": return "skip"
    if rec is None or rec["deleted"]: return self.tombstone(doc_id, t)
    if reg["version_id"] is None and reg["pending_version"] is None: return self.sync_content(doc_id, t, index)
    acl = self.fetch_acl(doc_id, t)
    if acl is None:
        self.db.execute("UPDATE doc_registry SET status='quarantined' WHERE doc_id=?", (doc_id,)); self.log_rev(t, "acl", doc_id)
        return "quarantined"
    cs =  == doc_id]
    changed = any((c["allow"], c["deny"]) != acl for c in cs) or reg["status"] == "quarantined"
    if changed:
        for c in cs: c["allow"], c["deny"] = acl
        self.db.execute("UPDATE doc_registry SET acl_version=acl_version+1, acl_fetched_at=?, status='current' WHERE doc_id=?", (t, doc_id))
        self.log_rev(t, "acl", doc_id)
    else:
        self.db.execute("UPDATE doc_registry SET acl_fetched_at=? WHERE doc_id=?", (t, doc_id))
    return "acl-updated" if changed else "acl-same"

The query path

The query path resolves the user’s principals, applies every predicate inside the candidate loop, and only then builds an answer. The predicates are: the chunk is published, its version is the registry’s current version for a non-deleted document, its effective dates cover the as-of day, and its allow set intersects the user’s principals without hitting a deny. An unmapped identity resolves to an empty principal set, which matches nothing. Chunks of restricted documents then pass through the source recheck. The answer is the best-matching sentence of the top chunk, with its chunk identifier as the citation, or no answer when the top score is below the floor. The prefilter=False branch exists only so the experiments can show the alternative.

def search(self, user, query, t, k=5, as_of=None, depth=None, tau=0.40):
    P = self.principals(user, t)
    day = int(t // 86400) if as_of is None else as_of
    reg = self.reg_snapshot()
    sc = self.live.scores(embed(query))
    order = np.argsort(-sc)
    if not self.cfg.prefilter: order = order[: depth or k]
    def current(c):
        if not self.cfg.registry_filter: return True
        r = reg.get(c["doc_id"]); return r is not None and r[1] == "current" and r[0] == c["version"]
    def dated(c):
        return (c["eff_from"] is None or c["eff_from"] <= day) and (c["eff_to"] is None or day < c["eff_to"])
    def permitted(c): return bool(P) and bool(c["allow"] & P) and not (c["deny"] & P)
    hits, related = [], []
    for i in order:
        c = self.live.chunks[i]
        if not (c["recorded_at"] is not None and current(c) and dated(c)): continue
        if self.cfg.prefilter and not permitted(c): continue
        hits.append((float(sc[i]), c))
        if self.cfg.prefilter and len(hits) == k: break
    if not self.cfg.prefilter:
        if not self.cfg.related_from_permitted: related = list(dict.fromkeys((c["title"], c["doc_id"]) for _, c in hits[:3]))   # drawn before the filter
        hits = [(s, c) for s, c in hits if permitted(c)][:k]
    if self.cfg.recheck_restricted or self.cfg.recheck_all:
        hits = [(s, c) for s, c in hits if (c["cls"] != "restricted" and not self.cfg.recheck_all) or self.oracle.can_read(user, c["doc_id"], t)]
    if self.cfg.prefilter or self.cfg.related_from_permitted: related = list(dict.fromkeys((c["title"], c["doc_id"]) for _, c in hits[:3]))
    res = dict(hits=hits, related=related, answer=None)
    if hits and hits[0][0] >= tau:                    # answer: best sentence of the top chunk, with a citation
        s, c = hits[0]; qt = set(toks(query))
        best = max(re.split(r"(?<=[.!?])\s+", c["text"]), key=lambda x: len(qt & set(toks(x))))
        res["answer"] = dict(text=best, cite=c["cid"], doc=c["doc_id"], version=c["version"], score=s)
    self._audit(user, t, P, res)
    return res

def principals(self, user, t):
    hit = self.resolver.get(user)
    if hit and hit[0] > t: return hit[1]
    p = self.dir.expand(user, t) if self.dir.known(user) else frozenset()    # unmapped identity: no principals
    self.resolver[user] = (t + self.cfg.resolver_ttl, p)
    return p

The simulation harness

Permission propagation is a property of timing, so the harness runs the real service functions on a simulated clock. A source change is announced by a webhook that arrives after a lognormally distributed delay (median 15 seconds), and 3% of webhooks never arrive. A poller lists the change log every 120 seconds and enqueues whatever it has not seen. Events then go to one of two single-threaded lanes: the ACL lane, which costs 0.2 seconds per document and runs ahead of content, and the content lane, at 3 seconds per document. A content change becomes searchable at the next 30-second publish tick. A purge job runs every 15 minutes. These parameters are assumptions chosen to be plausible and are stated so that you can change them; the measurements are about how the design behaves under them, not about any real connector.

def _enqueue(self, seq):
    if seq in self.enq: return
    self.enq.add(seq)
    _, _, kind, key = self.src.log[seq - 1]
    lane = "content" if kind == "doc" else "acl"
    k = len(self.src.children(key, self.now)) if kind == "folder_acl" else 1
    cost = (self.cost_content if lane == "content" else self.cost_acl) * max(1, k)
    t = max(self.now, self.lane_free[lane]) + cost
    self.lane_free[lane] = t; self.at(t, self._apply, seq)

def _apply(self, seq):
    _, _, kind, key = self.src.log[seq - 1]; s = self.svc
    if kind == "doc":
        s.sync_content(key, self.now)
        nxt = math.ceil(self.now / self.pub) * self.pub
        if nxt not in self.pub_at: self.pub_at.add(nxt); self.at(nxt, self._publish, nxt)
    elif kind == "doc_acl": s.sync_acl(key, self.now)
    elif kind == "doc_delete": s.sync_acl(key, self.now)
    elif kind == "folder_acl":
        for d in self.src.children(key, self.now): s.sync_acl(d, self.now)
    self.applied[seq] = self.now

To measure exposure, the harness probes the real query path: a user who is about to lose access keeps asking a question whose answer is in the document, every 10 seconds, and the exposure is the time until the document stops being returned.

Time in the index: effective dates and record time

Feature Pipelines and Point-in-Time Correctness separates when a fact holds in the world from when the system learned it. The same two clocks appear in retrieval. A policy revision has an effective period, and the pipeline records when it became servable. Quillfeather asks the first question at query time: a query is evaluated for an as-of day, which defaults to today, and a chunk is eligible only if its effective period covers that day. The second clock is the publication time, which stays null until the pointer flips, so pending chunks are invisible.

The demonstration publishes a revision of the expense policy today with an effective date 30 days out, and ends the old revision’s validity on the same day. A search for today still finds the old limit. A search as of day 31 finds the new one, and the old revision no longer appears.

== effective dating: a revision published now, effective in 30 days
  as of today    [('drive:expense-reports', '30 days'), ('drive:expense-reports-americas', '26 days')]
  as of day+31   [('drive:expense-reports-rev2', '20 days'), ('drive:expense-reports-americas', '26 days')]

Two limits are worth stating. The model supports effective-dated questions such as planning questions about next month’s policy, but it does not reconstruct what the system would have answered on an earlier date, because superseded chunks are deleted after the flip. Reproducing an old answer for an audit requires a retained index from that time, which is the same retention that rollback needs. And the supersession is declared by the source through an end date; the index does not infer that a new page replaces an old one.

Measuring permission propagation and freshness

An objective that is not measured is a hope. Three quantities matter, and the harness measures each by probing the query path, not by reading the pipeline’s own timestamps, because the pipeline’s timestamps would agree with its own copy.

ACL narrowing and deletion

In each cycle a user who can read a document sees it, the document’s ACL is narrowed to legal only, and the user keeps asking until the document disappears. The ACL is then restored and given time to propagate. Each configuration ran several hundred cycles on documents and users drawn at random, at random phases relative to the polling timer.

configuration                                   n    p50    p95    p99     max  <=300s  never
standard docs, webhook only                   340   20.0   60.0   90.0   180.0   96.2%     13  skipped=60
standard docs, webhook + poll 120s            400   20.0   50.0   80.0   100.0  100.0%      0  skipped=0
restricted docs, poll 120s, no recheck        400   20.0   50.0   90.0   120.0  100.0%      0  skipped=0
restricted docs, poll 120s, source recheck    400    0.0    0.0    0.0     0.0  100.0%      0  skipped=0

Read the rows in order. With webhooks only, a typical narrowing is effective within about 20 seconds and 95% within 60 seconds, which looks fine until the tail: 13 of 340 completed cycles, 3.8%, never became effective during the 30-minute probe because their webhook was lost. The run also had to skip 60 cycles, because a lost restore event had left the document hidden from a user who should have regained access. Losing events harms both directions: access that was removed stays in effect, and access that was restored stays missing. Adding the 120-second poll removes both problems: every cycle completes, the maximum is 100 seconds, and 100% are inside the 300-second objective. The poll turns an unbounded tail into a bounded one, and its interval is the knob that sets the bound. The standard to meet is the one in the glossary entry on access revocation: a removal has to reach every path that actually authorizes requests, which here includes the polling path as well as the event path.

The third row repeats the measurement on restricted HR documents with the index copy only, and the fourth adds the source recheck. With the recheck, the exposure is zero in every cycle, because the stale index copy no longer decides. The price is a source call whenever a restricted chunk is a candidate for the answer. Restricted material is 318 of the 524 chunks in this corpus, mostly because of the hold archive, so the recheck is a real cost here and should be measured for latency before a rollout.

Group membership

Membership is resolved at query time with a cache, so the bound is the cache lifetime. The harness removes a finance user from the finance group at a random moment and probes every 10 seconds.

membership removal: seconds until the removed user stops getting the document
  resolver ttl     0s  n=200  p50=   0.0  p95=   0.0  max=   0.0
  resolver ttl    60s  n=200  p50=  30.0  p95=  60.0  max=  60.0
  resolver ttl   300s  n=200  p50= 140.0  p95= 280.0  max= 300.0

The measured median is about half the lifetime and the maximum equals it, as expected for a cache that was filled at a random earlier moment. A lifetime of 60 seconds satisfies the 60-second objective; 300 seconds would not. The row with a lifetime of zero is the extreme: no cache, one directory call per query. Choosing the lifetime is a trade between directory load and the exposure window, and the stated objective is what makes the trade decidable. A token that carries group claims would behave like a cache whose lifetime is the token’s, as noted in Preserving Access Controls in Retrieval Data.

Content freshness

The harness edits a document and asks the query path for the new version every 5 seconds.

content freshness: seconds from edit to the new version being retrievable
  poll=None  n=120  p50=40  p95=105  p99=130  max=135  never(30min)=3  old version still served at that moment: 2
  poll=120.0  n=120  p50=40  p95=100  p99=115  max=135  never(30min)=0  old version still served at that moment: 0

Half of the edits were retrievable within 40 seconds, and the slowest completed one took 135 seconds. The floor is the sum of webhook delay, a 3-second ingestion, and a wait for the next 30-second publish tick. Without the poll, 3 of 120 edits were still invisible after 30 minutes, and 2 of those were still serving the old version when the probe stopped. With the poll, none were, so the 5-minute objective holds with margin. This is data freshness measured at the point consumers care about, the query result, in place of the last successful run of a job.

Where filtering happens

The placement of the permission predicate has two effects that are easy to conflate. The first is recall for users who can read little. The second is whether anything derived from unreadable chunks reaches the response. The harness compared three placements on the same corpus. pgvector documents that with approximate indexes the filter is applied after the index scan, so a selective filter returns fewer rows (its example is a condition matching 10% of rows with the default candidate list of 40, which leaves about four matches on average), and the Elasticsearch documentation says a filter outside the knn clause can return fewer than k results even when enough relevant documents exist. The experiment below reproduces the effect in this system.

C. filter placement for narrow-access askers (contractors, handbook questions)
  pre-filter           gold in top 5:  40/40  answered:  40/40  leaked chunks: 0
  post-filter, top 5   gold in top 5:  34/40  answered:  34/40  leaked chunks: 0
  post-filter, top 50  gold in top 5:  40/40  answered:  40/40  leaked chunks: 0

Contractors can read only the 20 handbook documents, while the hold archive holds eight near-identical revisions of each popular policy. With a post-filter over the top 5 candidates, the archive revisions fill the candidate list for 6 of 40 contractor questions, and the gold chunk is never seen. Widening the candidate list to 50 restores recall, which shows that the information was there. A pre-filter needs no tuning and answered all 40. No variant returned a chunk to a user who may not read it.

The second effect is where the placements really differ. The service also shows a short list of related documents. When that list is built from the candidates before the post-filter drops the unreadable ones, the titles of unreadable documents reach the page.

D. 'related documents' suggestions on withheld questions
  post-filter, related from permitted        leaked chunks: 0  queries with a leaked related title: 0/66
  post-filter, related before the filter     leaked chunks: 0  queries with a leaked related title: 66/66
  pre-filter                                 leaked chunks: 0  queries with a leaked related title: 0/66

On the 66 questions whose gold document the asker cannot read, the pre-filter and the post-filter that builds suggestions from permitted candidates leaked no titles. The post-filter that builds them first leaked at least one title in all 66, even though it never returned an unreadable chunk. A post-filter is safe only if nothing is shown, counted, ranked, expanded, or cached from the unfiltered candidates. A pre-filter makes that discipline structural, since the unreadable chunks are never candidates.

Neither placement helps against a stale copy. The next run narrows three documents at the source without letting the pipeline process the events, then counts chunks returned to users the source now denies.

== placement during an ACL propagation window (3 documents narrowed at the source, index copy not yet updated)
  pre-filter on the index copy               chunks returned to users the source now denies: 28
  post-filter (top 50) on the index copy     chunks returned to users the source now denies: 28
  pre-filter + recheck of every class        chunks returned to users the source now denies: 0

Pre-filter and post-filter return the same 28 chunks, because both trust the index’s copy of the permission. Only the recheck, which asks the source, removes them. Filter placement decides recall and side channels. Freshness of the permission copy decides this other class of leak, and it needs the bounded window, the recheck for restricted material, or both.

Evidence: tests, labeled questions, and what they can show

Tests that must fail when a control is removed

AI Platform Security and Tenant Isolation introduced mutation testing for isolation controls: a test suite has not demonstrated a control until deleting the control makes some test fail. Each test below builds a fresh world, performs a specific sequence of changes, and asserts a property. The matrix runs every test on the designed configuration and again with its control switched off.

test                                              designed   control removed  removed control
restricted doc: recheck beats stale index copy        pass              FAIL  recheck_restricted=False
edit after revoke: ACL fetched with content           pass              FAIL  acl_with_content=False
membership removal within resolver TTL                pass              FAIL  resolver_ttl=3600.0
answer cache: other principal                         pass              FAIL  cache_key_principals=False
answer cache: after revocation                        pass              FAIL  cache_key_principals=False
failed cleanup: old version hidden                    pass              FAIL  registry_filter=False
deleted before purge: not served                      pass              FAIL  registry_filter=False
rollback re-applies revocations                       pass              FAIL  replay_on_activate=False
unmapped identity gets nothing                        pass               n/a  -=
undeterminable ACL not indexed                        pass               n/a  -=
tombstoned document is not re-ingested                pass               n/a  -=
future-effective revision hidden until its date       pass               n/a  -=

Eight of the twelve tests were run with their control removed, and each failed as it should: a restricted document served from a stale copy, new text published under an old permission after an edit that followed a narrowing, a removed group member still served after the cache lifetime, one user’s cached answer reaching another, a cached answer surviving a revocation, a superseded version served after a failed cleanup, a deleted document served before the purge ran, and a rolled-back index resurrecting narrowed access. The last four rows are structural behaviors (unknown identity, undeterminable ACL, tombstone replay, future-effective revisions) with no switch to remove, so they pass on the designed system and have not been shown to fail on a broken one.

What the tests establish is narrow. They show that each named failure sequence is handled when the system runs as designed, and that each control is load-bearing for its test. They do not show that no other sequence leaks. The earlier article computed how many random queries are needed to find a rare defect; permission shapes that appear in one document in a thousand need directed tests, which is why these tests are written as scripts of specific changes and not as random sampling.

Labeled questions

The evaluation uses three question sets drawn from the 78 facts. The answerable set has 156 questions, each fact asked by two users who can read the document. The withheld set has 66 questions asked by users who cannot read the gold document, for facts that fewer than half the users can read; the right behavior is to abstain or to answer from something the asker may legitimately read, and never to reveal the gold document. The contractor set has 40 handbook questions. A question is counted as correctly answered only if the cited chunk belongs to the gold document and contains the gold value. Leakage is checked for every returned chunk against the oracle, which reads the source’s histories and shares no state with the index.

The abstention floor was calibrated on a development half of the questions, split by document, and reported on the other half. The floor is the lowest value at which at most 5% of development withheld questions are answered, subject to keeping development citation correctness above 90%.

dev split: answerable 78 withheld 32 | test split: answerable 78 withheld 34
  tau  dev cited-correct  dev withheld answered
 0.18              92.3%                  40.6%
 0.25              92.3%                   6.2%
 0.30              92.3%                   6.2%
 0.35              92.3%                   6.2%
 0.40              92.3%                   3.1%
 0.45              92.3%                   3.1%
 0.50              89.7%                   0.0%
chosen tau (lowest with withheld-answered <= 5% on dev and cited-correct >= 90%): 0.4
  test answerable: answer cites the gold chunk    78/78  100.0%  [95.3%, 100.0%]
  test answerable: answered                       78/78  100.0%  [95.3%, 100.0%]
  test withheld: answered anyway                   0/34    0.0%  [ 0.0%, 10.2%]
  chunks returned to askers who may not read them (all test questions): 0
  contractor questions: gold in top 5 40 / 40  leaked 0

The floor moves the withheld-answered rate from 40.6% to 3.1% on the development half while leaving citation correctness unchanged, so the chosen value is 0.40. On the held-out half, 78 of 78 answerable questions were answered with a correct citation (95% Wilson interval 95.3% to 100%), none of the 34 withheld questions was answered (0% to 10.2%), and no chunk went to a user who may not read it. The intervals are wider than the point estimates suggest. The test half is small, and the 156 answerable questions are only 78 distinct document-question pairs asked twice, so the effective sample is smaller than the interval treats it.

The development half has the more informative result: 92.3% citation correctness, with six misses that are three documents asked twice. They all have the same form. A question such as the advance notice for office visitors does not name a region, and the Asia-Pacific variant of the page outranks the global one. The system cited a real chunk that supports a true statement about the wrong region. The answer is grounded and still wrong for the asker, because grounding measures whether the answer follows its evidence and not whether the evidence is the right evidence. A clarifying question or a region-aware ranking would address it; no amount of permission work would.

The full-set run adds one more case. A legal user asked about the nightly hotel limit for executive travel, a board-only document she cannot read, and received a confident answer from an archived revision of the domestic travel policy: 135 dollars. Permission filtering removed the correct evidence and left a plausible neighbor. This is the quiet cost of correct filtering, and the abstention floor catches most but not all of it. The archive revisions carry no end date, which is a metadata gap in the source, not a retrieval problem; giving superseded revisions an end date would have excluded them from every user’s default query.

Releasing a change to the index

A new chunker, embedding model, or parser changes what every user sees, so it is released like a model. Offline tests are the first gate and cannot decide the release, since they run on a fixed set. Two stages follow: a shadow run, which replays real traffic against the candidate without showing its answers to anyone, and a canary, which serves the candidate to a small share of users. The release rules in Prompt and Model Promotion with Evaluation Gates apply: a hard gate for must-not-fail properties, and a comparison with uncertainty for quality.

Shadow: replay with real principals

The shadow run takes logged requests, each with its real user, and runs them on the live index and on the candidate. The checks are ordered by how they can fail. Leakage against the oracle must be exactly zero for a candidate to continue. Only then is quality compared, with a paired bootstrap on the per-request change in citation correctness and a non-inferiority margin of three percentage points. Replaying the 262 evaluation requests, the harness compared three candidates. The first rebuilds with sentence-level chunks and fresh ACLs. The other two were built from an ACL export taken before three documents were narrowed, which is how a rebuild from yesterday’s export goes wrong.

traffic replayed: 262 requests (156 answerable, 66 withheld, 40 contractor)
candidate                                            leaked  answered   cited-correct d (95% CI)  decision
live index (section chunks)                               0       194                          -  -
candidate 1: sentence chunks, fresh ACLs                  0       195    +0.038 [+0.019, +0.065]  PASS
candidate 2: sentence chunks, ACLs from the earlier export     62       198    +0.061 [+0.034, +0.092]  BLOCK (leak)
candidate 3: section chunks, ACLs from the earlier export     54       197    +0.023 [+0.008, +0.042]  BLOCK (leak)

Candidate 1 passes: no leaks, and an improvement of 3.8 points whose interval, 1.9 to 6.5 points, excludes zero. The two candidates built from the stale export are blocked on leakage, 62 and 54 chunks returned to users the source now denies. Their quality numbers are also higher than the live index, which is the reason a quality comparison alone cannot be the gate: leaked chunks are often relevant, so a candidate that leaks can look better. The leak check must come first and be a hard stop, not one more metric in a score.

Canary: assign by user, and know what the evidence can say

The canary routes a share of users to the candidate by a stable hash of the user identifier, so a user sees one behavior across a conversation and across caches. Assignment is random with respect to everything the pipeline cares about, which is the point made in Measuring an AI Development Tool Rollout: teams or users who volunteer differ from those who do not, and a before-and-after comparison credits the release with whatever else changed. With 32 users, the 5% stage covers 3 of them and the 25% stage covers 9, so a canary at this size can show a failure but cannot show a small improvement. A guardrail needs a number attached to it. After \( n \) requests with no leak, an exact one-sided 95% upper bound on the per-request leak rate is \( 1 – 0.05^{1/n} \), close to \( 3/n \):

  n=   50 requests with zero leaks: upper 95% bound on leak rate per request = 0.0582
  n=  262 requests with zero leaks: upper 95% bound on leak rate per request = 0.0114
  n= 1000 requests with zero leaks: upper 95% bound on leak rate per request = 0.0030
  n= 5000 requests with zero leaks: upper 95% bound on leak rate per request = 0.0006

After 262 requests, a leak rate as high as 1.1% remains consistent with the evidence. Reaching a bound near 0.06% takes about 5,000 requests. The bound assumes independent requests from a similar distribution, and a leak tied to one rare permission shape is not independent, so the canary is evidence for the common paths and the directed tests remain responsible for the rare ones. Staged ramp-up with a hold at each stage until the bound reaches the agreed risk follows from this. Each stage also keeps the previous index retained, so reversing it is a pointer change plus the revocation replay described below.

What user feedback can and cannot tell you

Thumbs-up and thumbs-down signals are cheap and tempting. On the replayed traffic the live index produced the following outcomes:

what user feedback could see on the live index (replayed traffic)
  answered, cites the gold chunk        180
  answered, wrong or other document      14
  abstained                              68

Feedback can attach only to the 194 answered requests. The 68 abstentions produce no rating at all, including the withheld questions where abstaining is the correct behavior and the answerable ones where the user is left without help. Among the answered requests, the 14 wrong ones look like the 180 right ones to a user who cannot read the gold document and so cannot see what was missed. A user cannot downvote a leak they do not know about, and cannot downvote an answer that was withheld from them. Feedback is therefore a lagging signal for complaints and a poor signal for confidentiality or recall. The decision to widen a rollout rests on the leak gate, the labeled set, and the audit sample, with feedback as one input among them.

Serving and recovery

Rebuild from the manifest

A rebuild is the baseline recovery and also a check. The service rebuilds a complete index from the source documents, skipping tombstoned and quarantined ones, and computes a digest over the sorted tuples of document, version, ordinal, content hash, and ACL.

== 1. rebuild from the manifest
live digest 6cccb8c7c9af  rebuilt digest 6cccb8c7c9af  equal: True  chunks: 524
after three unannounced ACL changes, documents whose ACL differs between live and rebuilt: ['hr:sick-leave', 'wiki:backup-retention', 'wiki:laptop-refresh']

In a consistent state, a reconciliation of the index against the source finds nothing: the live digest and the rebuilt digest are equal, which is the reproducibility invariant of Document Pipelines for Retrieval: Parsing, Versions, and Deletion in a concrete form. Three ACL changes whose events were lost make the digests differ, and the comparison names the documents. The rebuild is therefore also a drift detector: it is expensive to run continuously and cheap to run nightly on a small corpus, and on a large one a sampled version of the same comparison does the job.

Rollback to a retained index

The previous index is retained, and the rollback caution holds here: an older version is useful only if it can still work with current data and access requirements. A retained index has the permissions of its build time. Activating it naively would restore narrowed access. The service logs every narrowing, quarantine, and deletion in the revocation log, and activation replays the entries since the index was built before the index serves a request.

def activate(self, index, t):
    """Make an index live. A retained index is older than the revocations since it was built."""
    n = 0
    if self.cfg.replay_on_activate:
        for seq, kind, doc in self.db.execute("SELECT seq,kind,doc_id FROM revocation_log WHERE seq>? ORDER BY seq", (index.built_seq,)).fetchall():
            if kind == "delete": index.drop(lambda c, d=doc: c["doc_id"] == d)
            else: self.sync_acl(doc, t, index)
            n += 1
        index.built_seq = self.seq()
    self.live = index; self.epoch += 1
    return n

The drill snapshots an index, then narrows one document, deletes another, and changes the content of a third, and reactivates the snapshot three ways:

== 2. rollback to a retained index
  before rollback (current index)            narrowed doc returned: False  deleted doc returned: False  per-diem value served: ['70 dollars']
  rollback, no replay                        narrowed doc returned: True   deleted doc returned: False  per-diem value served: []
  rollback + replay (2 revocation entries)   narrowed doc returned: False  deleted doc returned: False  per-diem value served: []
  reconcile after replay: {'orphan_versions': [('drive:per-diem', 1)], 'current_docs_without_chunks': ['drive:per-diem']}
  after catch-up of ['drive:per-diem']       narrowed doc returned: False  deleted doc returned: False  per-diem value served: ['70 dollars']
  reconcile after catch-up: {'orphan_versions': [], 'current_docs_without_chunks': []}

Without replay, the narrowed document comes back for a user who lost it. The deleted document stays hidden even without replay, because the registry marks it deleted and the query path consults the registry, which is the point of that second line of defense. With the two-entry replay, both stay hidden. The content change shows the remaining cost: the old index holds the previous version of the per-diem page, so the registry’s current version has no chunks in it, and the page disappears from search until a catch-up ingestion restores it. The reconcile output names exactly that document, as both an orphan version and a current document without chunks. A rollback therefore trades freshness for safety, and it fails closed on the pages it cannot vouch for. This is the same release-level reasoning as in Versioning Models, Data, Prompts, and Evaluation Artifacts, with the addition that permissions are part of the combination being restored.

An erasure request

An owner asks for an HR case document to be erased. The sequence follows Access, Retention, and Erasure Across Data Copies and Document Pipelines for Retrieval: Parsing, Versions, and Deletion: tombstone first, so serving stops at once; physical removal from every copy; verification by direct inventory and a canary query; and evidence that records the action without the content. The inventory counts live chunks, chunks in retained indexes, embedding cache entries that came only from the document, and cached answers that cite it.

== 3. erasure request for hr:settlement-authority (hr/cases)
  before: {'live': 3, 'retained': 3, 'embed_cache': 3, 'answer_cache': 1}
  logical (tombstone applied, purge not yet run): {'live': 3, 'retained': 3, 'embed_cache': 3, 'answer_cache': 1} | served to u20: False
  after purge job: {'live': 0, 'retained': 0, 'embed_cache': 0, 'answer_cache': 0}
  canary query with the document's own opening words returns it: False
  audit rows that still name the document (identifiers only, no text): 1
  registry row: {'status': 'deleted', 'deleted_at': 34560002.46547472}

The tombstone makes the document unservable within seconds, while the 15-minute purge job removes the three live chunks, three retained chunks, three embedding cache entries, and one cached answer. A canary query using the document’s own opening words returns nothing. One audit row still names the document by identifier, which is intended: the audit records that a request once returned it, and holds no text. The delete tombstone stays in the registry so that a restored copy cannot bring the document back, and the test for that exact replay is in the matrix. The statement of scope still applies: erasure here covers the index and its caches. It does not reach a person’s memory, a copy exported to someone’s laptop, or a model fine-tuned on the text, and it covers backups only when the restore procedure re-applies tombstones.

A connector release goes wrong

The last drill covers a failure of the pipeline itself. A new connector release maps the group g-support to g-all in its normalization table. Five minutes later the scheduled full ACL resync copies the mistake into the index for the four support documents, which become readable by everyone in the company. This is the “wrong ACL copied faithfully” case from the earlier access-control article. The harness replays about 3 hours of traffic at one request per 5 seconds on average, samples 10% of responses against the oracle, and checks the sample every 5 minutes. The response steps follow the containment ladder of AI Platform Incident Response and Recovery. The delays between steps (10 minutes to act on the alert, then 15 to repair) are assumptions, not measurements.

timeline (minutes after the release):
  deploy     +   0.0
  resync     +   5.0
  alert      +  10.0
  contain    +  20.0
  fix        +  35.0
  verify     +  40.0
  lever_off  +  40.0
documents whose index ACL changed during the repair resync: ['wiki:escalation', 'wiki:first-reply', 'wiki:macro-review', 'wiki:refund-window']
drift after repair: []
requests since the release: 1467; responses with an unreadable document: 14
  first at +5.3 min, last at +18.9 min; distinct (user, document) pairs: 13; users: 11; documents: 4
  documents: ['wiki:escalation', 'wiki:first-reply', 'wiki:macro-review', 'wiki:refund-window']
  before containment: 19 document exposures; after containment: 0
sampler violations before the alert: 3 | sampler violations after containment: 0

Detection came from the sampler 10 minutes after the release, and containment was a configuration change that applied the source recheck to every class, not only restricted ones. That lever stops the leak without waiting for the repair, because the index copy stops deciding. The repair then removed the faulty mapping and resynced every ACL from the source. A drift scan, which compares each document’s index ACL with the source’s normalized ACL, found no differences, and only then was the lever lifted. The exposure that had already happened cannot be undone. The audit log, which stores chunk identifiers and principals but no text, lets the team select those responses precisely: 14 responses to 11 users covering 13 user-document pairs and all 4 documents, from 5.3 to 18.9 minutes after the release. This is the worklist that goes to the document owners and to security, with the same split used in the earlier incident article between effects the platform can reverse and effects that left it. Detection time depends on traffic volume and sampling rate. At this volume, a 10% sample produced three violating responses before the first check that fired; a service with a hundredth of the traffic would need a higher sampling rate or a different plan, so the sampler’s coverage is itself an objective worth stating.

What remains unsolved

• The source is the authority. The system reproduces whatever the source grants. A document shared with the whole company becomes discoverable by search, and fixing it happens in the source, with search making the oversharing visible sooner.
• Windows are bounded, not zero. Standard documents can be exposed for up to the stated window after a narrowing. The recheck closes the window for restricted material and costs a source call. Applying it to everything is a decision about cost and source availability, not a free upgrade.
• What a user already saw stays seen. Revocation and deletion stop future responses. They cannot recall an answer that was shown, exported, or copied.
• Inference across permitted passages. Per-document permissions do not stop two readable passages from jointly revealing something neither states.
• The model and the data are simple stand-ins. A hashed bag of words and template-generated questions make retrieval easy. Real embeddings, real paraphrase, real parsing failures, and real source APIs with partial responses would add failures that this system does not exercise.
• Small samples. 32 users, 177 documents, and a few hundred questions show mechanisms. They do not support claims about rates in a production corpus.

The system is permission-correct within stated windows, fresh within stated times, recoverable without reviving revoked access, and measured against an oracle that does not share its copy of the truth. The next case, An Approved Agent for Transactional Business Work, moves from reading to acting. The same company’s agent issues refunds and order changes, so the questions become who may authorize an action, what a human approval binds to, how a half-finished action resumes after a crash, and how a ledger and the agent’s records are reconciled.

References

• pgvector README: filtering with approximate indexes and iterative index scans
• Elastic documentation: Filter approximate kNN results
• OWASP Gen AI Security Project: LLM08:2025 Vector and Embedding Weaknesses