Document Pipelines for Retrieval
Consider a hypothetical support assistant that answers questions from a company’s returns guide. In March the guide changes: the return window drops from 30 days to 14. The refresh job ingests the new version and adds its passages to the search index, but the step that removes the old version’s passages fails quietly. For the next week, a customer asking about returns may be shown either window, depending on which passage ranks higher for the exact wording of the question. Nothing crashed, the new document is indexed correctly, and a spot check on the new text passes.
A second scenario, also hypothetical: a document is withdrawn at the source because it should no longer be shared. The row in the source system disappears. The index still holds its passages, its vectors, an entry in an embedding cache, and a cached answer. The document is gone from one place and fully retrievable from four.
Both failures have the same cause. In retrieval-augmented generation (RAG), a model answers using passages fetched at request time, as in the original formulation by Lewis and colleagues, which pairs a pretrained generator with a retrievable non-parametric memory (a RAG system in the usual practical sense). The passages come from a search index that is derived data: built from source documents by a pipeline, and correct only as long as the pipeline keeps the derivation faithful. This article covers the pipeline that does the deriving: where each document came from, how it is parsed and chunked, how the index is refreshed when sources change, how deletion reaches every derived copy, and what to validate before anything becomes retrievable.
The article assumes the idea from the preceding article on feature pipelines that derived data should carry the times at which its inputs became known and should be reproducible from versioned inputs. The same discipline applies here, with documents in place of event streams. It also assumes basic Python and the idea of a vector embedding; the glossary entry on embeddings covers the part that matters here, which is that vectors from different models are not comparable.
What a retrieval pipeline must keep true
The pipeline has a simple shape: fetch documents from sources, parse bytes into structured text, split the text into chunks, attach metadata, compute embeddings, and write chunks and vectors to one or more indexes. A serving layer queries those indexes, filters by permissions, and hands passages to a model. Before looking at each stage, state the invariants that the whole pipeline exists to protect, because every design choice below is justified by one of them.
- Traceability. Every indexed chunk traces to exactly one document version, and that version can be found in the source or the raw store.
- Currency. Only the current version of a document, or a version explicitly selected by an effective date, is retrievable.
- Deletion completeness. When a document is removed, no derived artifact of it remains retrievable, and the pipeline cannot bring it back.
- Reproducibility. Given the manifest of document versions and the parser, chunker, and embedding configuration, the index can be rebuilt with the same chunks.
- Atomic publication. A failed or partial run does not leave the serving layer with a mix of states that no consistent view of the sources could produce.
Retrieval quality, meaning whether the right passage ranks highly for a question, is a separate concern that the glossary entry on chunking and evaluation practice address. A pipeline that satisfies the five invariants can still retrieve poorly, but a pipeline that violates them cannot be trusted regardless of how well it retrieves.
Document provenance: knowing where a chunk came from
Provenance in the W3C PROV overview’s words is information about the entities, activities, and people involved in producing a piece of data, usable to assess its quality, reliability, or trustworthiness. The PROV vocabulary maps naturally onto the pipeline. A document version is an entity. Parsing and chunking are activities. The parser and the embedding service are software agents, and the content owner is a responsible person or team. A pipeline does not need to adopt the PROV serialization to use the idea; it needs to record enough that, for any chunk, someone can answer three questions: what was it made from, how was it made, and may it be served to this reader.
Three identities, not one
A single document_id is not enough, because documents change. Separate:
- Document identity (
doc_id): stable across edits, derived from the source system and source key, for example the path or record ID in the system of record. - Version identity (
version_id): one stored state of the document. It can be the source’s revision number, an object-store version ID, or a content hash. - Chunk identity: one passage of one version, discussed in the next section.
Version identity needs care. An object-store version ID is opaque: S3 assigns a unique version ID to an object when it is created or modified in a versioning-enabled bucket, and the glossary entry on object versions notes that such an ID is opaque, so its lexical order does not establish which version is newer. An ETag is a validator, not a universal checksum; the glossary entry on object versions and ETags explains why an ETag should not be assumed to be the MD5 of the file. A modification timestamp can change without any content change (a file copied or re-saved) and, if the source clock is coarse, may fail to change when content does. For these reasons a robust pipeline stores both the source’s version marker, used as a cheap first check, and a content hash of the retrieved bytes, used as the authority on whether anything really changed.
The registry
The pipeline’s memory is a table, here called the document registry, with one row per document. A minimal schema is shown in the table.
| Field | Why it is needed |
|---|---|
doc_id, source_system, source_key | Identify the document across versions and tie it back to the source |
version_id, content_hash | Detect change; select which chunks are current |
retrieved_at | When the pipeline last fetched the bytes, a record time in the sense of system time |
effective_from, effective_to | When the content applies in the world, which can differ from when it was fetched |
owner, access_labels | Who is responsible and who may read it, captured at ingestion |
status | current, superseded, deleted, quarantined |
parser_version, parse_config_hash | Which parsing produced the stored text |
chunker_version, embedding_model | Which downstream configuration produced the chunks and vectors |
raw_uri | Where the retrieved bytes are retained, if they are retained |
deleted_at, delete_reason | Tombstone data, without content |
The distinction between effective_from and retrieved_at matters for the same reason the clocks matter in feature pipelines: a terms-of-service document published today may take effect next month, and a question about next month’s rules should find it while a question about today’s should not. Whether to support effective-dated retrieval is a product decision, but if the registry stores the dates, the option stays open.
Two further provenance items belong in the registry or alongside it. First, the terms of use of a document: which products and audiences may use it, and whether it may be copied into an index at all. Second, the access labels captured at ingestion. These are a snapshot; if permissions change in the source, the snapshot is stale until the next refresh, which is why serving needs a current check as well. The follow-up article on access controls for retrieval is where that serving-side check is developed.
Parsing: from bytes to structured text
Parsing converts a file into a sequence of elements (headings, paragraphs, list items, table cells, captions) with enough position information to cite and re-find them. Everything downstream depends on it, and the quality of parsing varies enormously by format.
HTML and Markdown carry explicit structure, so a parser can recover heading levels and lists directly. Word processor formats store structure in styles, which authors apply inconsistently. PDF is the hard case, and the pypdf documentation is candid about why. It explains that the PDF format is about producing a visual result for printing and was not designed for parsing content; there is no semantic layer, so nothing marks which text is a header, footer, page number, table, or paragraph, and any such labeling is a heuristic guess. Tables are typically absolutely positioned text, sometimes down to individual letters, which makes rows and columns hard to recover. Scanned documents contain images instead of text, so a text extractor returns little or nothing, and the pypdf documentation says it is not OCR software and cannot detect OCR failures.
Failure modes worth testing for
| Failure | What it looks like in the output | Typical consequence |
|---|---|---|
| Wrong reading order | In a two-column page, lines from both columns interleave | Sentences that never existed are indexed |
| Repeated page furniture | Running headers, footers, and page numbers appear in every page’s text | Noise in every chunk; false keyword matches |
| Flattened tables | Cells joined into one run of numbers and labels | A value is detached from its row and column labels |
| Scanned pages | Empty or very short text for a page that is visually full | The page is silently absent from retrieval |
| Encoding and ligature errors | Unusual character substitutions, broken words | Lexical search misses; embeddings degrade |
| Lost structure | Headings flattened to ordinary paragraphs | Chunks lose the section context that qualifies them |
| Hidden or non-displayed text | Text present in the file but not visible to a reader | Content shown to the model that no human reader saw |
The last row is a security consideration as much as a quality one. Retrieved text is data, not instructions; a document can contain text written to influence the model, and parsing is the first place that text enters the system. Parse untrusted files with resource limits, treat extracted text as untrusted input to the model’s prompt, and keep the extraction faithful to what a reader would see where that matters.
A parser is a versioned dependency
Changing the parser, its configuration, or the OCR model changes the extracted text even when the document is unchanged. The registry therefore records parser_version and a hash of the parse configuration, and the refresh logic must treat a parser change as a reason to reprocess affected documents. The same holds for the chunker and embedding model. This is not a theoretical concern: upgrading a PDF library can alter table extraction for thousands of documents at once, and without version fields in the registry there is no way to know which chunks came from which behavior.
Parse-time signals double as quality measures. For each document, record characters per page, the fraction of pages with almost no text, the share of lines repeated across pages, language detection results, and the number of tables detected. A change in these between versions of the same document, or an outlier relative to other documents of the same type, is a cheap early warning that parsing went wrong before any retrieval metric would show it.
Chunks and their metadata
A chunk is the unit that is embedded, indexed, retrieved, and cited. The glossary entry on chunking discusses size, overlap, and structure-aware splitting at length. This section concerns what a chunk record must carry so that the pipeline, not just the retrieval model, can manage it.
Two keys for two jobs
A chunk needs an identity for ownership and deletion, and a different identity for reuse.
- Ownership key:
(doc_id, version_id, ordinal). It says which version produced this chunk and where in that version it sits. Deleting a document or retiring a version means deleting by this key’s prefix. - Content key: a hash of the normalized chunk text together with the embedding model identifier. Two chunks with the same content key have the same vector, so the vector can be computed once and reused when a new version contains the passage unchanged.
Collapsing the two into one ID is a common design error. A key that includes the version cannot be reused across versions, so every refresh re-embeds everything. A key based only on content makes deletion ambiguous, since the same passage text may occur in several documents and one document’s deletion must not remove another’s chunk.
Metadata fields
| Field | Used for |
|---|---|
doc_id, version_id, ordinal | Ownership, deletion, ordering |
heading_path | Disambiguating context, display, and optionally part of the embedded text |
source_offsets or page range | Citation and returning the original span |
effective_from, effective_to | Effective-dated retrieval |
access_labels | Filtering at query time |
parser_version, chunker_version, embedding_model | Rebuild decisions and mixed-state detection |
content_hash | Embedding reuse and duplicate detection |
language | Routing and analyzer selection |
The heading path deserves a remark. A sentence like “Staff may take 16 weeks” is ambiguous without its heading, and many pipelines prefix the heading path to the text before embedding. That is a reasonable choice, but it means the heading becomes part of the content key: renaming a section heading changes the embedded text of every chunk beneath it. Decide deliberately whether the heading is part of embedded text, and record the decision as part of the chunker configuration.
The embedding model is part of the schema
Vectors are meaningful only relative to the model that produced them. The glossary entry on embeddings notes that changing the model changes the coordinate space and that equal vector lengths do not make vectors from different models comparable. Therefore store the model identifier on each vector, check the dimension at write time, and never mix two models inside one index that serves similarity search. When the model changes, the refresh becomes a rebuild into a new index, described below.
Incremental refresh
The simplest refresh rebuilds everything: fetch all documents, parse, chunk, embed, and write a new index. For a small corpus this is a legitimate choice. It is idempotent by construction, easy to reason about, and has no change-detection logic to get wrong. It becomes impractical when the corpus is large, when embedding calls cost money or time, or when freshness requirements demand updates within minutes. Incremental refresh is the response, and it shares its vocabulary with the incremental processing pattern in batch pipelines: a bounded change set, retained state, and a clear answer to what counts as a change.
Classifying documents
Each run compares the source’s current listing with the registry and puts every document into one of four classes.
| Class | Condition | Action |
|---|---|---|
| New | In the source, not in the registry | Process fully |
| Changed | Version marker or content hash differs, or parser, chunker, or model configuration differs | Process; replace the old version |
| Unchanged | Same hash and same configuration | Skip |
| Missing | In the registry as current, absent from the source | Tombstone, then delete (see the next section) |
Use the cheap check first and the authoritative check second. A version marker comparison needs no download; if it differs, fetch and hash the bytes; if the hash matches the registry, the document did not really change and only the marker needs updating. For database sources, change data capture can supply the change set directly, including deletes, which a timestamp scan cannot see.
Never infer deletion from a failed or partial listing. If the listing call times out or returns a page fewer than expected, treating every unlisted document as deleted would remove the corpus. Require a complete, successfully paginated listing, and add a guard that holds the run for confirmation when the number of apparent deletions exceeds a threshold relative to corpus size. The threshold is a judgment you tune, not a standard value.
Chunk-level reuse
When a document changes, most of its chunks usually do not. Reuse works through the content key: after parsing and chunking the new version, look up each chunk’s content key in an embedding cache and call the embedding model only for misses. Whether reuse actually saves work depends strongly on how chunks are cut, and the following program shows the effect on a tiny corpus.
The program models the pipeline end to end. Its documents are invented, and its “embedding” is a hash-derived vector with no semantic meaning; that substitution is deliberate, because the point is to count how many embedding calls each refresh makes and to show state transitions, not to measure retrieval quality. Python 3.12 and the standard library were used by the author to produce the output shown after each section of the code. The first part defines the source system, the registry and index as in-memory dictionaries, and the toy documents.
"""Retrieval-pipeline refresh and deletion model. Illustrative documents; standard library only.
The 'embedding' is a hash-derived vector with no semantic meaning; it only makes reuse countable."""
import hashlib, re
def sha(s): return hashlib.sha256(s.encode()).hexdigest()[:12]
# ---- toy source system: doc_id -> (version_id, text, acl, effective_from)
SOURCE = {
"returns-guide": ("r1", "# Returns\n## Window\nItems may be returned within 30 days.\n"
"## Exceptions\nOpened headphones cannot be returned.\n"
"## Refunds\nRefunds go to the original payment method.", ["staff", "public"], "2025-01-01"),
"shipping-guide": ("s1", "# Shipping\n## Rates\nStandard shipping takes 5 days.\n"
"## Tracking\nA tracking link is emailed at dispatch.", ["public"], "2025-01-01"),
"warranty-terms": ("w1", "# Warranty\n## Coverage\nCovers defects for 12 months.\n"
"## Claims\nSubmit a claim with proof of purchase.", ["public"], "2025-01-01"),
}
PARSER_V, CHUNKER_V, EMBED_MODEL = "p1", "c1", "toy-hash-8"
registry, index, embed_cache, stats = {}, {}, {}, {"embed_calls": 0}
The parser turns text into elements carrying a heading path, a stand-in for what a real parser extracts from structure. Two chunkers are provided: a structural one that emits one chunk per element, and a fixed-window one that cuts every six words regardless of structure. The embedding function checks a cache keyed by model name and text hash and counts the calls it actually makes. That counter is the quantity of interest.
# ---- parse: bytes -> elements with a heading path
def parse(text):
path, out = [], []
for line in text.split("\n"):
m = re.match(r"(#+) (.*)", line)
if m:
depth = len(m.group(1)); path = path[:depth - 1] + [m.group(2)]
elif line.strip():
out.append({"heading_path": " > ".join(path), "text": line.strip()})
return out
# ---- chunk: structure-aware (one chunk per element) or fixed word windows
def chunk_structural(els): return [(e["heading_path"], e["text"]) for e in els]
def chunk_fixed(els, size=6):
words = " ".join(e["text"] for e in els).split()
return [("", " ".join(words[i:i + size])) for i in range(0, len(words), size)]
def embed(text):
key = (EMBED_MODEL, sha(text)) # content key: model + normalized text
if key not in embed_cache:
stats["embed_calls"] += 1
embed_cache[key] = [b / 255 for b in hashlib.sha256(text.encode()).digest()[:8]]
return embed_cache[key]
The validation gate is a function that returns a list of problems; an empty list means the version may be published. It is intentionally small and is discussed in the validation section.
# ---- validation gate: returns a list of problems; empty list means publishable
def validate(doc_id, els, chunks, acl, prev_chunk_count):
problems = []
if not els: problems.append("parse produced no text")
if not acl: problems.append("missing access labels (fail closed)")
if any(len(t.split()) == 0 for _, t in chunks): problems.append("empty chunk")
if prev_chunk_count and len(chunks) < 0.5 * prev_chunk_count: problems.append("chunk count fell by more than half")
return problems
The refresh function is the heart of the model. It runs in three phases per document. Phase 1 writes the new version’s chunks, keyed by ownership key, alongside the old version’s chunks. Phase 2 flips the registry’s version_id pointer, which is the moment the new version becomes current. Phase 3 removes the old version’s chunks. A document that fails validation is quarantined and its previous version continues to serve. Missing documents are tombstoned in the registry. A tombstoned document is never re-ingested, even if it reappears in the source.
def refresh(source, chunker=chunk_structural, fail_old_delete=False):
"""Phase 1: write new version's chunks as pending. Phase 2: flip pointer. Phase 3: delete old chunks."""
report = {"new": [], "changed": [], "unchanged": [], "quarantined": {}, "deleted": []}
before = stats["embed_calls"]
for doc_id, (ver, text, acl, eff) in source.items():
reg = registry.get(doc_id)
if reg and reg["status"] == "deleted":
report["quarantined"][doc_id] = ["tombstoned: refusing to re-ingest"]; continue
chash = sha(text)
if reg and reg["content_hash"] == chash and reg["chunker"] == chunker.__name__:
report["unchanged"].append(doc_id); continue
els = parse(text); chunks = chunker(els)
prev = sum(1 for c in index.values() if c["doc_id"] == doc_id and c["version_id"] == (reg or {}).get("version_id"))
problems = validate(doc_id, els, chunks, acl, prev)
if problems:
report["quarantined"][doc_id] = problems; continue
for i, (hp, t) in enumerate(chunks): # phase 1: pending chunks, keyed by doc/version/ordinal
index[(doc_id, ver, i)] = {"doc_id": doc_id, "version_id": ver, "ordinal": i, "heading_path": hp,
"text": t, "vec": embed(t), "acl": acl, "effective_from": eff,
"parser": PARSER_V, "chunker": chunker.__name__, "model": EMBED_MODEL}
old = reg["version_id"] if reg else None
registry[doc_id] = {"version_id": ver, "content_hash": chash, "status": "current",
"parser": PARSER_V, "chunker": chunker.__name__} # phase 2: pointer flip
if old and old != ver and not fail_old_delete: # phase 3: remove superseded chunks
for k in [k for k in index if k[0] == doc_id and k[1] == old]: del index[k]
(report["changed"] if old else report["new"]).append(doc_id)
# source listing is complete here, so missing ids are real deletions (see text about partial listings)
for doc_id, reg in registry.items():
if doc_id not in source and reg["status"] == "current":
reg["status"] = "deleted"; report["deleted"].append(doc_id) # tombstone first
report["embed_calls"] = stats["embed_calls"] - before
return report
The remaining functions complete the model. retrieve is a toy keyword scorer; its important feature is that it filters through the registry, so a chunk is returned only if its document is current and its version matches the registry pointer. reconcile computes set differences in both directions between what the registry says should exist and what the index contains.
def purge_deleted():
for doc_id, reg in registry.items():
if reg["status"] == "deleted":
for k in [k for k in index if k[0] == doc_id]: del index[k]
def retrieve(query, role="public", filter_current=True):
q, hits = set(query.lower().split()), []
for k, c in index.items():
if role not in c["acl"]: continue
reg = registry]
if filter_current and (reg["status"] != "current" or reg["version_id"] != c["version_id"]): continue
score = len(q & set(c["text"].lower().replace(".", "").split()))
if score: hits.append((score, c["doc_id"], c["version_id"], c["text"]))
return sorted(hits, reverse=True)[:3]
def reconcile():
"""Set differences in both directions between registry expectations and the index."""
orphans = sorted({(k[0], k[1]) for k in index
if registry.get(k[0], {}).get("version_id") != k[1] or registry[k[0]]["status"] != "current"})
missing = sorted(d for d, r in registry.items() if r["status"] == "current"
and not any(k[0] == d and k[1] == r["version_id"] for k in index))
return {"orphan_versions": orphans, "current_docs_without_chunks": missing}
The first demonstration loads three documents, then changes one sentence in one section of the returns guide and refreshes.
if __name__ == "__main__":
print("== 1. initial load"); r = refresh(SOURCE)
print(" new:", r["new"], "| embed calls:", r["embed_calls"], "| chunks:", len(index))
print("\n== 2. one sentence changes in one section (returns-guide v2)")
SOURCE["returns-guide"] = ("r2", SOURCE["returns-guide"][1].replace("30 days", "14 days"),
SOURCE["returns-guide"][2], "2025-02-01")
r = refresh(SOURCE)
print(" changed:", r["changed"], "unchanged:", r["unchanged"], "| embed calls:", r["embed_calls"])
print(" index versions for returns-guide:", sorted({k[1] for k in index if k[0] == "returns-guide"}))== 1. initial load
new: ['returns-guide', 'shipping-guide', 'warranty-terms'] | embed calls: 7 | chunks: 7
== 2. one sentence changes in one section (returns-guide v2)
changed: ['returns-guide'] unchanged: ['shipping-guide', 'warranty-terms'] | embed calls: 1
index versions for returns-guide: ['r2']The initial load makes 7 embedding calls for 7 chunks. After the edit, the refresh classifies one document as changed and two as unchanged, and makes exactly one embedding call: the changed sentence. The other two chunks of the returns guide come from the cache, and the two unchanged documents are skipped without parsing. The index now holds only version r2 of the returns guide, because phase 3 removed r1. Those numbers are specific to this tiny corpus. They illustrate the mechanism; they are not a prediction of savings in a real corpus, where the changed fraction, the chunk count per document, and the edit pattern determine the result.
Chunk boundaries decide the blast radius of an edit
The second demonstration compares the two chunkers when a new paragraph is inserted near the top of a document.
print("\n== 3. same edit, but a prefix is inserted: structural versus fixed-window chunks")
for name, ch in (("structural", chunk_structural), ("fixed_6_words", chunk_fixed)):
registry.clear(); index.clear(); embed_cache.clear(); stats["embed_calls"] = 0
base = SOURCE["returns-guide"][1].replace("14 days", "30 days")
src = {"returns-guide": ("r1", base, ["public"], "2025-01-01")}
refresh(src, ch)
first = stats["embed_calls"]
src["returns-guide"] = ("r2", "# Returns\n## Notice\nThis guide covers online orders.\n" + base.split("\n", 1)[1],
["public"], "2025-01-01")
r = refresh(src, ch)
print(f" {name:<14} initial embeds {first:>2} | embeds after insertion {r['embed_calls']:>2}")== 3. same edit, but a prefix is inserted: structural versus fixed-window chunks
structural initial embeds 3 | embeds after insertion 1
fixed_6_words initial embeds 4 | embeds after insertion 4With structural chunking, the insertion adds one chunk, so one embedding call is needed after the change. With fixed six-word windows, inserting words at the start shifts every later window boundary, so every chunk’s text changes and all four are re-embedded. The structural chunker localizes change to the edited element; the fixed-window chunker spreads it across everything after the edit. Overlapping windows behave like the fixed case in this respect, because an edit changes the overlap regions of its neighbors as well. This is an engineering argument for splitting along document structure when documents have reliable structure, in addition to the retrieval-quality arguments in the chunking glossary entry. It also shows why the chunker configuration belongs in the registry: changing the window size or overlap changes every boundary and makes all chunks new.
Publishing: pointer flips and index swaps
There are two publication patterns, and they apply to different changes.
Per-document pointer flip. Used for ordinary edits. New chunks are written first, with the new version_id and not yet visible to queries; the registry pointer then flips; old chunks are removed last. Queries filter on the registry, so during the window in which both versions are physically present only the current one is returned. This is the pattern the program implements. Its correctness depends on the query path honoring the filter, which is why the third phase’s failure is handled below rather than assumed away.
Whole-index swap. Used when a configuration change affects every chunk: a new embedding model, a new chunker, or a new parser. Build a complete second index in the background from the manifest, run validation and canary queries against it, then switch the serving alias from the old index to the new one. Keep the old index for a bounded time to allow rollback, and decide the length of that window together with any deletion duties, since the old index also contains everything that was supposed to be removed since it was built. A swap avoids the mixed-embedding-space problem that an in-place update would create.
Concurrency and ordering also need attention. Two runs overlapping on the same document can write interleaved versions, so serialize runs per document with a lease or use a single writer. An older version arriving after a newer one (a delayed message, a retried job) must not overwrite it; compare using the source’s revision order where one exists and treat object-store version IDs as unordered. The glossary entries on idempotency and, for the broader pattern in batch pipelines, the article on incremental and idempotent pipelines give the principles: stable operation identity, retry-safe writes, and progress recorded only after publication.
A failed cleanup step
The third demonstration reproduces the opening scenario. A new version of the returns guide is published, but phase 3 fails, leaving the old chunks in the index.
registry.clear(); index.clear(); embed_cache.clear(); stats["embed_calls"] = 0
SOURCE["returns-guide"] = ("r1", SOURCE["returns-guide"][1].replace("14 days", "30 days"), ["staff", "public"], "2025-01-01")
refresh(SOURCE)
SOURCE["returns-guide"] = ("r2", SOURCE["returns-guide"][1].replace("30 days", "14 days"), ["staff", "public"], "2025-02-01")
print("\n== 4. new version published, but removal of the old version's chunks fails")
refresh(SOURCE, fail_old_delete=True)
q = "returned within days"
print(" unfiltered top hits:", [(h[1], h[2], h[3]) for h in retrieve(q, filter_current=False)][:2])
print(" current-version filter:", [(h[1], h[2], h[3]) for h in retrieve(q)][:1])
print(" reconcile:", reconcile())
for k in [k for k in index if k[0] == "returns-guide" and k[1] == "r1"]: del index[k]
print(" after retrying the delete:", reconcile())== 4. new version published, but removal of the old version's chunks fails
unfiltered top hits: [('returns-guide', 'r2', 'Items may be returned within 14 days.'), ('returns-guide', 'r1', 'Items may be returned within 30 days.')]
current-version filter: [('returns-guide', 'r2', 'Items may be returned within 14 days.')]
reconcile: {'orphan_versions': [('returns-guide', 'r1')], 'current_docs_without_chunks': []}
after retrying the delete: {'orphan_versions': [], 'current_docs_without_chunks': []}Without the version filter, the query returns both the 14-day passage and the superseded 30-day passage. With the filter that the pipeline’s registry enables, only the current passage is returned. The reconciliation check reports ('returns-guide', 'r1') as an orphan version: chunks in the index that the registry does not consider current. After the delete is retried, reconciliation is clean. Two lessons follow. First, a query-time filter is a safety net, not a substitute for cleanup, because orphans cost storage, compete for candidate slots in approximate searches, and become retrievable again if the filter is ever bypassed. Second, reconciliation is the control that turns a silent failure into an alert, and it should run after every refresh.
Deletion
Deletion in a retrieval system is harder than it looks, because the same document exists in several derived forms and each store has its own deletion semantics. Distinguish four situations that call for different handling:
- Source removal. The document no longer exists at the source; retrieval should stop returning it.
- Supersession. A newer version exists; the old one should stop being retrievable but might be retained for audit.
- Erasure or withdrawal. The document, or personal data inside it, must be removed under a legal or contractual obligation or an owner’s decision, including from backups and caches within the stated limits. The article on access, retention, and erasure treats the obligations side; this article treats the mechanics of reaching every copy.
- Access revocation. The document still exists but a reader may no longer see it. This is a permissions change and is handled at query time, not by deleting.
Where the copies live
| Artifact | Keyed by | How it is removed | How to verify |
|---|---|---|---|
| Source object, raw store copy | Object key and version ID | Versioned stores keep history; see the next paragraph | List versions; confirm none remain if permanent removal is required |
| Parsed text | doc_id, version_id | Delete by key prefix | Look up by doc_id |
| Chunk store and keyword index | (doc_id, version_id, ordinal) | Delete by doc_id across all versions | Count by doc_id; reconcile |
| Vector index | Same ownership key | Delete by ownership key; understand engine cleanup | Query by id and by canary question |
| Embedding cache | Content key | Remove entries whose text came only from the deleted document | Check the cache for the deleted text’s hashes |
| Answer or retrieval caches | Query or request | Invalidate by document dependency | Replay known queries |
| Evaluation sets, logs, prompt traces | Varies | Apply the same removal where they hold document text | Search by hash or identifier |
| Backups and restored copies | Snapshot | Expire on their schedule; re-apply deletions on restore | Check the restore procedure, not just the live store |
Object stores need special mention. In an S3 bucket with versioning enabled, a simple delete request does not remove the object; S3 inserts a delete marker that becomes the current version, a GET then behaves as if the object were gone, and the earlier versions still exist. Permanently removing a version requires a delete request that names the version ID. That means “deleted from the bucket” can mean either “hidden” or “gone”, and the raw-store step of a deletion pipeline must say which one is required.
Engines delete in two steps
Index engines commonly mark data as deleted before they physically remove it. Elasticsearch’s documentation explains that a shard is a Lucene index made of immutable segments, and that smaller segments are periodically merged into larger ones to keep the index size down and to expunge deletes. A deleted document is therefore invisible to search after the delete but may remain in a segment file until a merge rewrites that segment. With pgvector, which stores vectors in PostgreSQL, the project’s README shows a plain DELETE statement for removing a vector, has a Vacuuming section noting that VACUUM can take a while on HNSW indexes and can be sped up by reindexing first, and mentions dead tuples as one reason a query may return fewer results than expected. From those statements it is reasonable to infer, though the README does not spell it out, that deleted rows linger in the table and index until vacuumed. Neither engine’s behavior is a defect; both suggest that “the delete returned success” and “the bytes are gone from disk” are different statements, which matters if a retention or erasure obligation is defined in terms of physical removal. Decide which meaning your obligation requires and check what your engine does at that level.
There is a related retrieval-time subtlety. pgvector’s documentation states that with approximate indexes, filtering is applied after the index is scanned, so a filter that matches few rows can return fewer results than requested; it describes iterative index scans as a way to scan more of the index until enough results are found. A query-time filter on status = 'current' or on access labels is exactly such a filter. If many nearest neighbors belong to tombstoned or non-permitted documents, top-k results may shrink or be empty even though valid matches exist deeper in the index. Another reason to physically remove deleted data promptly is that it stops competing for the candidate list.
Tombstone first, then delete
The safe order is to record the deletion before performing it, as the following sequence makes explicit.
- Write a tombstone in the registry: set
status = deleted,deleted_at, anddelete_reason, without storing the document’s content. A delete tombstone retains the key and version evidence that prevents a replay or stale copy from resurrecting the row. - Serving now excludes the document, because queries filter through the registry. This takes effect immediately and does not wait for any physical step.
- Run the physical deletions across the artifacts in the table above, each with retries, tracked per destination so that one failure does not hide the others. The glossary entry on data deletion makes the same point: one successful operation is not the whole request, and a failed index deletion should restrict serving while it is repaired.
- Verify by direct lookup of the ownership key and by canary questions that previously retrieved the document.
- Record evidence of the action and outcome without copying the deleted payload into the evidence.
The fourth demonstration follows that order for a document withdrawn at the source.
print("\n== 5. warranty-terms removed at the source")
gone = SOURCE.pop("warranty-terms")
r = refresh(SOURCE); print(" tombstoned:", r["deleted"], "| chunks still physically present:",
sum(1 for k in index if k[0] == "warranty-terms"))
print(" retrieval for 'warranty defects months':", retrieve("warranty defects months"))
purge_deleted(); print(" after physical purge, chunks present:", sum(1 for k in index if k[0] == "warranty-terms"))
SOURCE["warranty-terms"] = gone # a stale copy reappears in the source listing
r = refresh(SOURCE); print(" stale copy reappears:", r["quarantined"])
print(" retrieval still empty:", retrieve("warranty defects months") == [])
del SOURCE["warranty-terms"]== 5. warranty-terms removed at the source
tombstoned: ['warranty-terms'] | chunks still physically present: 2
retrieval for 'warranty defects months': []
after physical purge, chunks present: 0
stale copy reappears: {'warranty-terms': ['tombstoned: refusing to re-ingest']}
retrieval still empty: TrueAfter the refresh, the document is tombstoned while two of its chunks are still physically present, and retrieval already returns nothing for a question that matched them before, because the registry filter excludes it. After the physical purge, no chunks remain. When a stale copy reappears in the source listing, which stands in for a restored backup or a replayed file, the pipeline refuses to ingest it, because the tombstone is consulted before anything else. Retrieval remains empty.
The tombstone must outlive the longest period during which a stale copy could reappear: the replay horizon of the upstream system, the retention of backups that could be restored, and the lifetime of mirrors. It should carry only identifying fields. Keeping the document’s text in order to remember that it was deleted would defeat the purpose. The glossary entry on data retention adds the discipline of naming the start event and the action at expiry, and applies to tombstones and registry rows as well as to content.
What deletion does not cover
Deleting a source document from the retrieval pipeline does not remove information from a model trained or fine-tuned on it, and deleting a training file does not establish removal of its influence from model parameters; that requires a separate assessment. It does not reach text that a user copied into a conversation, a downstream export, or a third-party processor unless those are in the tracked destinations. And a query that returns nothing proves only what that interface returned. State the scope of each deletion and the exceptions that remain, as the data deletion glossary entry advises.
Ingestion validation
Validation is the gate between “the pipeline produced something” and “the index may serve it”. Checks belong at each stage, with a clear action for each failure: block the whole run, quarantine one document, or warn.
| Stage | Check | Catches | Typical action |
|---|---|---|---|
| Fetch | Size within limits, type matches extension, checksum matches source when provided | Truncated downloads, wrong formats, oversized files | Quarantine document |
| Parse | Non-empty text; characters per page above a floor; repeated-line ratio below a ceiling | Scanned pages, extraction failure, page furniture | Quarantine; flag for OCR |
| Chunk | No empty chunks; token length within model limits; chunk count within a band of the previous version | Splitter bugs, silently truncated documents | Quarantine or hold for confirmation |
| Metadata | Required fields present and parseable; at least one access label | Orphaned permissions, unparseable dates | Quarantine; fail closed on missing labels |
| Embed | Vector dimension equals the model’s; no NaN or all-zero vectors; model identifier recorded | Model mismatch, failed calls | Block the batch |
| Index | Registry-to-index counts reconcile in both directions | Orphans, missing chunks, partial writes | Block publication or alert |
| Behavior | Canary questions with known answer passages are retrieved; canaries about deleted documents return nothing | Regressions that structural checks cannot see | Block swap; alert |
The gate in the model
The validation function in the model checks four things: that parsing produced text, that access labels exist, that no chunk is empty, and that the chunk count did not fall by more than half relative to the previous version. The fifth demonstration submits three bad inputs.
print("\n== 6. validation gate")
SOURCE["empty-scan"] = ("e1", "", ["public"], "2025-01-01")
SOURCE["no-labels"] = ("n1", "# Notes\n## A\nSome text here.", [], "2025-01-01")
SOURCE["shipping-guide"] = ("s2", "# Shipping", ["public"], "2025-03-01")
r = refresh(SOURCE)
for d, p in r["quarantined"].items(): print(" quarantined", d, "->", p)
print(" shipping-guide still serves version:", registry["shipping-guide"]["version_id"])== 6. validation gate
quarantined shipping-guide -> ['parse produced no text', 'chunk count fell by more than half']
quarantined empty-scan -> ['parse produced no text']
quarantined no-labels -> ['missing access labels (fail closed)']
shipping-guide still serves version: s1The empty scan and the document with no access labels are quarantined and never reach the index. The shipping guide’s new version is quarantined for two reasons, and the previous version continues to serve, so a bad update degrades to a stale document and not to a missing one. The access-label rule is a deliberate fail-closed choice: a document with no labels is not indexed rather than being indexed as public. The one-half threshold in the chunk-count check is an arbitrary value chosen for the example; real thresholds depend on how much documents of a given type legitimately change between versions, and they should be tuned from observed distributions.
Contracts with content owners
Some checks cannot be done by the pipeline alone, such as whether an effective date is correct or whether a document is allowed in the index. A data contract between content owners and the pipeline states what a producer supplies and what consumers may rely on: required metadata fields, who the owner is, how withdrawals are signaled, and what happens on a failed check. Enforcement still lives in the pipeline; the contract makes the expectations explicit and lets a failed check be traced to a responsible party.
Reconciliation and canaries are different tools
Reconciliation compares sets: every current document in the registry has the expected number of chunks in the index, and every chunk in the index belongs to a current version of a registered document. It finds structural drift. Canary queries test behavior: a small set of questions with known answer passages, and a set of questions about withdrawn documents that must return nothing. They find failures that look structurally fine, such as a parser change that kept all chunks but scrambled their text. Neither replaces the other, and neither measures answer quality; they establish that the pipeline’s invariants hold.
A hypothetical timeline
The table below combines several failures in an illustrative scenario, invented for this article and not drawn from any real system, to show how the controls interact.
| Time | Event | Without the controls | With the controls |
|---|---|---|---|
| Day 0 | Version 1 of a returns guide is indexed | Same | Registry row, ownership keys, content keys recorded |
| Day 3 | Version 2 published; index cleanup step times out | Both versions retrievable; answers vary | Registry flips to v2; query filter hides v1; reconciliation flags orphan v1 and the cleanup is retried |
| Day 5 | A scanned appendix is added; the parser returns empty text | The appendix is silently absent | Quarantined; previous version keeps serving; owner notified |
| Day 8 | A document is withdrawn at the source | Passages, vectors, and cached answers remain | Tombstone excludes it at once; purge across artifacts; canary confirms |
| Day 12 | A backup of the raw store is restored | The withdrawn document is re-ingested | Tombstone blocks ingestion |
The scenario is constructed to show that the controls are cheap relative to the failures: a registry row, a status filter, a reconciliation query, and a handful of canary questions.
Choosing the level of machinery
None of this is required for every system. A small, slowly changing corpus with no access restrictions and no deletion obligations can be rebuilt nightly into a fresh index and swapped atomically; that design is idempotent and has no change-detection code to be wrong. Move toward incremental refresh when rebuild time or embedding cost makes it impractical, or when freshness requirements exceed what a rebuild schedule allows. Add the registry, tombstones, and reconciliation as soon as there is more than one version of any document, any permission, or any removal duty, which is nearly always. Managed retrieval services can perform parts of this work, and the checklist above is still useful for asking what the service does for you and what it leaves to you: version handling, delete semantics, how quickly deletes take effect, and what is retained.
A pipeline that satisfies the five invariants does not guarantee a correct answer. A retrieved passage can still be misread by the model, may not answer the question, or may be an outdated document that the source itself never updated. What the pipeline can promise is narrower and testable: each passage came from a known version of a known document, only current and permitted passages are retrievable, deleted documents stay deleted, and a failed run does not publish a half-built state.
References
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401)
- W3C, PROV-Overview: An Overview of the PROV Family of Documents
- pypdf documentation, Extract Text from a PDF
- Amazon S3 User Guide, Retaining multiple versions of objects with S3 Versioning
- Amazon S3 User Guide, Deleting object versions from a versioning-enabled bucket
- Elasticsearch documentation, Merge settings
- pgvector README (filtering, iterative index scans, vacuuming)
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.