Building a Retrieval Index with Traceable Evidence

Consider a hypothetical support assistant that answers questions from a company’s product guides. A user asks how long the warranty on a product lasts, the assistant answers “9 months,” and the answer cites a passage. Whether that citation is worth anything depends on three things the index has to guarantee. The passage must exist in the guide, in the version that is current. A reviewer must be able to find the passage’s exact words from the citation alone. And if the index is rebuilt tomorrow, the same question must retrieve the same evidence, or someone must be able to say why it does not.

A retrieval index is derived data. The article on document pipelines for retrieval covers how source files become parsed versions, how a registry tracks them, and how deletion reaches every copy. Lewis and colleagues, in the paper that introduced retrieval-augmented generation, name providing provenance for a model’s outputs and updating its knowledge as the problems an explicit retrievable memory addresses; both depend on the index being built and maintained carefully. This article starts at the handoff: the parsed text of a known document version arrives, and the application builds a searchable index in which every hit can be traced back to its source. That means deciding what a chunk is, how chunks are embedded and how the index changes when documents change, what a retrieval result must carry so that it can be checked, how to name the state of the index, and which tests detect that the index has stopped doing its job. The embedding concepts are those of the article on embeddings and retrieval representations, and the tests follow the principles of reproducible evaluation. The retrieval step in the context of a full application is outlined in the overview of retrieval-augmented generation and tool-using agents; this article expands its index and evidence steps.

A small corpus to build against

Every number below comes from one synthetic setup, so the mechanisms can be observed and counted. The corpus is 300 generated product guides with one version each. A guide has a title, three sections (Returns, Warranty, Shipping), and a few sentences per section, with a different number in each guide for the return window, warranty length and shipping time. The product name appears only in the title. The body sentences do not repeat it, which is common in real documents: a section reads “Covered defects are repaired for 9 months” and relies on its heading for the subject. The gold set has 120 questions, each naming one guide and one kind of fact, such as “How long is the warranty on Tunelo?”. A gold item stores the document and the answer sentence, not a chunk identifier.

The embedding is a stand-in. It hashes each word into one of 128 buckets with a sign, sums the signed counts and normalizes to unit length. It has no semantics, no term weighting and no learned notion of similarity. What it provides is determinism and a vector interface, so that counts of embedding calls, index sizes and failures can be reproduced exactly. A real encoder changes the retrieval numbers, not the mechanisms around them; calls to a hosted embedding service cannot be run here and are not part of the setup. All code was run with Python 3.12 and NumPy 2.5.3. The library is one module, idx.py, and each script imports from it.

import hashlib, json, random, re
from dataclasses import dataclass, asdict
import numpy as np

def h(s, n=12):
    return hashlib.sha256(s.encode()).hexdigest()[:n]

SYL = ["ka", "lo", "mi", "ra", "ze", "tu", "vo", "ne", "si", "pa", "do", "fu"]

def make_doc(name, rng):
    p = dict(ret=rng.randrange(7, 61), war=rng.randrange(3, 37),
             ship=rng.randrange(2, 15), free=rng.randrange(20, 200, 10))
    text = (f"# {name} Service Guide\n"
            f"## Returns\nItems may be returned within {p['ret']} days of delivery. "
            f"Opened items need the original packaging. Refunds go to the original payment method.\n"
            f"## Warranty\nCovered defects are repaired for {p['war']} months. "
            f"Claims need proof of purchase. Accidental damage is not covered.\n"
            f"## Shipping\nStandard shipping takes {p['ship']} days. "
            f"A tracking link is sent when the parcel is dispatched. "
            f"Orders over {p['free']} dollars ship free.\n")
    return text, p

def make_corpus(n_docs, seed=7):
    rng = random.Random(seed)
    names = set()
    while len(names) < n_docs:
        names.add("".join(rng.choice(SYL) for _ in range(3)).capitalize())
    store, params = {}, {}
    for name in sorted(names):
        store[(name.lower(), "v1")], params[name.lower()] = make_doc(name, rng)
    return store, params

def make_gold(params, store, n, seed=11):
    """Each gold item names a document and the answer sentence; no chunk id is stored."""
    rng = random.Random(seed)
    items = []
    for doc_id in rng.sample(sorted(params), n):
        p, name = params[doc_id], doc_id.capitalize()
        kind = rng.choice(["ret", "war", "ship"])
        q, anchor = {
            "ret": (f"What is the return window for {name}?",
                    f"Items may be returned within {p['ret']} days of delivery."),
            "war": (f"How long is the warranty on {name}?",
                    f"Covered defects are repaired for {p['war']} months."),
            "ship": (f"How many days does standard shipping take for {name}?",
                     f"Standard shipping takes {p['ship']} days."),
        }[kind]
        items.append({"query": q, "doc_id": doc_id, "anchor": anchor, "kind": kind})
    return items

STOP = set("the a an is are of for to how what many long do i on be may and or with when go does".split())

def tokens(text):
    out = []
    for w in re.findall(r"[a-z0-9]+", text.lower()):
        if w in STOP:
            continue
        if len(w) > 4:
            w = re.sub(r"(ed|ing|s)$", "", w)
        out.append(w)
    return out

class HashEmbedder:
    """Signed feature hashing of word tokens. No semantics; it only makes vectors deterministic."""
    def __init__(self, dim=128, salt="a"):
        self.dim, self.salt = dim, salt
    @property
    def ident(self):
        return f"hash-{self.dim}-{self.salt}"
    def embed(self, text):
        v = np.zeros(self.dim)
        for t in tokens(text):
            d = int(h(self.salt + t, 8), 16)
            v[d % self.dim] += 1.0 if (d >> 20) & 1 else -1.0
        n = np.linalg.norm(v)
        return v / n if n else v

Chunk structure

What a chunk record holds

A chunk is the unit that is embedded, retrieved and cited. The document pipeline article gives a chunk an ownership key, (doc_id, version_id, ordinal), which says which version produced it and where it sits, and a separate content key for reusing vectors. A retrieval index needs four more things from each chunk. The span is the character range [start, end) that the chunk occupies in the stored text of its version. The heading path is the chain of headings above it. The text is what is shown to a model and cited. The embedded text is what the encoder actually sees, which can differ from the text.

The record is built around one invariant: source[start:end] == text, where source is the stored parsed text of that version. The provenance checks later in the article rest on it. It holds when the chunker slices the source and never rewrites it. A chunker that normalizes whitespace, joins hyphenated words or strips markup while it chunks produces text that no longer matches any slice, and the span becomes a claim that cannot be checked. Any normalization belongs upstream, in the parse step, and its output is the stored text that spans refer to. That stored text has to be kept immutable per version. If the application needs to point at a page of the original PDF, the parse step also has to emit a map from offsets to pages; the span alone does not give one.

Boundaries

The chunker works on sentences. It packs consecutive sentences into a chunk until a word budget would be exceeded, and it starts a new chunk when a section changes unless the configuration allows chunks to cross sections. A sentence is never split, so a single long sentence can exceed the budget. The unit of the budget matters: an embedding model truncates or rejects input beyond its token limit, so a production budget should be counted in tokens with the model’s own tokenizer. Words are used here only because the stand-in has no tokenizer.

@dataclass(frozen=True)
class ChunkConfig:
    max_words: int = 60
    cross_section: bool = False
    header: bool = True          # prepend the heading path to the text that is embedded

SENTENCE = re.compile(r"\S[^.!?\n]*[.!?]")

def units(text):
    """Body sentences with character offsets into `text` and the heading path in force."""
    path, pos, out = [], 0, []
    for line in text.splitlines(keepends=True):
        m = re.match(r"(#+) (.*)", line)
        if m:
            path = path[:len(m.group(1)) - 1] + [m.group(2).strip()]
        else:
            for s in SENTENCE.finditer(line):
                out.append((pos + s.start(), pos + s.end(), " > ".join(path)))
        pos += len(line)
    return out

@dataclass
class Chunk:
    doc_id: str
    version_id: str
    ordinal: int
    start: int
    end: int
    heading_path: str
    text: str
    embed_text: str

def chunk_document(doc_id, version_id, text, cfg):
    groups, cur, words = [], [], 0
    for a, b, path in units(text):
        n = len(text[a:b].split())
        new_section = cur and path != cur[0][2]
        if cur and (words + n > cfg.max_words or (new_section and not cfg.cross_section)):
            groups.append(cur); cur, words = [], 0
        cur.append((a, b, path)); words += n
    if cur:
        groups.append(cur)
    chunks = []
    for i, g in enumerate(groups):
        a, b, path = g[0][0], g[-1][1], g[0][2]
        body = text[a:b]                     # contiguous slice of the source version
        chunks.append(Chunk(doc_id, version_id, i, a, b, path, body,
                            f"{path}\n{body}" if cfg.header else body))
    return chunks

The first guide in the corpus, chunked with a budget of 20 words, shows what the records look like. Section boundaries are respected, a long section is split between sentences, and each chunk carries offsets into the guide.

from idx import *

store, params = make_corpus(300)             # 300 synthetic guides, one version each
gold = make_gold(params, store, 120)         # 120 questions with answer-sentence anchors
emb = HashEmbedder(128, "a")

doc_id = sorted(params)[0]
print(store[(doc_id, "v1")])
for c in chunk_document(doc_id, "v1", store[(doc_id, "v1")], ChunkConfig(max_words=20)):
    print(c.ordinal, (c.start, c.end), repr(c.heading_path), repr(c.text))

# Dodora Service Guide
## Returns
Items may be returned within 50 days of delivery. Opened items need the original packaging. Refunds go to the original payment method.
## Warranty
Covered defects are repaired for 9 months. Claims need proof of purchase. Accidental damage is not covered.
## Shipping
Standard shipping takes 13 days. A tracking link is sent when the parcel is dispatched. Orders over 160 dollars ship free.

0 (34, 125) 'Dodora Service Guide > Returns' 'Items may be returned within 50 days of delivery. Opened items need the original packaging.'
1 (126, 168) 'Dodora Service Guide > Returns' 'Refunds go to the original payment method.'
2 (181, 288) 'Dodora Service Guide > Warranty' 'Covered defects are repaired for 9 months. Claims need proof of purchase. Accidental damage is not covered.'
3 (301, 388) 'Dodora Service Guide > Shipping' 'Standard shipping takes 13 days. A tracking link is sent when the parcel is dispatched.'
4 (389, 423) 'Dodora Service Guide > Shipping' 'Orders over 160 dollars ship free.'

Chunk 1, “Refunds go to the original payment method.”, is a fragment that names no product and does not even say whether it concerns returns. Its meaning is carried entirely by its heading path. Chunks 3 and 4 are the same section of the same document, split at a sentence boundary. The spans are contiguous slices of the source, so the invariant holds, and a reader who knows the version and the span can recover the exact characters.

Heading context

A fragment like chunk 1 is hard to retrieve and hard to interpret when retrieved alone. The cheapest remedy is to prepend the heading path to the text before embedding, while keeping the stored and cited text unchanged. The chunker does this when header=True: embed_text is the heading path, a newline and the chunk text. Anthropic describes a more expensive variant, contextual retrieval, in which a language model writes a short explanation of where each chunk sits in its document and that explanation is prepended before embedding and lexical indexing; it reports a 35% reduction in the retrieval failure rate for contextual embeddings alone (5.7% to 3.7%) in its experiments. The heading path is deterministic and needs no model call. A generated context costs one model call per chunk and can say things the headings do not.

Whatever goes into the embedded text is part of the embedding recipe, discussed below, and changing it changes every vector under the affected headings.

Choosing size and context with a gold set

Relevance is decided by covers: a chunk is relevant to a question when it belongs to the question’s document and its span contains the answer sentence in that chunk’s own version. evaluate runs every question against an index and returns hit@1, hit@5 and MRR@10.

def covers(chunk, gold, store):
    if chunk.doc_id != gold["doc_id"]:
        return False
    a = store[(chunk.doc_id, chunk.version_id)].find(gold["anchor"])
    return a >= 0 and chunk.start <= a and a + len(gold["anchor"]) <= chunk.end

def evaluate(index, gold, store, k=10, nprobe=None):
    ranks = []
    for g in gold:
        hits, _ = index.search(index.query_vector(g["query"]), k, nprobe)
        r = next((i + 1 for i, (row, _) in enumerate(hits) if covers(index.chunks[row], g, store)), None)
        ranks.append(r)
    n = len(gold)
    return {"hit@1": sum(r == 1 for r in ranks) / n,
            "hit@5": sum(r is not None and r <= 5 for r in ranks) / n,
            "mrr@10": sum(1 / r for r in ranks if r) / n}, ranks

The grid below builds the index under six configurations and runs the 120 gold questions against each. build_index, defined in the next section, chunks every version in the store, embeds the chunks and returns an index. Hit@1 is the share of questions whose top result covers the answer sentence, hit@5 the share with a covering chunk in the top five, and MRR@10 the mean reciprocal rank of the first covering chunk, as defined in the embeddings article.

from idx import *

store, params = make_corpus(300)
gold = make_gold(params, store, 120)
emb = HashEmbedder(128, "a")

print(f"{'max_words':>9} {'cross':>5} {'header':>6} {'chunks':>6} {'words/chunk':>11} {'hit@1':>6} {'hit@5':>6} {'mrr@10':>6}")
for cfg in [ChunkConfig(20, False, False), ChunkConfig(60, False, False), ChunkConfig(400, True, False),
            ChunkConfig(20, False, True), ChunkConfig(60, False, True), ChunkConfig(400, True, True)]:
    ix = build_index(store, cfg, emb, {})
    m, _ = evaluate(ix, gold, store)
    wpc = sum(len(c.text.split()) for c in ix.chunks) / len(ix.chunks)
    print(f"{cfg.max_words:9d} {str(cfg.cross_section):>5} {str(cfg.header):>6} {len(ix.chunks):6d} {wpc:11.1f} "
          f"{m['hit@1']:6.3f} {m['hit@5']:6.3f} {m['mrr@10']:6.3f}")

max_words cross header chunks words/chunk  hit@1  hit@5 mrr@10
       20 False  False   1500        12.0  0.000  0.008  0.004
       60 False  False    900        20.0  0.000  0.008  0.002
      400  True  False    300        64.0  0.008  0.008  0.010
       20 False   True   1500        12.0  0.442  0.925  0.640
       60 False   True    900        20.0  0.258  0.900  0.512
      400  True   True    300        64.0  0.217  0.517  0.344

Without the heading path the index is at chance in every configuration, whatever the chunk size. The product name occurs only in the title line, which is a heading, so no chunk text contains it, and the encoder cannot tell one guide’s warranty chunk from another’s. Even the 400-word cross-section configuration does not help: its chunk begins at the first sentence after the title and contains the section headings between sentences but not the title.

With the heading path, smaller chunks do better here: hit@5 is 0.925 at 20 words, 0.900 at 60 and 0.517 at 400. A 400-word budget with cross-section packing puts a guide’s three sections in one chunk, so the vocabulary of returns, warranty and shipping competes inside one vector and a question about one of them matches only a third of it. This blur is a property of the bag-of-words stand-in, which has no way to focus on part of a chunk. The general pattern is that a large chunk blurs topics and a small one loses context, and a learned encoder shifts where the best point lies. It has to be measured on the target corpus. The gap between 0.925 and 0.900 is small enough that, with 120 questions, it is not evidence by itself.

The remaining errors are not about chunking. The next script groups the top-1 misses of the best configuration by where the wrong chunk came from.

from collections import Counter
from idx import *

store, params = make_corpus(300)
gold = make_gold(params, store, 120)
cfg = ChunkConfig(20, False, True)
ix = build_index(store, cfg, HashEmbedder(128, "a"), {})
m, ranks = evaluate(ix, gold, store)

misses = Counter()
for g, r in zip(gold, ranks):
    if r != 1:
        top = ix.chunks[ix.search(ix.query_vector(g["query"]), 1)[0][0][0]]
        misses["other document" if top.doc_id != g["doc_id"] else "same document, other chunk"] += 1
print("top-1 misses:", dict(misses))
for kind in ("ret", "war", "ship"):
    sel = [r for g, r in zip(gold, ranks) if g["kind"] == kind]
    print(f"{kind:5s} n={len(sel):3d} hit@1={sum(r == 1 for r in sel) / len(sel):.2f} "
          f"hit@5={sum(r is not None and r <= 5 for r in sel) / len(sel):.2f}")

top-1 misses: {'other document': 65, 'same document, other chunk': 2}
ret   n= 47 hit@1=0.51 hit@5=0.91
war   n= 33 hit@1=0.21 hit@5=0.85
ship  n= 40 hit@1=0.55 hit@5=1.00

Of 67 top-1 misses, 65 are another guide’s chunk for the same section. The product name is one signed count among about ten in a vector that treats every word alike, so a wrong guide’s “Shipping” chunk scores almost the same as the right one. The warranty questions fare worst at hit@1 (0.21): the body of a warranty chunk shares fewer words with the question than a returns or shipping chunk does, so even the topic match comes mostly from the heading. A dense or lexical retriever with proper term weighting handles the name better, and exact identifiers are the standard case for a lexical leg; that combination is the subject of the article on hybrid retrieval, reranking, and query transformation. The slicing by question kind is the tool to notice: a single average of 0.44 hit@1 hides that shipping questions are twice as easy as warranty questions.

Returning more than the matched chunk

Small chunks match well and read poorly; large chunks read well and match poorly. A common compromise is to match on small chunks and return the enclosing section or neighboring chunks as the context for the model. The record needs one more span for that: the parent range in the same version, which has the same form as the chunk’s own span. The cost is more prompt tokens per hit and a larger share of text that is not the matched evidence. The citation should then say which span matched and which span was supplied, so a reader can tell the sentence that supports a claim from the context around it.

Embeddings and index updates

The index object

The index holds the chunks, one vector per chunk, an alive flag per chunk, and optionally an inverted-file structure for approximate search, described below. Its constructor looks up each chunk’s content key in a cache before calling the embedder and counts the calls it makes. query_vector refuses a query embedder whose identifier differs from the index’s, and search either scans every row or probes a few cells, skipping rows that are marked dead. manifest and version are used in the versioning section.

class Index:
    def __init__(self, embedder, cfg, chunks, cache):
        self.embedder, self.cfg, self.chunks = embedder, cfg, chunks
        self.embed_calls = 0
        rows = []
        for c in chunks:
            key = h(embedder.ident + "|" + c.embed_text)        # content key: model + embedded text
            if key not in cache:
                cache[key] = embedder.embed(c.embed_text); self.embed_calls += 1
            rows.append(cache[key])
        self.V = np.vstack(rows)
        self.alive = np.ones(len(chunks), dtype=bool)
        self.ivf = None

    def manifest(self, store):
        docs = sorted((d, v, h(t)) for (d, v), t in store.items())
        return {"embedder": self.embedder.ident, "chunk_config": asdict(self.cfg), "documents": docs,
                "ivf": self.ivf["params"] if self.ivf else None}

    def version(self, store):
        return h(json.dumps(self.manifest(store), sort_keys=True), 16)

    def query_vector(self, query, embedder=None):
        e = embedder or self.embedder
        if e.ident != self.embedder.ident:
            raise ValueError(f"query model {e.ident} != index model {self.embedder.ident}")
        return e.embed(query)

    def search(self, qv, k=5, nprobe=None):
        """Exact scan, or IVF probing when nprobe is given. Returns (hits, rows scanned)."""
        if nprobe is None:
            rows = np.arange(len(self.chunks))
        else:
            cells = np.argsort(-(self.ivf["centroids"] @ qv))[:nprobe]
            rows = np.concatenate([self.ivf["lists"] for c in cells])
        rows = rows[self.alive[rows]]                              # skip tombstoned rows
        order = np.argsort(-(self.V[rows] @ qv), kind="stable")[:k]
        return [(int(rows[i]), float(self.V[rows[i]] @ qv)) for i in order], len(rows)

    def extend(self, chunks, cache):
        """Append chunks. An existing IVF structure keeps its centroids and only assigns new rows."""
        new = Index(self.embedder, self.cfg, chunks, cache)
        self.embed_calls += new.embed_calls
        base = len(self.chunks)
        self.chunks += chunks
        self.V = np.vstack([self.V, new.V])
        self.alive = np.concatenate([self.alive, new.alive])
        if self.ivf is not None:
            assign = np.argmax(new.V @ self.ivf["centroids"].T, axis=1)
            for r, j in enumerate(assign):
                self.ivf["lists"][j] = np.append(self.ivf["lists"][j], base + r)

    def build_ivf(self, nlist, seed=0, iters=15):
        rng = np.random.default_rng(seed)
        C = self.V[rng.choice(len(self.V), nlist, replace=False)].copy()
        for _ in range(iters):
            assign = np.argmax(self.V @ C.T, axis=1)
            for j in range(nlist):
                m = self.V[assign == j]
                if len(m):
                    c = m.mean(axis=0); C[j] = c / (np.linalg.norm(c) or 1)
        assign = np.argmax(self.V @ C.T, axis=1)
        self.ivf = {"centroids": C, "lists": [np.where(assign == j)[0] for j in range(nlist)],
                    "params": {"nlist": nlist, "seed": seed, "iters": iters}}

def build_index(store, cfg, embedder, cache):
    chunks = []
    for (doc_id, ver), text in sorted(store.items()):
        chunks += chunk_document(doc_id, ver, text, cfg)
    return Index(embedder, cfg, chunks, cache)

The embedding recipe is part of the index

A vector is meaningful only relative to the model that produced it, as the embeddings article explains. In practice the unit that has to match between index and query is a recipe: the model and its version, the dimension, the way the embedded text is built (heading path, prefixes such as “query:” and “passage:” where a model requires them), and normalization. The index stores the recipe’s identifier and the query path checks it.

The failure that the check prevents does not raise an error. The following script queries an index built with one stand-in model using query vectors from a second stand-in model of the same dimension, then shows the guard, and then compares a candidate index built with the second model against the current one before any switch.

from idx import *

store, params = make_corpus(300)
gold = make_gold(params, store, 120)
cfg, cache = ChunkConfig(20, False, True), {}
ix_a = build_index(store, cfg, HashEmbedder(128, "a"), cache)
print("initial build: embed calls", ix_a.embed_calls)

# 1. two embedding models with the same dimension
other = HashEmbedder(128, "b")
ok = bad = 0
for g in gold:
    rows = [r for r, _ in ix_a.search(ix_a.query_vector(g["query"]), 5)[0]]
    ok += any(covers(ix_a.chunks[r], g, store) for r in rows)
    rows = [r for r, _ in ix_a.search(other.embed(g["query"]), 5)[0]]        # no guard: query from the wrong model
    bad += any(covers(ix_a.chunks[r], g, store) for r in rows)
print(f"hit@5 with the index's model: {ok / len(gold):.3f}; with the other model's query vectors: {bad / len(gold):.3f}")
try:
    ix_a.query_vector("x", other)
except ValueError as e:
    print("guard:", e)

# 2. candidate index built with the other model, compared before switching
ix_b = build_index(store, cfg, other, {})
overlap = []
for g in gold:
    a = {r for r, _ in ix_a.search(ix_a.query_vector(g["query"]), 5)[0]}
    b = {r for r, _ in ix_b.search(ix_b.query_vector(g["query"]), 5)[0]}
    overlap.append(len(a & b) / 5)
print(f"top-5 overlap between the two indexes: {sum(overlap) / len(overlap):.3f}; "
      f"gold hit@5: old {evaluate(ix_a, gold, store)[0]['hit@5']:.3f}, new {evaluate(ix_b, gold, store)[0]['hit@5']:.3f}")

# 3. content-key reuse when one document changes
d = sorted(params)[5]
for label, new_text in [("one sentence edited", store[(d, "v1")].replace("within", "within roughly", 1)),
                        ("one sentence inserted", store[(d, "v1")].replace("## Returns\n", "## Returns\nReturns are free for members. ", 1))]:
    s2 = {k: v for k, v in store.items() if k[0] != d}
    s2[(d, "v2")] = new_text
    ix2 = build_index(s2, cfg, HashEmbedder(128, "a"), cache)
    print(f"{label}: {len(ix2.chunks)} chunks, {ix2.embed_calls} embedding call(s)")
print("spans before:", [(c.start, c.end) for c in ix_a.chunks if c.doc_id == d])
print("spans after: ", [(c.start, c.end) for c in ix2.chunks if c.doc_id == d])

initial build: embed calls 1500
hit@5 with the index's model: 0.925; with the other model's query vectors: 0.000
guard: query model hash-128-b != index model hash-128-a
top-5 overlap between the two indexes: 0.152; gold hit@5: old 0.925, new 0.625
one sentence edited: 1500 chunks, 1 embedding call(s)
one sentence inserted: 1500 chunks, 1 embedding call(s)
spans before: [(34, 125), (126, 168), (181, 289), (302, 389), (390, 423)]
spans after:  [(34, 155), (156, 198), (211, 319), (332, 419), (420, 453)]

Hit@5 falls from 0.925 to 0.000 with the wrong model’s query vectors, and nothing fails: the dimensions match, so the matrix product runs and returns confident-looking scores. The guard turns it into an exception. The candidate index shows the other half of a migration. The two stand-ins differ only in the salt of their hash, yet their top-5 sets overlap by 0.152 and the gold hit@5 drops from 0.925 to 0.625, because a different set of words collides in the 128 buckets. A real model change differs in far more ways, and a vendor’s newer version of a model is a different recipe; the point is the procedure. Build the candidate beside the current index, compare both on the gold set and on neighbor overlap, and switch only when the candidate is at least as good on the slices that matter. The switch itself is the alias swap described in the document pipeline article, and the old index is kept for rollback.

Reusing vectors when documents change

The cache key is the model identifier plus the embedded text, the content key of the pipeline article, and it contains no offsets and no version. The last part of the script above edits one sentence of one guide, then inserts a sentence into another version of the same guide. In each case 1,500 chunks are produced and one embedding call is made. The unchanged chunks keep their vectors, and their records are rewritten with the new version identifier and new offsets. The printed spans show the effect of the insertion: the first chunk grows to three sentences and every later chunk of the guide moves by 30 characters, but only the chunk whose text changed needed a vector.

This works because the chunker is structure-aware. With fixed windows, an insertion shifts every later boundary and every later chunk gets new text, as the pipeline article shows with a count of re-embedding calls. Reuse does not remove the need to rewrite records: the span of a chunk is a fact about one version and has to be recomputed for the new one.

Exact and approximate search

An exact scan scores every chunk. For N chunks of dimension d that is N·d multiply-adds per query, which for the 1,500 chunks here is 192,000. The cost grows linearly with N, and beyond some corpus size and latency budget an approximate nearest-neighbor index replaces it. Two families are common. Graph indexes such as HNSW, introduced by Malkov and Yashunin, build a multi-layer proximity graph and are reported to scale logarithmically in search; the pgvector documentation characterizes HNSW as having a better speed-recall tradeoff than IVFFlat at the price of slower builds and more memory, with no training step. Inverted-file (IVF) indexes cluster the vectors and search only the clusters nearest to the query; the Faiss documentation describes IVF as the basis of non-exhaustive search in large collections.

The IVF structure in the code above is the simplest version. build_ivf runs k-means on the vectors to get nlist centroids and stores, for each centroid, the rows nearest to it. A query is compared with the centroids, the nprobe nearest cells are selected, and only their rows are scored. If cells were balanced, the work would be about nlist·d for the centroids plus N·(nprobe/nlist)·d for the rows. Approximation costs recall, and recall against the exact result is a measurement separate from the quality of the embedding. Define it for a query set \(Q\) as

\[ \mathrm{ANNRecall@}k=\frac{1}{|Q|}\sum_{q\in Q}\frac{\bigl|\{r\in\mathrm{approx}_k(q):\ s(q,r)\ge s^{\mathrm{exact}}_k(q)\}\bigr|}{k}, \]

where \(\mathrm{approx}_k(q)\) is the approximate top-\(k\), \(s(q,r)\) is the score of row \(r\) and \(s^{\mathrm{exact}}_k(q)\) is the score of the exact \(k\)-th result. Counting an approximate result as correct when it scores at least as high as the exact \(k\)-th result makes ties harmless; comparing sets of row numbers would penalize an index for returning a different but equally scored row. The value is 1 when every approximate result is as good as an exact one, which always holds at nprobe = nlist.

def ann_recall(index, queries, nprobe, k=5):
    """Share of approximate results that score at least as high as the exact k-th result."""
    ok = scanned = 0
    for q in queries:
        qv = index.query_vector(q)
        exact, _ = index.search(qv, k)
        floor = exact[-1][1] - 1e-9                      # ties at the k-th score count as exact
        approx, n = index.search(qv, k, nprobe)
        ok += sum(s >= floor for _, s in approx) / k
        scanned += n
    return ok / len(queries), scanned / len(queries)

The script below measures it on 100 queries over the 1,500-chunk index with 40 cells. Then it appends a second family of documents without retraining, and rebuilds the cells. The second family is a set of 140 generated safety notices with different vocabulary (batteries, chargers, cable exchanges); queries about them are the “new family”.

import random
from idx import *

store, params = make_corpus(300)
cfg, cache = ChunkConfig(20, False, True), {}
ix = build_index(store, cfg, HashEmbedder(128, "a"), cache)
queries = [g["query"] for g in make_gold(params, store, 100, seed=3)]

print("nprobe  recall vs exact  rows scanned (of", len(ix.chunks), ")")
ix.build_ivf(nlist=40, seed=0)
sizes = sorted(len(l) for l in ix.ivf["lists"])
print("list sizes: smallest", sizes[0], "median", sizes[len(sizes) // 2], "largest", sizes[-1])
for nprobe in (1, 2, 4, 8, 40):
    r, s = ann_recall(ix, queries, nprobe)
    print(f"{nprobe:6d}  {r:15.3f}  {s:12.1f}")

# a second product family with different vocabulary arrives
def make_notice(name, rng):
    n = rng.randrange(5, 30)
    return (f"# {name} Safety Notice\n## Hazard\nThe battery may overheat when charged with a damaged cable. "
            f"Stop using the charger if the casing feels hot.\n## Remedy\nContact the service desk to arrange "
            f"a free cable exchange within {n} days.\n")
rng = random.Random(5)
notices = {}
for _ in range(150):
    name = "".join(rng.choice(SYL) for _ in range(3)).capitalize() + "x"
    notices[(name.lower(), "v1")] = make_notice(name, rng)
new_queries = [f"What should I do if the {d.capitalize()} charger overheats?" for d, _ in list(notices)[:60]]
old_queries = queries[:60]

new_chunks = []
for (d, v), t in sorted(notices.items()):
    new_chunks += chunk_document(d, v, t, cfg)
ix.extend(new_chunks, cache)                      # centroids unchanged
print("\nafter appending", len(notices), "documents without retraining")
for nprobe in (2, 4, 8):
    print(f"nprobe={nprobe}: recall vs exact, original family {ann_recall(ix, old_queries, nprobe)[0]:.3f}, "
          f"new family {ann_recall(ix, new_queries, nprobe)[0]:.3f}")
ix.build_ivf(nlist=40, seed=0)
print("after rebuilding the cells")
for nprobe in (2, 4, 8):
    print(f"nprobe={nprobe}: recall vs exact, original family {ann_recall(ix, old_queries, nprobe)[0]:.3f}, "
          f"new family {ann_recall(ix, new_queries, nprobe)[0]:.3f}")

nprobe  recall vs exact  rows scanned (of 1500 )
list sizes: smallest 1 median 14 largest 289
     1            0.514          69.5
     2            0.620          90.4
     4            0.892         227.2
     8            0.952         324.6
    40            1.000        1500.0

after appending 140 documents without retraining
nprobe=2: recall vs exact, original family 0.597, new family 0.027
nprobe=4: recall vs exact, original family 0.860, new family 0.027
nprobe=8: recall vs exact, original family 0.920, new family 0.090
after rebuilding the cells
nprobe=2: recall vs exact, original family 0.870, new family 1.000
nprobe=4: recall vs exact, original family 0.940, new family 1.000
nprobe=8: recall vs exact, original family 0.947, new family 1.000

At nprobe = 4, recall is 0.892 while scanning 227 rows, about 15% of the index. The work is roughly 40·128 + 227·128 ≈ 34,000 multiply-adds against 192,000 for the exact scan, a 5.6-fold reduction. At nprobe = 1 the cost is lower and recall is 0.514. The balanced estimate for nprobe = 4 is 1,500·4/40 = 150 rows, but 227 were scanned because the cells are far from balanced: the median cell holds 14 chunks and the largest 289. The largest cell holds 289 copies of the same refund sentence, one per guide, which differ only in the product name in their heading. The other large cells each hold one section type. Queries about those sections land in the large cells more often than average.

Updating an approximate index

An index that changes needs a place for new rows. Manning, Raghavan and Schütze describe the standard pattern for inverted indexes: a large main index, a small auxiliary index for new documents that is searched alongside it with merged results, an invalidation bit vector for deletions, and a periodic merge when the auxiliary index grows. The code above has the same shape. extend appends rows, and for IVF it assigns each new row to the nearest existing centroid. The alive vector is the invalidation bit vector, and the search skips rows that are marked dead. Marking a row dead is a query-time filter and not physical removal; how a particular engine reclaims the space, and what that means for deletion obligations, is covered in the document pipeline article.

The second half of the output shows what appending without retraining does. After 140 notice documents are added, recall for the original family stays near its earlier level (0.860 at nprobe = 4), but recall for queries about the new family is 0.027 and is only 0.090 at nprobe = 8. The centroids were fitted to guide vocabulary. A short query and the long chunk that answers it are each assigned to whichever old centroid happens to be nearest, and they often land in different cells, so the probed cells do not contain the answer. After the cells are rebuilt on all the rows, recall for the new family is 1.000 at every setting shown. The size of the effect belongs to this toy, whose two families share almost no words. The mechanism is general: cell centroids encode the data distribution at training time. The pgvector README advises creating an IVFFlat index only after the table holds data, because the timing affects recall.

Three signals indicate that an IVF structure needs rebuilding: ANN recall on a fixed set of probe queries falls below its floor, the largest cells grow much faster than the average, and the fraction of rows added since training passes a threshold. The first needs the exact scan as reference, which is affordable on a sample of queries. A rebuild is a new index version and goes through the same checks and alias swap as any other change. Graph indexes have no centroid training step, but the same measurement of approximate against exact results applies to them.

Provenance: what a hit must carry

The W3C PROV overview defines provenance as information about the entities, activities and people involved in producing a piece of data, which can be used to assess its quality, reliability or trustworthiness. For a retrieval hit the entity is a chunk of a document version, the activities are the parse, chunk and embed steps with their configurations, and the question a reviewer asks is concrete: what text is this, where exactly did it come from, and which index returned it?

A hit that carries only chunk text answers none of them. The evidence record below carries the ownership key, the span, the heading path, a hash of the span’s text, the rank and score, and the version of the index that produced the hit. The resolve function takes a record and the store of source versions, and returns the source text if, and only if, it is exactly what was retrieved.

def evidence(index, row, score, rank, index_version):
    c = index.chunks[row]
    return {"evidence_id": f"{c.doc_id}:{c.version_id}:{c.ordinal}", "doc_id": c.doc_id,
            "version_id": c.version_id, "span": , "heading_path": c.heading_path,
            "span_hash": h(c.text), "rank": rank, "score": round(score, 4),
            "index_version": index_version}

def resolve(ev, store):
    src = store.get((ev["doc_id"], ev["version_id"]))
    if src is None:
        return "version_missing", None
    a, b = ev["span"]
    if h(src[a:b]) != ev["span_hash"]:
        return "span_mismatch", None
    return "ok", src[a:b]

A second helper checks the invariant for a whole index against a store of source versions and lists the chunks that violate it.

def span_errors(index, store):
    return [(c.doc_id, c.ordinal) for c in index.chunks
            if store.get((c.doc_id, c.version_id), "") != c.text]

The script retrieves the top three evidence records for one question, resolves the first, and then breaks the source in two ways: an in-place edit that leaves the version identifier unchanged, and a removal of the version from the store.

import json
from idx import *

store, params = make_corpus(300)
gold = make_gold(params, store, 120)
ix = build_index(store, ChunkConfig(20, False, True), HashEmbedder(128, "a"), {})
iv = ix.version(store)

g = gold[0]
hits, _ = ix.search(ix.query_vector(g["query"]), 3)
print(g["query"], "| answer:", g["anchor"])
records = [evidence(ix, row, score, rank, iv) for rank, (row, score) in enumerate(hits, 1)]
for r in records:
    print(json.dumps(r))
print(resolve(records[0], store))

# the source is edited in place: same version id, different text
edited = dict(store)
d = records[0]["doc_id"]
edited[(d, "v1")] = store[(d, "v1")].replace("Standard shipping takes", "Standard shipping usually takes")
print("after in-place edit:", resolve(records[0], edited)[0], "| spans broken in index:", len(span_errors(ix, edited)))
print("version removed from raw store:", resolve(records[0], {})[0])

How many days does standard shipping take for Tunelo? | answer: Standard shipping takes 14 days.
{"evidence_id": "tunelo:v1:3", "doc_id": "tunelo", "version_id": "v1", "span": [302, 389], "heading_path": "Tunelo Service Guide > Shipping", "span_hash": "6f2846174c4b", "rank": 1, "score": 0.6708, "index_version": "2b2e35563c3d5b4c"}
{"evidence_id": "dokalo:v1:3", "doc_id": "dokalo", "version_id": "v1", "span": [302, 388], "heading_path": "Dokalo Service Guide > Shipping", "span_hash": "52c82d58629d", "rank": 2, "score": 0.6674, "index_version": "2b2e35563c3d5b4c"}
{"evidence_id": "fufuze:v1:3", "doc_id": "fufuze", "version_id": "v1", "span": [302, 389], "heading_path": "Fufuze Service Guide > Shipping", "span_hash": "6f2846174c4b", "rank": 3, "score": 0.6325, "index_version": "2b2e35563c3d5b4c"}
('ok', 'Standard shipping takes 14 days. A tracking link is sent when the parcel is dispatched.')
after in-place edit: span_mismatch | spans broken in index: 2
version removed from raw store: version_missing

The first record resolves to the shipping sentences of the guide, with a span of 87 characters. The two lower-ranked records are other guides’ shipping chunks, which is the identifier weakness seen earlier. After an in-place edit the span hash no longer matches and resolve reports span_mismatch; the same edit leaves two chunks of the index with spans that no longer equal their text. The version identifier would not have caught this, because in-place edits are exactly the case in which it did not change. In a correct pipeline an edit produces a new version whose content hash differs, so the check is a second line of defense that costs one hash. With the version removed from the store, resolution returns version_missing and returns no text, and fails closed.

The last case has a consequence for retention. A record can be resolved only while the parsed text of its version is kept. When a source is erased under the rules in the pipeline article, the evidence logged before the erasure becomes unresolvable, which is the intended outcome. Decide what audit logs keep. Identifiers, spans and hashes allow an investigation to confirm what was retrieved when the text is available elsewhere, without storing the text itself; the logging of prompts and retrieved content is treated in the article on privacy, data leakage, and output controls.

The record establishes where a chunk came from. It does not establish that the chunk supports a claim made in an answer, that the reader is permitted to see it, or that it is the best chunk for the question. Whether an answer is supported by its evidence, and what to do when evidence is missing or conflicting, is the subject of the article on grounded answers and abstention. Per-reader authorization is checked at query time, as in the article on preserving access controls in retrieval data. Display citation numbers are assigned after the final evidence is selected, but the answer’s log keeps the stable evidence_id beside each number.

Versioning the index

A deployed index is a build output, and queries are answered by one specific build. Naming the build lets a request log say which index answered it, lets an evaluation result say which index it measured, and lets a rollback target a known state. The version of an index is a hash of its manifest, which lists every input that determines its content: the embedding recipe, the chunk configuration, and the list of document versions with their content hashes. The version method in idx.py hashes that manifest as JSON.

Query-time settings such as nprobe and k are not in the manifest, since they do not change what is stored. They are logged with each query. Settings that change what is stored, such as the number of cells and the k-means seed, are in the manifest once IVF cells exist.

diff_manifest compares two manifests and names what changed, and content_keys lists the content keys of an index so that two indexes can be compared chunk by chunk.

def diff_manifest(m1, m2):
    out = []
    if m1["embedder"] != m2["embedder"]:
        out.append(f"embedder {m1['embedder']} -> {m2['embedder']}")
    c1, c2 = m1["chunk_config"], m2["chunk_config"]
    if c1 != c2:
        out.append("chunk_config " + ", ".join(f"{k}: {c1[k]} -> {c2[k]}" for k in c1 if c1[k] != c2[k]))
    if m1["ivf"] != m2["ivf"]:
        out.append(f"ivf {m1['ivf']} -> {m2['ivf']}")
    d1 = {(d, v) for d, v, _ in m1["documents"]}
    d2 = {(d, v) for d, v, _ in m2["documents"]}
    out.append(f"documents: {len(d2 - d1)} versions added, {len(d1 - d2)} versions removed")
    return out

def content_keys(index):
    return {h(index.embedder.ident + "|" + c.embed_text) for c in index.chunks}

The first script builds the index twice from the same inputs, once with a different chunk budget, once with IVF cells, and then applies a change set.

from idx import *

store, params = make_corpus(300)
emb, cache = HashEmbedder(128, "a"), {}
cfg_a, cfg_b = ChunkConfig(20, False, True), ChunkConfig(60, False, True)
ix_a = build_index(store, cfg_a, emb, cache)
print("version A:", ix_a.version(store), "| rebuilt from the same inputs:", build_index(store, cfg_a, emb, {}).version(store))
ix_b = build_index(store, cfg_b, emb, cache)
print("version B:", ix_b.version(store), diff_manifest(ix_a.manifest(store), ix_b.manifest(store)))
ix_ivf = build_index(store, cfg_a, emb, cache)
ix_ivf.build_ivf(nlist=40, seed=0)
print("A with IVF cells:", ix_ivf.version(store), diff_manifest(ix_a.manifest(store), ix_ivf.manifest(store)))

# three guides get a new version, one is withdrawn
ids = sorted(params)
edited = dict(store)
for d in ids[:3]:
    edited[(d, "v2")] = edited.pop((d, "v1")).replace("Items may be returned within", "Items may be returned within roughly")
del edited[(ids[3], "v1")]
ix_c = build_index(edited, cfg_a, emb, cache)
print("\nafter 3 edits and 1 withdrawal:", ix_c.version(edited), diff_manifest(ix_a.manifest(store), ix_c.manifest(edited)))
ka, kc = content_keys(ix_a), content_keys(ix_c)
print(f"chunks kept {len(ka & kc)}, removed {len(ka - kc)}, added {len(kc - ka)}; embedding calls {ix_c.embed_calls}")

version A: 2b2e35563c3d5b4c | rebuilt from the same inputs: 2b2e35563c3d5b4c
version B: 03b67ba5f1300f89 ['chunk_config max_words: 20 -> 60', 'documents: 0 versions added, 0 versions removed']
A with IVF cells: 6da0582259c53d1c ["ivf None -> {'nlist': 40, 'seed': 0, 'iters': 15}", 'documents: 0 versions added, 0 versions removed']

after 3 edits and 1 withdrawal: 6439e2239f25fe31 ['documents: 3 versions added, 4 versions removed']
chunks kept 1492, removed 8, added 3; embedding calls 3

Rebuilding from identical inputs gives the same version, and changing the chunk budget or adding IVF cells gives a different one, with diff_manifest naming the change. Equal versions mean equal inputs. They do not by themselves guarantee equal vector bytes. A hosted embedding service may not return bit-identical vectors on a second call, and some graph builders depend on insertion order or random seeds, so the seeds belong in the manifest and equality of behavior should be checked with the neighbor overlap and gold metrics rather than assumed.

The update example shows the diff that a review should read before an alias switch. Three guides get a new version and one is withdrawn; the manifest reports three versions added and four removed. At chunk level, 1,492 chunks are kept, 8 are removed and 3 are added, and the build makes three embedding calls. That is exactly what the change set predicts: the withdrawn guide’s five chunks plus one edited chunk in each of three guides are removed (eight), and the three edited chunks are added. A diff much larger than the change set predicts points to an unintended change upstream, such as a parser upgrade that altered the text of many documents, and should stop the release.

Each retrieval is logged with the index version, the query-time settings and the evidence records. The previous index version is kept until the rollback window ends, and evaluation runs are pinned to a version. A metric that does not say which index produced it cannot be reproduced, which is the requirement of reproducible evaluation applied to a component whose content changes whenever documents do.

Retrieval tests

An index fails in different ways, and each failure is caught by a different kind of test. A single quality number catches only one of them.

Test	Question it answers	Catches	Runs
Structural invariants	Does every span equal its source slice; are keys unique; does the recipe match?	Chunker or parser changes that break provenance	Every build, as a gate
Lifecycle canaries	Are only current versions alive; does a withdrawn document return nothing?	Failed cleanup, resurrected documents	After every refresh
Gold-set quality	Do the questions retrieve their answer sentences, overall and by slice?	Quality regressions from chunking, recipe or data changes	Every build and on a schedule
ANN fidelity	How close is approximate search to exact search on probe queries?	Stale cells, wrong nprobe	Every build and after appends
Change comparison	Is the candidate index better or worse than the current one, query by query?	Regressions hidden by averages	Before an alias switch

Gold sets that survive re-chunking

A gold item for a chunk index has to name the evidence in a way that survives a change of chunking. If it stores a chunk identifier, any change to the chunk size invalidates it. The gold items here store the document and the answer sentence, and a chunk counts as relevant when its span covers the sentence in the version it belongs to. The sets of questions, their sampling, the split between tuning and test data, and the thresholds follow the article on reproducible test sets; the metrics are defined in the embeddings article. The script below compares the two labeling methods. It labels every question with the chunk of index A (20-word chunks) that covers the answer, then checks how many of those labels still point at the answer in index B, which uses a 60-word budget.

from collections import Counter
from math import comb
from idx import *

store, params = make_corpus(300)
gold = make_gold(params, store, 120)
emb, cache = HashEmbedder(128, "a"), {}
ix_a = build_index(store, ChunkConfig(20, False, True), emb, cache)    # 20-word chunks
ix_b = build_index(store, ChunkConfig(60, False, True), emb, cache)    # 60-word chunks

# labels that name a chunk versus labels that name the answer sentence
by_key = {(c.doc_id, c.version_id, c.ordinal): c for c in ix_b.chunks}
labels = [(g, next((c.doc_id, c.version_id, c.ordinal) for c in ix_a.chunks if covers(c, g, store))) for g in gold]
still = sum(key in by_key and covers(by_key[key], g, store) for g, key in labels)
print(f"chunk-id labels made on index A that still point at the answer in index B: {still} of {len(gold)}")
print(f"anchor labels with a covering chunk in index B: {sum(any(covers(c, g, store) for c in ix_b.chunks) for g in gold)} of {len(gold)}")

# paired comparison on the same questions
(ma, ra), (mb, rb) = evaluate(ix_a, gold, store), evaluate(ix_b, gold, store)
print("\nA:", {k: round(v, 3) for k, v in ma.items()})
print("B:", {k: round(v, 3) for k, v in mb.items()})
def sign_test(only_a, only_b):
    n = len(only_a) + len(only_b)
    tail = sum(comb(n, i) for i in range(max(len(only_a), len(only_b)), n + 1))
    return n, min(1, 2 * tail / 2 ** n)

for label, ok in (("hit@5", lambda r: r is not None and r <= 5), ("hit@1", lambda r: r == 1)):
    only_a = [g["kind"] for g, a, b in zip(gold, ra, rb) if ok(a) and not ok(b)]
    only_b = [g["kind"] for g, a, b in zip(gold, ra, rb) if ok(b) and not ok(a)]
    n, p = sign_test(only_a, only_b)
    print(f"{label}: only in A {len(only_a)} {dict(Counter(only_a))}, only in B {len(only_b)} {dict(Counter(only_b))}; "
          f"two-sided sign test on {n} discordant questions: p = {p:.3f}")

chunk-id labels made on index A that still point at the answer in index B: 47 of 120
anchor labels with a covering chunk in index B: 120 of 120

A: {'hit@1': 0.442, 'hit@5': 0.925, 'mrr@10': 0.64}
B: {'hit@1': 0.258, 'hit@5': 0.9, 'mrr@10': 0.512}
hit@5: only in A 7 {'ret': 2, 'ship': 4, 'war': 1}, only in B 4 {'ret': 2, 'war': 2}; two-sided sign test on 11 discordant questions: p = 0.549
hit@1: only in A 25 {'ship': 19, 'ret': 6}, only in B 3 {'ship': 1, 'war': 2}; two-sided sign test on 28 discordant questions: p = 0.000

Only 47 of the 120 chunk-identifier labels still point at the answer in index B, because the chunk with ordinal 3 means something different in each configuration. All 120 anchor labels find a covering chunk. An anchor has its own failure mode: when the guide changes and the sentence no longer occurs in the new version, the item is stale, and the evaluation can see it because the search for the anchor fails. A gold set needs a maintenance step that flags such items instead of counting them as retrieval failures.

With 120 questions, a hit@5 of 0.925 has a standard error of about \(\sqrt{0.925\cdot0.075/120}\approx0.024\), so an interval of roughly plus or minus 0.05. That is wider than the difference between the first two configurations in the grid above.

Comparing two index versions

The same script compares the indexes on the same questions, which allows a paired comparison: count the questions on which one index gets the answer into the top five and the other does not.

Seven questions are answered in the top five only by A and four only by B. Under the hypothesis that the two are equally good, each of the 11 discordant questions is equally likely to go either way, and the two-sided sign test, which the glossary entry on the McNemar test works through with an example, gives p = 0.55, so hit@5 does not separate the two indexes. On hit@1 the picture is different: 25 questions are answered at rank 1 only by A and 3 only by B, a difference far beyond what that hypothesis allows (p below 0.001), and 19 of the 25 are shipping questions. A 60-word chunk holds the whole shipping section, including the free-shipping sentence, while a 20-word chunk holds only the delivery-time and tracking sentences; the extra sentence adds words that the delivery-time question does not share. The two indexes retrieve about the same evidence and rank it differently, and the difference is concentrated in one kind of question. A bootstrap over questions, as in the glossary entry on the bootstrap, gives an interval for the difference in a metric such as MRR. The discordant questions are the more useful output: a short list that a person can read, each pointing to a specific chunk to inspect.

Lifecycle tests and what quality metrics miss

The suite is a function that returns a named result for each check. The structural check uses span_errors, the lifecycle checks use the alive flags and a canary question about a withdrawn guide, and the quality check reuses evaluate.

def run_tests(index, store, current, deleted, gold, floor=0.85):
    res = {}
    res["spans equal source slice"] = not span_errors(index, store)
    res["only current versions alive"] = all(current.get(c.doc_id) == c.version_id
                                            for c, alive in zip(index.chunks, index.alive) if alive)
    leaks = 0
    for d in deleted:                                     # canary query about a withdrawn document
        hits, _ = index.search(index.query_vector(f"What is the return window for {d.capitalize()}?"), 10)
        leaks += sum(index.chunks[r].doc_id == d for r, _ in hits)
    res["deleted-document canaries return nothing"] = leaks == 0
    m, _ = evaluate(index, gold, store)
    res[f"gold hit@5 >= {floor} (got {m['hit@5']:.3f})"] = m["hit@5"] >= floor
    return res

The next script runs it on two indexes built from the same updated corpus. The first is the correct result of the update. The second simulates a refresh whose cleanup step failed: the old versions of the three edited guides and the withdrawn guide were never removed. Gold items about the changed or withdrawn guides are retired for this run, since their answers changed.

from idx import *

store, params = make_corpus(300)
gold = make_gold(params, store, 120)
emb, cache = HashEmbedder(128, "a"), {}
cfg = ChunkConfig(20, False, True)
ids = sorted(params)

edited = dict(store)                                    # 3 guides get v2, one is withdrawn
for d in ids[:3]:
    edited[(d, "v2")] = edited.pop((d, "v1")).replace("Items may be returned within", "Items may be returned within roughly")
gone = ids[3]
del edited[(gone, "v1")]
current = {d: "v1" for d in ids}
current.update({d: "v2" for d in ids[:3]})
del current[gone]
gold_now = [g for g in gold if g["doc_id"] not in ids[:4]]    # items about changed or withdrawn guides are retired

healthy = build_index(edited, cfg, emb, cache)
print("healthy index after the update")
for name, ok in run_tests(healthy, edited, current, [gone], gold_now).items():
    print("  ", "PASS" if ok else "FAIL", name)

leftover = {(d, "v1"): store[(d, "v1")] for d in ids[:4]}   # the cleanup step never removed the old versions
broken_store = {**edited, **leftover}
broken = build_index(broken_store, cfg, emb, cache)
print("index after a failed cleanup step")
for name, ok in run_tests(broken, broken_store, current, [gone], gold_now).items():
    print("  ", "PASS" if ok else "FAIL", name)

healthy index after the update
   PASS spans equal source slice
   PASS only current versions alive
   PASS deleted-document canaries return nothing
   PASS gold hit@5 >= 0.85 (got 0.923)
index after a failed cleanup step
   PASS spans equal source slice
   FAIL only current versions alive
   FAIL deleted-document canaries return nothing
   PASS gold hit@5 >= 0.85 (got 0.923)

Both indexes pass the structural check, since every chunk still equals its source slice. The gold-set test passes on both with a hit@5 of 0.923, because the current chunk for each question still ranks high, and the stale chunks compete only for some of the lower positions. Only the two lifecycle tests, which ask a different question, fail on the broken index. A quality metric measures whether the right evidence is among the results. It does not measure whether wrong evidence is absent, so a retrieval suite needs both kinds of test.

The floor of 0.85 in the gold-set test is an example. A real threshold comes from the task’s acceptance criteria, as described in the article on framing an AI application. These tests do not tell whether the model’s answer is correct or faithful to the evidence; separating retrieval metrics from answer metrics is the subject of evaluating retrieval and answers separately. They also do not cover questions that are absent from the gold set, or facts that the corpus never contained.

How much of this a small system needs

A small corpus that changes rarely can use an exact scan, rebuild the whole index on each release, and keep only the pieces that cost almost nothing: the span invariant, a manifest hash, anchor-based gold items and two lifecycle canaries. That is a few dozen lines on top of a chunker. The ANN structure, the update path and the migration procedure are justified when corpus size, latency or freshness requirements rule out the simple design, and each one brings a measurement with it: ANN recall against exact search, the diff between versions, and a paired comparison before a switch.

References

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401)
• W3C, PROV-Overview: An Overview of the PROV Family of Documents
• Manning, Raghavan, and Schütze, Introduction to Information Retrieval, section on dynamic indexing
• Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (arXiv:1603.09320)
• Faiss documentation
• pgvector README (IVFFlat and HNSW indexes)
• Anthropic, Introducing Contextual Retrieval