Hybrid Retrieval and Reranking

Consider a hypothetical help center for a subscription product, split into chunks and indexed. Three users arrive in the same minute. The first types what does ERR-4029 mean, and exactly one chunk contains that code. The second types get my money back on a monthly subscription; the right chunk says “refund”, never “money back”, and it is the monthly-plan version of that policy, not the annual one. The third has just asked how refunds work and now types what about monthly plans?, a question with no topic word in it at all. A single search call, however good, serves these three differently: the first needs exact token matching, the second needs synonym knowledge plus a qualifier, and the third needs the conversation before it can be searched.

The stage between the index and the generator has to handle all three. How should it be built? It has to run more than one retriever and combine their lists, reorder a candidate list with a more expensive scorer, rewrite a query before it is searched, and measure what each addition costs and which kind of query it helped. The starting point is the index described in the preceding article on building a retrieval index with traceable evidence: chunks with stable identifiers, provenance, and an index version. The mechanics of BM25, embedding training, similarity, reciprocal rank fusion at the level of a formula, nDCG, and the paired bootstrap are in the article on embeddings and retrieval representations and are used here without being rederived. The overview of retrieval-augmented generation places this stage in the whole workflow.

The retrieval stage is a funnel with a recall ceiling

A production retrieval stage is usually several steps in sequence, each seeing fewer documents and spending more per document than the one before it. The steps below are the ones this article covers. Not every system needs all of them.

Step	Input	Output	Cost grows with	Cannot fix
Query transformation	User text and conversation	One or more search strings, optional filters	Number of model calls	Information the user never gave and the history does not contain
Candidate generation (legs)	Search strings	A ranked list per retriever, depth \(n\) each	Number of legs; depth \(n\)	A relevant chunk that no leg ranks within depth \(n\)
Fusion	The ranked lists	One list, truncated to \(M\) candidates	Almost nothing	Chunks outside the lists
Reranking	Top \(M\) candidates and the query	The same \(M\) candidates in a new order	\(M\) times the per-pair cost	A relevant chunk outside the \(M\) candidates
Cut	Reordered list	Top \(k\) chunks for the generator	Prompt tokens	Evidence that was ranked below \(k\)

Two consequences follow from the structure. First, the funnel has a ceiling. If \(R\) is the set of relevant chunks for a query and \(C_M\) is the candidate set after fusion, then whatever comes later can return at most \(|R\cap C_M|\) of them, so

\[ \mathrm{recall@}k\ (\text{final}) \;\le\; \mathrm{recall}(C_M) \;=\; \frac{|R\cap C_M|}{|R|}. \]

A reranker, a larger model, or a better prompt cannot raise the left side above the right side. That is why the experiments below report the recall of the candidate pool separately from the quality of the final ordering: they answer different questions, and a drop in the final score can come from either.

Second, every step is a place where a query can be changed without the reader of the final answer noticing. A rewrite that drops an identifier, a fusion that outvotes the one decisive match, and a reranker that reorders confidently on a truncated passage all produce a plausible-looking list. For that reason each step should leave a record. For each query the system stores the chunk identifier, the index version, the rank and score from each leg, the fused rank, the reranker score if any, and whether the chunk entered the final context. The chunk identifier and index version come from the index; the ranks are what lets someone later answer why a chunk was or was not shown.

A synthetic help center and its stand-ins

The experiments in this article run on a collection written for the purpose, so that every claim about a mechanism can be checked against labels that are known by construction. The collection describes a fictional subscription product. It has twenty topics, such as refunds, data export, invoices, rate limiting, and webhooks. Each topic has one chunk per plan type (annual, monthly, enterprise, trial), which gives 80 chunks that differ mainly in the plan and in one number. Twelve more chunks describe error codes, each tied to one topic. The corpus has 92 chunks in total, and the generator is short enough to show in full.

def build_corpus(seed=7):
    rng = random.Random(seed)
    chunks = []
    for key, (word, unit, _) in TOPICS.items():
        for plan in PLANS:
            n = rng.randint(2, 30)
            filler = " ".join(rng.choice(FILLER) for _ in range(8))
            text = (f"{word.capitalize()} rules for {plan} plans. On {plan} plans the "
                    f"{word} allowance is {n} {unit}. {filler}")
            chunks.append({"id": f"{key}-{plan}", "topic": key, "plan": plan, "text": text})
    codes = rng.sample(range(4000, 4100), 12)
    for code, key in zip(codes, list(TOPICS)):
        word = TOPICS[key][0]
        chunks.append({"id": f"err-{code}", "topic": key, "plan": None,
                       "text": f"Error ERR-{code} is raised when a {word} operation fails. "
                               f"Check the {word} settings and retry."})
    return chunks

Each topic also has query-only synonyms that never appear in the chunks, such as “reimbursement” and “money” for refunds. The 60 test queries come in five slices of twelve, each constructed to exercise one failure mode. Relevance is binary and defined by the generator.

Slice	Example query	Relevant chunks	What it tests
identifier	what does err 4029 mean	The one error chunk	Exact token matching; the code is typed as ERR-4029, err 4029, or err4029
paraphrase	get my tier on a trial subscription	The topic chunk for that plan	A synonym the chunks do not contain, plus a qualifier
exact	seats allowance monthly plans	The topic chunk for that plan	Query words that do appear in the chunk
followup	what about monthly plans? after how does reimbursement work?	The topic chunk for that plan	A question that is only meaningful with the previous turn
twopart	dollars on monthly plans and levy on annual plans	Two chunks, one per part	Two intents, each with its own qualifier, in one query

Three components are stand-ins and have to be understood as such. The lexical leg is a standard BM25 (\(k_1=1.2\), \(b=0.75\)) over tokens produced by an analyzer; the next section shows the analyzer. The dense leg is not a trained model: it is a deterministic function written to behave the way trained encoders typically behave in three respects that matter here. It places a topic word and its synonyms at the same point. It gives plan words only a small weight, so the plan is a weak signal in a vector. And it handles a token it has no concept for, such as an error code, by hashing character pairs into a vector, a crude subword model in which similar-looking codes land near each other. The third, a cross-encoder reranker, is described where it is used.

DIM = 128
CONCEPTS = {}                       # word -> concept key
for key, (word, _, syns) in TOPICS.items():
    for w in [word] + syns:
        CONCEPTS[w] = key
for p in PLANS:
    CONCEPTS[p] = "plan:" + p
_vec_cache = {}

def _hashed(name):
    if name not in _vec_cache:
        r = random.Random(hashlib.md5(name.encode()).hexdigest())
        _vec_cache[name] = [r.gauss(0, 1) for _ in range(DIM)]
    return _vec_cache[name]

def embed(text, w_topic=1.0, w_plan=0.4, w_unknown=0.3):
    """Stand-in for a learned encoder. Known topic words and their synonyms share one
    concept vector, plan words get a small weight, and unknown tokens (identifiers)
    fall back to hashed character bigrams, a crude subword model."""
    v = [0.0] * DIM
    for t in re.split(r"[^a-z0-9]+", text.lower()):
        if not t or t in STOP:
            continue
        c = CONCEPTS.get(t)
        if c:
            w = w_plan if c.startswith("plan:") else w_topic
            parts = [(c, w)]
        else:
            parts = [(t[i:i + 2], w_unknown / max(1, len(t) - 1)) for i in range(max(1, len(t) - 1))]
        for name, w in parts:
            for i, x in enumerate(_hashed(name)):
                v[i] += w * x
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]

Because the encoder and reranker are built to have these properties, the results below show what the mechanisms do when a leg has the properties described. They do not measure how any real embedding model, vector database, or reranker performs, and the numbers should not be carried over to a real collection. What carries over is the procedure: per-slice measurement, a recall ceiling reported next to the final score, and a comparison against simple baselines. All programs were run with Python 3.12 and the standard library only.

Two legs, two kinds of failure

What the lexical leg matches depends on the analyzer

BM25 matches the tokens its analyzer emits, so the analyzer is part of the retriever. The first version below lowercases, splits on every character that is not a letter or digit, and drops stop words. The second also splits where letters meet digits. The reason is visible in how people type identifiers: the same error code arrives as ERR-4029, err 4029, and err4029, while the chunk contains the first form.

STOP = {"the", "a", "an", "of", "on", "for", "to", "is", "are", "my", "i", "do", "how",
        "what", "does", "can", "and", "in", "get", "it", "about", "plans", "plan", "mean"}

def analyze_v1(text):
    """Lowercase, split on anything that is not a letter or digit, drop stop words."""
    return [t for t in re.split(r"[^a-z0-9]+", text.lower()) if t and t not in STOP]

def analyze_v2(text):
    """Same, but also split at letter-digit boundaries so 'err4021' and 'ERR-4021' agree."""
    tokens = []
    for t in re.split(r"[^a-z0-9]+", text.lower()):
        tokens += re.findall(r"[a-z]+|[0-9]+", t)
    return [t for t in tokens if t not in STOP]

The program below runs both analyzers on the three spellings and then scores BM25 on the twelve identifier queries, four of which use the unseparated spelling. It then prints the baseline comparison of the lexical leg, the dense leg, and a plain reciprocal rank fusion of the two, with the nDCG@3 of each slice and of all 60 queries.

from pipeline import *
chunks = build_corpus(); Q = build_queries(chunks)
# analyzer demo
for q in ["what does ERR-4029 mean", "what does err 4029 mean", "what does err4029 mean"]:
    print(f"{q!r:30s} v1={analyze_v1(q)}  v2={analyze_v2(q)}")
for name, an in [("v1", analyze_v1), ("v2", analyze_v2)]:
    R = Retriever(chunks, an)
    per = evaluate(lambda q: R.search(q["text"], "bm25"), [x for x in Q if x["slice"]=="identifier"])
    print("analyzer", name, "identifier nDCG@3", round(sum(x[1] for x in per["identifier"])/12,3))
R = Retriever(chunks)
table([("bm25", evaluate(lambda q: R.search(q["text"],"bm25"), Q)),
       ("dense", evaluate(lambda q: R.search(q["text"],"dense"), Q)),
       ("rrf c=60", evaluate(lambda q: R.search(q["text"],"hybrid"), Q))])

'what does ERR-4029 mean'      v1=['err', '4029']  v2=['err', '4029']
'what does err 4029 mean'      v1=['err', '4029']  v2=['err', '4029']
'what does err4029 mean'       v1=['err4029']  v2=['err', '4029']
analyzer v1 identifier nDCG@3 0.667
analyzer v2 identifier nDCG@3 1.0
                             identifier paraphrase      exact   followup    twopart     all
bm25                              1.000      0.210      1.000      0.208      0.102   0.504
dense                             0.344      1.000      1.000      0.042      0.564   0.590
rrf c=60                          0.480      0.741      1.000      0.147      0.333   0.540

With the first analyzer the unseparated spelling becomes the single token err4029, which occurs in no chunk, so four of the twelve queries return nothing useful and the slice scores 8/12, or 0.667. With the second analyzer all three spellings give the same two tokens and the slice scores 1.000. The fix is cheap but it is a decision about the data: it works because identifiers in this collection are a short letter prefix followed by digits, and a collection with identifiers such as a1b2-c3 needs its own rules. Real systems often add a separate exact-match field for identifiers, so that the code is matched as one unit rather than as pieces, and they add phrase or field boosts for titles. The principle is the same: the lexical leg is as good as the normalization that makes the query and the chunk agree on tokens.

Reading the baseline

The table at the end of the output is the starting point for everything after it. The lexical leg is exact on identifiers (1.000) and on queries that reuse the chunk’s own words (1.000), and it fails on paraphrases (0.210): the synonym matches nothing, so only the plan word contributes, and that word occurs in a fifth of the chunks. The dense leg is the mirror image. It handles paraphrases (1.000) and exact wording (1.000), and it handles identifiers poorly (0.344), because an error code has no concept vector and the character-pair fallback only roughly separates one code from another. This is the same trade-off that the article on embeddings and retrieval representations shows with a trained toy encoder, now seen from the system side: each leg has a kind of query it answers well and a kind it cannot.

Two slices are hard for both legs, for different reasons. On the follow-up slice neither leg has the topic, since the topic is in the previous turn, so no search of this question alone can succeed. On the two-part slice the lexical leg scores 0.102 and the dense leg 0.564. The dense vector for dollars on monthly plans and levy on annual plans is a sum of two topic concepts and two plan concepts, and the sum does not record which plan belongs to which topic. A single vector per query has to compress the pairing away. Fusion cannot restore information that neither leg uses, and the slices where both legs fail point to query transformation, not to better fusion.

Fusing the legs

Fusion has to combine two ranked lists whose scores are on unrelated scales: a BM25 score is a sum of idf-weighted terms with no upper bound, and a cosine similarity lives in a narrow band that depends on the model. Adding raw scores lets whichever leg has the larger numbers decide. Two families avoid that. Rank-based fusion uses only positions. Score-based fusion rescales each leg’s scores to a common range before combining them.

Reciprocal rank fusion and what its constant does

Cormack, Clarke, and Büttcher introduced reciprocal rank fusion (RRF) as an unsupervised way to combine rankings, and report that it beat the individual systems it combined and the Condorcet Fuse baseline on TREC runs. With ranks counted from 1 and an optional weight \(w_j\) per list, the score of document \(d\) is

\[ \mathrm{RRF}(d)=\sum_{j} \frac{w_j}{c+\mathrm{rank}_j(d)}, \]

where the sum runs over the lists that contain \(d\) and \(c\) is a constant. A document missing from a list gets nothing from it. Elasticsearch’s documentation, as one concrete example, calls this constant rank_constant with a default of 60, and uses a separate rank_window_size for how many results each query contributes before fusion; those two numbers are the constant \(c\) and the depth \(n\) of this article.

The constant sets how steeply credit falls with rank. The first block of the next output shows the weight of rank \(r\) relative to rank 1, \((c+1)/(c+r)\), for three constants. With \(c=1\) the document at rank 10 earns 18% of the credit of rank 1, so the top positions dominate. With \(c=60\) it earns 87%, so ranks 1 through 20 count almost the same and the fused score mostly counts how many lists a document appears in. Large \(c\) turns RRF into agreement voting; small \(c\) keeps it closer to the leading positions of each list.

from pipeline import *
chunks = build_corpus(); Q = build_queries(chunks); R = Retriever(chunks)

print("RRF weight of rank r relative to rank 1, for three constants c")
print(f"{'c':>4s}" + "".join(f"{'r='+str(r):>9s}" for r in (2, 5, 10, 20)))
for c in (1, 10, 60):
    print(f"{c:4d}" + "".join(f"{(c + 1) / (c + r):9.3f}" for r in (2, 5, 10, 20)))
print()
rows = []
for name, kw in [("bm25 only", dict(mode="bm25")), ("dense only", dict(mode="dense")),
                 ("rrf c=60 1:1", dict()), ("rrf c=5 1:1", dict(c=5)),
                 ("rrf c=60 2:1 (bm25 heavy)", dict(weights=[2, 1])),
                 ("rrf c=60 1:2 (dense heavy)", dict(weights=[1, 2])),
                 ("min-max 0.5/0.5", dict(fusion="minmax")),
                 ("z-score 0.5/0.5", dict(fusion="zscore"))]:
    rows.append((name, evaluate(lambda q, kw=kw: R.search(q["text"], **kw), Q)))
table(rows)

print("\nrecall of the candidate pool (union of both legs at depth d), all queries")
for d in (3, 5, 10, 20, 40):
    vals = []
    for q in Q:
        lex, den = R.legs(q["text"], d)
        pool = {x for x, _ in lex} | {x for x, _ in den}
        vals.append(len(pool & q["rel"]) / len(q["rel"]))
    print(f"depth {d:2d}: pool recall = {sum(vals)/len(vals):.3f}")

RRF weight of rank r relative to rank 1, for three constants c
   c      r=2      r=5     r=10     r=20
   1    0.667    0.333    0.182    0.095
  10    0.917    0.733    0.550    0.367
  60    0.984    0.938    0.871    0.762

The table of fusion variants

The second block compares fusion methods on the same legs. The rows with rrf use the constant and weights shown, with the weights ordered lexical to dense; min-max and z-score rescale each leg’s returned scores before a weighted sum. The depth of each leg is 20.

                             identifier paraphrase      exact   followup    twopart     all
bm25 only                         1.000      0.210      1.000      0.208      0.102   0.504
dense only                        0.344      1.000      1.000      0.042      0.564   0.590
rrf c=60 1:1                      0.480      0.741      1.000      0.147      0.333   0.540
rrf c=5 1:1                       0.574      0.908      1.000      0.105      0.366   0.591
rrf c=60 2:1 (bm25 heavy)         0.605      0.458      1.000      0.105      0.250   0.484
rrf c=60 1:2 (dense heavy)        0.428      0.969      1.000      0.105      0.461   0.593
min-max 0.5/0.5                   0.803      0.969      1.000      0.105      0.442   0.664
z-score 0.5/0.5                   0.969      0.803      1.000      0.147      0.294   0.643

Three readings stand out. RRF at its usual constant and equal weights scores 0.540 over all queries, below the dense leg alone (0.590) and well below the lexical leg on identifiers (0.480 against 1.000). Fusion made the system worse than one of its parts on a slice where one part was perfect. Weighting does not fix it in one direction: giving the lexical leg twice the weight lifts identifiers to 0.605 and drops paraphrases from 0.741 to 0.458. A single global weight is a compromise between slices that want opposite things. Third, min-max fusion (0.664 overall) keeps identifiers at 0.803 and paraphrases at 0.969, and the z-score variant reaches 0.969 on identifiers and 0.803 on paraphrases.

Why a decisive match can lose a vote

The identifier slice shows what RRF throws away. The next program traces one identifier query that the fused list gets wrong, using the record format described earlier: for the top fused chunks, the rank in each leg and the fused rank.

import json
from pipeline import *
chunks = build_corpus(); Q = build_queries(chunks); R = Retriever(chunks)
ident = [x for x in Q if x["slice"] == "identifier"]
q = next(x for x in ident if R.search(x["text"])[0] not in x["rel"])   # first identifier query RRF gets wrong
print("query:", q["text"], "| relevant:", sorted(q["rel"]))

def trace(query, depth=20, k=3):
    lex, den = R.legs(query, depth)
    lex_rank = {d: r for r, (d, _) in enumerate(lex, 1)}
    den_rank = {d: r for r, (d, _) in enumerate(den, 1)}
    fused = rrf([[d for d, _ in lex], [d for d, _ in den]])
    return [{"chunk": d, "bm25_rank": lex_rank.get(d), "dense_rank": den_rank.get(d),
             "fused_rank": r} for r, d in enumerate(fused[:k], 1)]

for row in trace(q["text"]): print(json.dumps(row))
lex, den = R.legs(q["text"], 20)
rel = list(q["rel"])[0]
print("relevant chunk: bm25 rank", [d for d, _ in lex].index(rel) + 1,
      "dense rank", ([d for d, _ in den].index(rel) + 1) if rel in [d for d, _ in den] else None)
print("bm25 scores of top 4:", [round(s, 2) for _, s in lex[:4]])
print("rrf score of relevant chunk :", round(1 / 61 + (1 / (60 + [d for d, _ in den].index(rel) + 1) if rel in [d for d, _ in den] else 0), 4))
top = trace(q["text"])[0]
print("rrf score of fused rank 1  :", round(sum(1 / (60 + r) for r in (top["bm25_rank"], top["dense_rank"]) if r), 4))
bm_gap = lex[0][1] / lex[1][1]
print(f"bm25 score ratio, rank 1 to rank 2: {bm_gap:.2f}; rrf score ratio: {(1/61)/(1/62):.3f}")

query: what does err 4072 mean | relevant: ['err-4072']
{"chunk": "err-4016", "bm25_rank": 3, "dense_rank": 1, "fused_rank": 1}
{"chunk": "err-4038", "bm25_rank": 6, "dense_rank": 5, "fused_rank": 2}
{"chunk": "err-4029", "bm25_rank": 4, "dense_rank": 9, "fused_rank": 3}
relevant chunk: bm25 rank 1 dense rank None
bm25 scores of top 4: [6.54, 2.14, 2.14, 2.14]
rrf score of relevant chunk : 0.0164
rrf score of fused rank 1  : 0.0323
bm25 score ratio, rank 1 to rank 2: 3.06; rrf score ratio: 1.016

The relevant chunk is first in the lexical list with a score of 6.54, three times the 2.14 of the next chunk, and absent from the top 20 of the dense list. A chunk about a different error code is third in the lexical list and first in the dense list. RRF sees positions only: the relevant chunk earns 1/61 = 0.0164 from one list, and the competitor earns 1/63 + 1/61 = 0.0323 from two. The ratio between the first two lexical scores is 3.06, and the ratio between the corresponding RRF contributions is 1.016. A margin that indicates a match on a rare token is converted into a one-position difference, which two weaker votes then outweigh. This is the general weakness of rank fusion: it assumes the legs are exchangeable voters, and it treats agreement as evidence. For identifier-like queries agreement between a precise leg and an imprecise one is not evidence of anything.

Score fusion and its own traps

Score fusion keeps the margin. Min-max normalization maps a leg’s returned scores to \([0,1]\) with \(\tilde s=(s-s_{\min})/(s_{\max}-s_{\min})\), and the fused score is \(\sum_j w_j\tilde s_j(d)\), with 0 for a document a leg did not return. The z-score variant subtracts the list mean and divides by its standard deviation. OpenSearch’s normalization processor, as one documented example, offers min-max and L2 normalization and combines the normalized scores by arithmetic, geometric, or harmonic mean with weights that must sum to 1.

Each choice hides an assumption that can fail on real queries. Normalization depends on the returned list, so changing the depth \(n\) changes every normalized score. The top document of every leg is mapped to 1.0 by min-max even when that leg has found nothing good, so a leg that returns only weak matches still casts a full vote. A single outlier, such as the exact-match score on an identifier, compresses everything else toward the bottom of the range under min-max, which helps the identifier slice and flattens the rest of that list. The z-score is less sensitive to a fixed range and more sensitive to the shape of the score distribution, which differs between a query with one strong match and a query with fifty similar ones. In this synthetic collection the two methods trade places by slice. On a real collection the same ambiguity applies, which is why the choice is a measurement and not a default.

The comparison is also not noise at this sample size. The paired bootstrap on per-query nDCG@3 (the procedure from the embeddings article), run on the 60 queries and on two slices, gives the following intervals for min-max minus RRF.

from pipeline import *
chunks = build_corpus(); Q = build_queries(chunks); R = Retriever(chunks)

def per_query(system, qs):
    return [ndcg_at(system(q), q["rel"], 3) for q in qs]

rrf_sys = lambda q: R.search(q["text"])
mm_sys = lambda q: R.search(q["text"], fusion="minmax")
for label, qs in [("all 60 queries", Q),
                  ("identifier slice (12)", [q for q in Q if q["slice"] == "identifier"]),
                  ("paraphrase slice (12)", [q for q in Q if q["slice"] == "paraphrase"])]:
    d, lo, hi = paired_bootstrap(per_query(mm_sys, qs), per_query(rrf_sys, qs))
    print(f"min-max minus rrf, nDCG@3, {label:22s}: {d:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]")

min-max minus rrf, nDCG@3, all 60 queries        : +0.124  95% CI [+0.061, +0.202]
min-max minus rrf, nDCG@3, identifier slice (12) : +0.322  95% CI [+0.125, +0.542]
min-max minus rrf, nDCG@3, paraphrase slice (12) : +0.228  95% CI [+0.072, +0.426]

The difference over all queries is +0.124 with an interval that excludes zero. The slice intervals are wide, [+0.125, +0.542] on twelve identifier queries, which is what twelve queries buy. They are enough to say min-max beats equal-weight RRF on this collection, and not enough to estimate by how much for a real one.

Depth, and the filters that come before ranking

Each leg returns \(n\) results, and the union of the legs is the raw material for everything later. The last block of the earlier output measured the recall of that union by depth: 0.792 at depth 3, 0.892 at 5, 0.967 at 10, and 1.000 at 20 and 40. Pool recall rises quickly and then flattens, and the knee is where the benchmark should set depth. The extra depth beyond the knee costs reranker work and adds distractors without adding evidence; depth below it caps the final recall. The right depth depends on the collection and on the later stages, so it is tuned on the benchmark and re-checked when the corpus grows.

recall of the candidate pool (union of both legs at depth d), all queries
depth  3: pool recall = 0.792
depth  5: pool recall = 0.892
depth 10: pool recall = 0.967
depth 20: pool recall = 1.000
depth 40: pool recall = 1.000

One rule applies regardless of method. Permission, tenant, language, and effective-date filters belong inside every leg, before ranking, so that each leg’s top \(n\) is already a list of chunks the user may see. A filter applied after fusion starves the list when the user can read few chunks, and a reranker that is shown text the user may not read has already used it. The article on preserving access controls in retrieval data works through placement of the filter and the arithmetic of why post-filtering loses results.

Reranking: a second opinion that costs per candidate

A bi-encoder or BM25 builds each chunk’s representation without looking at the query, which is what lets the chunk side be computed once and indexed. A cross-encoder reads the query and one chunk together and outputs one relevance score, so the model can see which query word the chunk’s qualifier answers. That is also why it cannot be precomputed: its cost is paid per query and per candidate. The article on embeddings and retrieval representations introduces the distinction; here the question is how to place the reranker in the funnel. Nogueira and Cho showed an early and influential version, a BERT model that scores query-passage pairs and reorders a first-stage list on MS MARCO, and Thakur and colleagues report in the BEIR benchmark that reranking and late-interaction models achieve the best average zero-shot results at high computational cost.

Rerankers differ in what they score. A pointwise reranker scores each candidate alone and sorts by the score. A pairwise reranker compares two candidates at a time. A listwise reranker takes the whole candidate list and produces an order; Sun and colleagues study this with prompted language models that generate a permutation of the candidates, and report results competitive with supervised rerankers on standard benchmarks. A pointwise cross-encoder costs one forward pass per candidate and is easy to batch. A listwise language-model reranker puts all candidates in one prompt, so its cost is the prompt length, its output can fail to be a valid permutation, and its result can depend on the order in which the candidates are presented. A test that shuffles the input order of a fixed candidate set and measures how much the output changes shows how much a listwise reranker is a function of the list and how much of its position.

A stand-in for the cross-encoder

The reranker in the experiments reads a query and a chunk together and returns one number. It adds 2 when the chunk’s topic matches a topic in the query (synonyms included), adds 1 when the chunk’s plan appears in the query, subtracts 1 when the query names a plan and the chunk is for a different one, and adds 3 when the chunk contains the exact error code in the query. Gaussian noise with scale \(\sigma\) stands in for the model’s own errors. With \(\sigma=0\) it is an oracle for single-intent queries, and the experiments vary \(\sigma\) to show how a reranker of different reliability behaves in the funnel. It sees the whole query, so it has no way to pair a plan with a topic when a query has two of each, and the experiment below leaves the two-part slice out for that reason.

CODE = re.compile(r"[a-z]+\s*-?\s*(\d{4})")

def joint_score(query, chunk, sigma=0.0, seed=0):
    """Stand-in for a cross-encoder: reads query and chunk together and returns one score.
    Topic match, plan match and an exact identifier match add to the score; a plan that
    conflicts with the query's plan subtracts. Gaussian noise of scale sigma stands in for
    the model's own errors."""
    q = query.lower()
    q_topics = {CONCEPTS[t] for t in re.split(r"[^a-z]+", q) if t in CONCEPTS and not CONCEPTS[t].startswith("plan:")}
    q_plans = {t for t in re.split(r"[^a-z]+", q) if t in PLANS}
    m = CODE.search(q)
    s = 0.0
    if chunk["topic"] in q_topics: s += 2.0
    if q_plans and chunk["plan"] in q_plans: s += 1.0
    if q_plans and chunk["plan"] and chunk["plan"] not in q_plans: s -= 1.0
    if m and f"err-{m.group(1)}" in chunk["text"].lower(): s += 3.0
    return s + random.Random(f"{seed}|{query}|{chunk['id']}").gauss(0, sigma)

The program below applies it to the top \(M\) candidates of a first stage (RRF of both legs, depth 40) for several values of \(M\) and \(\sigma\). The score is nDCG@3 over the 36 queries of the identifier, paraphrase, and exact slices. The last lines compute how much text the reranker reads, with a query of eight words and the corpus’s mean chunk length.

from pipeline import *
chunks = build_corpus(); Q = build_queries(chunks); R = Retriever(chunks)
single = [q for q in Q if q["slice"] in ("identifier", "paraphrase", "exact")]

def mean_ndcg(system, qs):
    return sum(ndcg_at(system(q), q["rel"], 3) for q in qs) / len(qs)

print("nDCG@3 on 36 single-intent queries; first stage = RRF (c=60) of both legs, depth 40")
print(f"{'rerank depth':>13s}" + "".join(f"{'sigma='+str(s):>10s}" for s in (0, 1, 2, 3)))
print(f"{'none':>13s}{mean_ndcg(lambda q: R.search(q['text'], depth=40), single):10.3f}")
for rd in (3, 5, 10, 20, 40):
    row = []
    for s in (0, 1, 2, 3):
        row.append(mean_ndcg(lambda q: R.search(q["text"], depth=40, rerank_depth=rd, sigma=s), single))
    print(f"{rd:13d}" + "".join(f"{v:10.3f}" for v in row))

print("\ncost of the rerank stage: pairs scored x tokens per pair (query + chunk)")
q_tok = 8; c_tok = sum(len(c["text"].split()) for c in chunks) / len(chunks)
print(f"mean chunk length in this toy corpus: {c_tok:.1f} words")
for rd in (5, 20, 100):
    print(f"rerank depth {rd:3d}: {rd} pairs, about {rd * (q_tok + c_tok):.0f} words read per query")

nDCG@3 on 36 single-intent queries; first stage = RRF (c=60) of both legs, depth 40
 rerank depth   sigma=0   sigma=1   sigma=2   sigma=3
         none     0.740
            3     0.833     0.803     0.730     0.720
            5     0.889     0.854     0.679     0.655
           10     0.972     0.900     0.619     0.463
           20     1.000     0.848     0.490     0.316
           40     1.000     0.817     0.400     0.223

cost of the rerank stage: pairs scored x tokens per pair (query + chunk)
mean chunk length in this toy corpus: 21.1 words
rerank depth   5: 5 pairs, about 145 words read per query
rerank depth  20: 20 pairs, about 582 words read per query
rerank depth 100: 100 pairs, about 2909 words read per query

Reading the grid

The first row is the first stage alone, 0.740. At \(\sigma=0\) the reranker is perfect once it sees 20 candidates, and more candidates only help: 0.833 at \(M=3\), 0.972 at \(M=10\), 1.000 at 20 and 40. This is the recall ceiling at work. A reranker of depth 3 can only reorder the three chunks it was given, and the relevant chunk is not always among them.

At \(\sigma=2\) the sign flips: the score falls from 0.730 at \(M=3\) to 0.400 at \(M=40\), and the reranked list is worse than the first stage (0.740) at every \(M\). The reason is arithmetic. In this stand-in the correct chunk leads the best distractor by 1 to 3 score points. A noise draw of scale \(\sigma\) is added to every candidate, and the largest of \(M\) independent draws from a normal distribution grows roughly like \(\sigma\sqrt{2\ln M}\), which is 1.5\(\sigma\) for \(M=3\) and 2.7\(\sigma\) for \(M=40\). Each extra candidate is another chance for a wrong chunk to draw a large positive error. At \(\sigma=1\) the two forces meet: the best depth is 10 (0.900), and the score declines to 0.817 at 40. A reranker with errors comparable to the score gaps between candidates has a best depth, and past it every extra candidate is a net loss.

A real cross-encoder is not Gaussian noise on top of an oracle, and its error depends on the query and the chunk. The pattern carries over as a measurement to make. Run the reranker at several depths on the benchmark, plot the final nDCG@k against \(M\), and pick the depth at the plateau or the peak, not the largest depth the latency budget allows. When the reranker is a language model, the same sweep also measures the cost of the extra prompt length.

The cost side is the second half of the output. The reranker reads about 145 words for 5 candidates, 582 for 20, and 2,909 for 100, in proportion to \(M\). For a transformer cross-encoder the work per pair is at least linear in the length of the pair and grows faster than linear when attention over the pair dominates, and the latency is set by how many pairs fit in a batch. If a batch of \(B\) pairs takes \(t_B\), reranking \(M\) candidates takes about \(\lceil M/B\rceil\,t_B\), so doubling \(M\) roughly doubles the latency once \(M\) exceeds the batch size. The latency table later in the article uses assumed per-pair costs to show how this adds to the end-to-end time.

What a reranker needs from its input

Two properties of the input matter as much as the model. First, rerankers have a maximum input length, and text beyond it is cut. A chunk whose qualifier (“for enterprise plans only”) sits at the end of a long passage can be scored without ever being read. The glossary entry on reranking covers input limits and how to count them with the reranker’s own tokenizer; the practical consequence for chunking is to put conditions near the start of the chunk or to keep chunks shorter than the reranker window. Second, a reranker’s score measures relevance to the query. It does not measure that the chunk is current, permitted, or true, so those properties have to be settled by filters before ranking and by the answer stage after it.

The score is also not calibrated across queries. A fixed cutoff such as “discard chunks below 0.4” is a threshold on a number whose distribution changes with query length and topic, so it has to be set on labeled queries from the target collection and rechecked after a model change. Whether the evidence that remains is enough to answer, and what to do when it is not, is the subject of the article on grounded answers and handling missing evidence; the reranker’s scores are an input to that decision, not the decision.

The configuration table later in the article adds a caution. With the same stand-in at \(\sigma=1\) and \(M=10\), reranking lifts the identifier slice from 0.803 to 0.928 and lowers the exact-wording slice from 1.000 to 0.886. Where the first stage is already right, a reranker with any error can only make it worse. Reranking is therefore evaluated per slice, and a system can route around it for slices where the first stage is reliable, for example when the top fused score is far above the second. Whether such a gate pays is again a benchmark question.

Query transformation

A query transformation changes what is searched. The user’s text goes in, and one or more search strings, possibly with filters, come out. Several different transformations share the name, because they fix different failures.

Transformation	What it fixes	How it fails	Added cost
Standalone rewrite of a follow-up	A question that depends on earlier turns	Resolves a reference to the wrong entity; adds a fact the user never stated	One model call before retrieval
Normalization	Spelling, casing, separators, units	Over-normalizes identifiers or removes a qualifier	Negligible
Multi-query expansion	Vocabulary the user did not use	Paraphrases drift; one bad variant adds distractors	Several retrievals plus fusion
Hypothetical-document embedding	A short question that embeds far from long answers	The invented answer anchors search on wrong specifics	One generation, one embedding
Step-back (more general question)	A question too specific for the available text	The general query returns generic chunks that crowd out the specific one	One call, one more retrieval
Decomposition	Several intents in one query	Splits a single intent; merge rule rewards the wrong chunks	One call, several retrievals
Filter extraction	A condition that should be a metadata filter, such as plan or version	A wrong filter returns nothing, or hides the right chunk	One call or a rule

The literature supports several of these. Anantha and colleagues released QReCC, a dataset for rewriting conversational questions into standalone ones and then retrieving, which frames the follow-up problem as its own task. Ma and colleagues propose a rewrite step before retrieval for retrieval-augmented language models and train the rewriter with feedback from the reader. Gao and colleagues propose hypothetical document embeddings: a language model writes a document that answers the query, an encoder embeds that document, and its vector is used to find real neighbors. Zheng and colleagues propose step-back prompting, which has the model first state a more general question or principle. Each has been shown to help on some benchmarks, and none is a default to switch on.

Making a follow-up question stand alone

The follow-up slice scored 0.208 on the lexical leg, 0.042 on the dense leg, and 0.105 after min-max fusion. The topic is in the previous turn, so a standalone rewrite should recover it. The stand-in below carries the topic word from the most recent earlier turn into the current question. A language model would do this with more flexibility and with more ways to go wrong; the rule keeps the experiment deterministic. It is paired with a second stand-in, a rewriter that drifts, to show the failure that a guard has to catch.

def standalone_rewrite(query, history):
    """Deterministic stand-in for an LLM that makes a follow-up question self-contained:
    carry the topic word from the latest earlier turn into the current question."""
    for turn in reversed(history or []):
        for t in re.split(r"[^a-z]+", turn.lower()):
            if t in CONCEPTS and not CONCEPTS[t].startswith("plan:"):
                return f"{TOPICS[CONCEPTS[t]][0]} {query.rstrip('?')}"
    return query

def drifting_rewrite(query):
    """A stand-in for a bad rewrite: it paraphrases the identifier away."""
    return re.sub(r"(?i)err[- ]?(\d{4})", "error number", query)

def expand_variants(query):
    """Stand-in for 'write two alternative phrasings': swap query words for synonyms."""
    variants = [query]
    toks = re.split(r"(\W+)", query)
    for j in (0, 1):
        out = []
        for t in toks:
            c = CONCEPTS.get(t.lower())
            if c and not c.startswith("plan:"):
                pool = [TOPICS[0]] + TOPICS[2]
                out.append(pool[(pool.index(t.lower()) + 1 + j) % len(pool)] if t.lower() in pool else t)
            else:
                out.append(t)
        variants.append("".join(out))
    return list(dict.fromkeys(variants))

def split_parts(query):
    """Split 'A on X plans and B on Y plans' into one sub-query per part."""
    return [p.strip() for p in re.split(r"\band\b", query)]

The second function replaces an error code with the words error number, which is the kind of change a model makes when it “cleans up” a query. A guard protects what the lexical leg needs: it extracts the identifiers and plan names from the original query, checks that each still appears in the rewrite, and falls back to the original query when one is lost.

def protected_tokens(query):
    """Tokens a rewrite must not lose: identifiers and plan names."""
    toks = set(PLANS) & set(re.split(r"[^a-z]+", query.lower()))
    toks |= {f"err{m}" for m in re.findall(r"(?i)err[- ]?(\d{4})", query)}
    return toks

def guarded(original, rewritten):
    lost = {t for t in protected_tokens(original)
            if t not in re.sub(r"[^a-z0-9]", "", rewritten.lower()) and t not in rewritten.lower()}
    return (original, lost) if lost else (rewritten, set())

The program below evaluates three things on the hybrid min-max configuration at depth 20: the follow-up slice with and without the standalone rewrite, the identifier slice with the drifting rewrite unguarded and guarded, and, for the later subsections, multi-query expansion and decomposition.

from pipeline import *
chunks = build_corpus(); Q = build_queries(chunks); R = Retriever(chunks)
base = lambda text: R.search(text, fusion="minmax", depth=20)
by = lambda s: [q for q in Q if q["slice"] == s]
mean = lambda system, qs: sum(ndcg_at(system(q), q["rel"], 3) for q in qs) / len(qs)

def with_rewrite(q):
    return base(standalone_rewrite(q["text"], q.get("history")))

def multi_query(q):
    lists = [base(v) for v in expand_variants(q["text"])]
    return rrf(lists)

def decomposed(q, merge=interleave):
    return merge([base(part) for part in split_parts(q["text"])])

def rewrite_guarded(q):
    text, _ = guarded(q["text"], drifting_rewrite(q["text"]))
    return base(text)

print("follow-up turns (12 queries), nDCG@3")
fu = by("followup")
print("  original question only :", round(mean(lambda q: base(q["text"]), fu), 3))
print("  after standalone rewrite:", round(mean(with_rewrite, fu), 3))
for q in fu[:2]:
    print("   ", q["history"], "|", q["text"], "->", standalone_rewrite(q["text"], q["history"]))
print("paraphrase queries (12), nDCG@3")
pa = by("paraphrase")
print("  single query            :", round(mean(lambda q: base(q["text"]), pa), 3))
print("  three phrasings, RRF    :", round(mean(multi_query, pa), 3))
print("  e.g.", expand_variants(pa[0]["text"]))
print("BM25 leg alone on paraphrase, single vs three phrasings")
print("  single:", round(mean(lambda q: R.search(q["text"], "bm25"), pa), 3),
      " three phrasings:", round(mean(lambda q: rrf([R.search(v, "bm25") for v in expand_variants(q["text"])]), pa), 3))
print("two-part queries (12), nDCG@3")
tp = by("twopart")
print("  one query               :", round(mean(lambda q: base(q["text"]), tp), 3))
print("  decomposed, RRF merge   :", round(mean(lambda q: decomposed(q, rrf), tp), 3))
print("  decomposed, round-robin :", round(mean(decomposed, tp), 3))
print("  e.g.", split_parts(tp[0]["text"]))
print("identifier queries (12), nDCG@3")
ident = by("identifier")
print("  original                :", round(mean(lambda q: base(q["text"]), ident), 3))
print("  drifting rewrite, no guard:", round(mean(lambda q: base(drifting_rewrite(q["text"])), ident), 3))
print("  drifting rewrite, guarded :", round(mean(rewrite_guarded, ident), 3))
print("  guard on one query:", guarded(ident[0]["text"], drifting_rewrite(ident[0]["text"])))

follow-up turns (12 queries), nDCG@3
  original question only : 0.105
  after standalone rewrite: 1.0
    ['how does reimbursement work?'] | what about monthly plans? -> refund what about monthly plans
    ['how does backup work?'] | what about trial plans? -> export what about trial plans

identifier queries (12), nDCG@3
  original                : 0.803
  drifting rewrite, no guard: 0.178
  drifting rewrite, guarded : 0.803
  guard on one query: ('what does ERR-4029 mean', {'err4029'})

The standalone rewrite lifts the follow-up slice from 0.105 to 1.000, because the rewritten query contains the topic word that the original lacked. The drifting rewrite on identifier queries drops the slice from 0.803 to 0.178 (all three runs use the min-max hybrid at depth 20): the error code is gone, and the rewritten query matches nothing in particular. The guard sees that err4029 is missing from the rewrite and returns the original query, which restores 0.803. The last output line shows the guard’s decision on one query, including the token it found missing.

A rule-based rewriter that scores 1.000 on twelve queries it was written for says little about a model. What it shows is the structure around a rewriter that should be kept whatever the rewriter is: the rewrite is a proposal, a deterministic check compares it with the original, and the original is used when the check fails. The same structure applies to latency. The rewrite is a model call that sits in front of retrieval, so its time is added in series, and a timeout has to fall back to the original query and not fail the request.

The sketch below shows the calling wrapper for a model-based rewriter. It is illustrative and was not run against a live service; call_model is a placeholder for whatever client the application uses. The prompt contract, the structured output, and the retry and timeout behavior follow the interface principles in the article on context and prompt interfaces and the article on integrating model APIs.

import json

REWRITE_INSTRUCTIONS = """Rewrite the last user message as one self-contained search query.
Rules: keep identifiers, error codes, product names and quoted text exactly as written;
do not add facts that are not in the conversation; if the message is already
self-contained, return it unchanged. Answer as JSON: {"standalone_query": "..."}"""

def rewrite_query(call_model, history, question, timeout_s=2.0):
    """Returns (query_to_search, reason). Any failure falls back to the original question."""
    prompt = {"instructions": REWRITE_INSTRUCTIONS, "history": history[-4:], "message": question}
    try:
        data = json.loads(call_model(prompt, timeout=timeout_s))
        candidate = str(data["standalone_query"]).strip()
    except Exception as exc:                       # timeout, invalid JSON, missing field
        return question, f"fallback: {type(exc).__name__}"
    final, lost = guarded(question, candidate)     # the guard from the experiment above
    return final, ("fallback: lost " + ",".join(sorted(lost))) if lost else "rewritten"

The conversation history and the user’s message are untrusted text, and they go into a prompt whose output becomes a search string. Treat that output as data: it is searched, and it is not allowed to widen what the user may see. Filters derived from the rewrite should be limited to relevance attributes such as plan or product version, and permission filters should come from the authenticated identity alone.

Expansion and multi-query retrieval

Expansion searches more than one phrasing and fuses the lists. The stand-in expand_variants produces the original query and two variants in which topic words are replaced by synonyms. The three result lists are fused with RRF. This is a place where RRF is the right tool, because the lists are three views of one intent and agreement between them is meaningful.

paraphrase queries (12), nDCG@3
  single query            : 0.969
  three phrasings, RRF    : 1.0
  e.g. ['get my tier on a trial subscription', 'get my promote on a trial subscription', 'get my upgrade on a trial subscription']
BM25 leg alone on paraphrase, single vs three phrasings
  single: 0.21  three phrasings: 0.272

On the paraphrase slice the hybrid configuration already scores 0.969, and three phrasings lift it to 1.000. On the lexical leg alone, where the synonym gap is the whole problem, three phrasings raise 0.210 to 0.272: the variants help a little because one of them sometimes contains a word that the chunks use, and most do not. The gain is small relative to its cost of three retrievals, and in this collection a dense leg that already knows synonyms covers the same gap. Expansion earns its cost when the legs cannot bridge a vocabulary gap by themselves and when the variants come from knowledge of the collection (its glossary, its own synonyms), not from a model’s general sense of the words. Hypothetical-document embedding has the same cost structure with a sharper risk: the generated document contains details the model made up, and the embedding of those details retrieves chunks that talk about them. Whether it helps is a measurement on the target queries.

Decomposition and the merge rule

A query with two intents is the case where one search is least suited, because both legs see a mixture. Decomposition splits the query into sub-queries, searches each, and merges the results. The merge rule matters more than the split. The program evaluated two rules: the RRF fusion of the sub-query lists, and a round-robin that takes the first result of each list, then the second, skipping repeats.

two-part queries (12), nDCG@3
  one query               : 0.442
  decomposed, RRF merge   : 0.243
  decomposed, round-robin : 0.859
  e.g. ['dollars on monthly plans', 'levy on annual plans']

One search for the whole query scores 0.442. Merging the sub-query lists with RRF drops it to 0.243, below the undivided query, and round-robin lifts it to 0.859. The cause is the agreement logic of RRF. The two sub-queries ask for different chunks, so the chunk that a user needs from each sub-query appears in only one list, while a compromise chunk, such as the topic of the first part with the plan of the second, ranks fairly high in both lists and collects two contributions. For the first two-part query in the slice, the RRF merge put currency-annual ahead of the needed currency-monthly for that reason. Agreement is a good merge signal across views of one intent and a bad one across different intents. Round-robin guarantees each sub-query a share of the final list, which is what the user wants here. The general rule is to choose the merge by what the lists represent: fuse when they are redundant views of one need, and give each its quota when they are different needs.

Decomposition needs a router that decides when to split, and the router is another place to be wrong. Splitting a single-intent query halves each sub-query’s budget and invites the wrong-merge failure above. Cap the number of sub-queries, split only on a detected structure (several question clauses, conjunctions that join two noun phrases with their own qualifiers), and include single-intent queries that contain the word “and” in the benchmark so the router is measured on its false alarms.

What each configuration costs and buys

The configuration table puts the pieces together. Seven configurations run on all 60 queries, from each leg alone to the full pipeline of rewrite, decomposition, hybrid retrieval, and reranking. The full pipeline rewrites a follow-up when the query has history, splits on “and”, runs the min-max hybrid for each part, optionally reranks the top 10 of each at \(\sigma=1\), and merges parts by round-robin. The second block of the output is a latency model with assumed stage costs: 5 ms for the lexical leg, 30 ms for embedding the query and searching, 1 ms for fusion, 400 ms for a rewrite call, and 20 ms plus 3 ms per pair for reranking, with legs and sub-queries running in parallel. Those figures are assumptions chosen to show how the stages add, not measurements of any system.

from pipeline import *
chunks = build_corpus(); Q = build_queries(chunks); R = Retriever(chunks)

def hybrid(text, rerank=0, sigma=1.0):
    return R.search(text, fusion="minmax", depth=20, rerank_depth=rerank, sigma=sigma)

def full_pipeline(q, rerank=10, sigma=1.0):
    text = q["text"]
    if q.get("history"):                                   # query transformation: follow-up
        text, _ = guarded(text, standalone_rewrite(text, q["history"]))
    parts = split_parts(text) if " and " in text else    # decomposition
    return interleave([hybrid(p, rerank, sigma) for p in parts])

CONFIGS = [
    ("bm25",                       lambda q: R.search(q["text"], "bm25"),            dict(llm=0, pairs=0, lat=0)),
    ("dense",                      lambda q: R.search(q["text"], "dense"),           dict(llm=0, pairs=0, lat=0)),
    ("hybrid, rrf",                lambda q: R.search(q["text"]),                    dict(llm=0, pairs=0, lat=0)),
    ("hybrid, min-max",            lambda q: hybrid(q["text"]),                      dict(llm=0, pairs=0, lat=0)),
    ("hybrid + rerank(10)",        lambda q: hybrid(q["text"], 10),                  dict(llm=0, pairs=10, lat=10)),
    ("+ rewrite/decompose",        lambda q: full_pipeline(q, rerank=0),             dict(llm=1, pairs=0, lat=0)),
    ("+ rewrite/decompose + rerank", lambda q: full_pipeline(q),                     dict(llm=1, pairs=20, lat=10)),
]
rows = [(n, evaluate(f, Q)) for n, f, _ in CONFIGS]
table(rows)

# modeled latency: assumed per-stage costs, not measurements
LEG, EMBED, FUSE, LLM = 5, 30, 1, 400           # ms (assumptions): lexical leg, query embedding+ANN, fusion, one rewrite call
RERANK_FIXED, RERANK_PER_PAIR = 20, 3           # ms (assumptions)
print("\nmodeled latency with assumed stage costs (ms); legs and sub-queries run in parallel")
for n, _, m in CONFIGS:
    legs = {"bm25": LEG, "dense": EMBED}.get(n, max(LEG, EMBED) + FUSE)
    rr = (RERANK_FIXED + RERANK_PER_PAIR * m["lat"]) if m["pairs"] else 0
    print(f"{n:30s} llm calls={m['llm']}  pairs (worst case)={m['pairs']:2d}  modeled latency={legs + rr + LLM * m['llm']:4d}")

                             identifier paraphrase      exact   followup    twopart     all
bm25                              1.000      0.210      1.000      0.208      0.102   0.504
dense                             0.344      1.000      1.000      0.042      0.564   0.590
hybrid, rrf                       0.480      0.741      1.000      0.147      0.333   0.540
hybrid, min-max                   0.803      0.969      1.000      0.105      0.442   0.664
hybrid + rerank(10)               0.928      0.969      0.886      0.105      0.531   0.684
+ rewrite/decompose               0.803      0.969      1.000      1.000      0.859   0.926
+ rewrite/decompose + rerank      0.928      0.969      0.886      0.969      0.667   0.884

modeled latency with assumed stage costs (ms); legs and sub-queries run in parallel
bm25                           llm calls=0  pairs (worst case)= 0  modeled latency=   5
dense                          llm calls=0  pairs (worst case)= 0  modeled latency=  30
hybrid, rrf                    llm calls=0  pairs (worst case)= 0  modeled latency=  31
hybrid, min-max                llm calls=0  pairs (worst case)= 0  modeled latency=  31
hybrid + rerank(10)            llm calls=0  pairs (worst case)=10  modeled latency=  81
+ rewrite/decompose            llm calls=1  pairs (worst case)= 0  modeled latency= 431
+ rewrite/decompose + rerank   llm calls=1  pairs (worst case)=20  modeled latency= 481

Reading the table

The comparison reads best by rows, from the top. Fusion with min-max lifts the overall score from 0.504 (lexical) and 0.590 (dense) to 0.664 and improves the identifier and paraphrase slices at once, at a modeled latency of 31 ms, close to the slower leg because the legs run in parallel. The second-to-last row is the large step: 0.926 overall, driven by the follow-up slice (0.105 to 1.000) and the two-part slice (0.442 to 0.859). It also costs the most in latency, 431 ms in this model, because one model call enters in series. The query transformation fixed the slices that no amount of retrieval tuning could, and it did so by adding a call that is about 400 ms of the 431.

The reranker rows are the surprise. With a reranker of noise scale 1, the configuration with reranking scores 0.684 against 0.664 without it, and the full pipeline with reranking scores 0.884 against 0.926 without it. The identifier slice improves (0.803 to 0.928), the exact slice worsens (1.000 to 0.886), and the two-part slice worsens (0.859 to 0.667) probably because each sub-query reranks its own candidates and the noise acts twice. A table like this is why the reranker is a decision per slice. In this synthetic collection, the best configuration is the one without it, and a collection with a stronger reranker or harder first-stage ordering would give a different answer.

Two tendencies hold beyond this toy. The stages that add a model call add latency in series, while those that add retrievals can run in parallel, so the shape of the latency budget differs by stage. And each stage helps the slice it targets and can hurt the others, so the quality of the whole is the mix of slices weighted by how often real queries fall into them. If follow-ups are 40% of traffic, the 0.105 to 1.000 jump is the main result. If they are 2%, it is a detail.

When to add each stage

Situation	Stage worth testing	Evidence to look for before keeping it
Collection of a few thousand chunks; users type the vocabulary of the documents	One leg, exact scan; skip approximate indexes	Hybrid does not beat the single leg on the benchmark by more than its confidence interval
Queries with codes, names, or part numbers	A lexical leg or exact-match field; normalization rules for identifiers	Identifier-slice recall of the lexical leg exceeds that of the dense leg
Users describe things in their own words	A dense leg, or expansion with collection-specific synonyms	Paraphrase-slice recall of the dense leg exceeds that of the lexical leg
Both kinds of queries in one stream	Hybrid with weights or normalization tuned per the benchmark	Fused result at least matches the better leg on every slice, not only on average
Right chunk is in the pool but not near the top	A reranker at the depth where the pool recall plateaus	Final nDCG@k rises over the first stage on most slices and does not fall on slices the first stage already handles
Multi-turn use	Standalone rewrite with a guard and a fallback	Follow-up slice recall with history approaches that of the equivalent standalone questions
Compound questions	Decomposition with a per-sub-query merge	Recall of all relevant chunks improves, and single-intent queries containing “and” do not get worse

The table is conditional on purpose. A stage that is not justified by a failure on a slice of the benchmark is a stage that adds latency, cost, and another thing to version, and none of the stages repairs a chunk that was split badly or an index that is out of date; those are problems of the index itself.

Designing the benchmark for this stage

The numbers in this article are the kind of evidence a benchmark for this stage should produce, so the design questions follow from what the experiments needed. The benchmark has to say, for a change to the stage, which slice changed, whether the change is larger than the noise, what it cost, and at which stage in the funnel it happened.

Queries, slices, and judgments

Sample queries from the traffic the application will see, and label each with the slice it belongs to: identifier lookups, paraphrases, exact wording, follow-ups with their history, multi-intent questions, and queries with no answer in the collection. The slices in this article were chosen to isolate one failure each, which is useful for diagnosis. A benchmark for decisions needs real query frequencies, because the overall score is the slice scores weighted by how often each slice occurs. Queries written by hand or by a model to cover a failure are a good addition for coverage and a poor basis for a headline number. The article on reproducible test sets covers building and versioning such sets in general; the retrieval-specific parts are below.

• Judge at the unit the system returns. If the index returns chunks, relevance is judged per chunk, against a pinned version of the index. When the index is rebuilt with different chunk boundaries, the chunk identifiers change and the judgments need to be remapped or redone; this is where the index version and chunk provenance from the index article pay off.
• Pool the candidates for judging. Judge the union of the top results of every variant under comparison, not just the current system’s. A chunk that no judged system retrieved is unjudged, and a scorer treats unjudged as not relevant, which penalizes a new system for finding good chunks the old one missed. Sample unjudged chunks from new variants and judge them before trusting a comparison.
• Record all relevant chunks, and score the multi-evidence queries on all of them. A two-part query has two relevant chunks, and a system that finds one has half the evidence. The recall of the relevant set, and the fraction of queries for which all of it is found, are separate numbers.
• Include queries with no answer. For these the retrieval metric has no relevant chunk to find, so they are scored by what the stage returns: how high the best score is, and whether it separates from queries that do have answers. Their use belongs to the answer stage, covered in the article on missing evidence and in the article on evaluating retrieval and answers separately.

What to measure at each stage

Report a small, fixed set of numbers per slice and per configuration. The recall of the candidate pool at depth \(M\) shows the ceiling. The final recall@\(k\) and nDCG@\(k\) at the \(k\) the generator will receive show what the application gets. The gap between them shows what fusion and reranking did with the pool. Alongside them go the costs: the number of model calls, the number of reranked pairs, and latency percentiles at the query volume the system will carry, not an average. A change that raises nDCG by 0.02 and doubles p95 latency is a decision, and the table that makes it visible has both columns.

Tuning without fooling yourself

The configuration space is larger than it looks: the fusion method, the weights, the constant \(c\), the depth of each leg, the rerank depth, which transformations are on. Searching it on the same queries that report the result inflates the result. The next program makes that concrete on this collection. It evaluates 60 configurations (four fusion methods, five weights, three depths) on all 60 queries, then repeats 300 times: it picks the best configuration on 15 randomly chosen queries and scores that configuration on the other 45.

import random, itertools
from pipeline import *
chunks = build_corpus(); Q = build_queries(chunks); R = Retriever(chunks)

GRID = [(f, w, d) for f in ("rrf5", "rrf60", "minmax", "zscore")
        for w in (0.2, 0.35, 0.5, 0.65, 0.8) for d in (5, 10, 20)]

def run_cfg(text, f, w, d):
    if f.startswith("rrf"):
        return R.search(text, depth=d, c=int(f[3:]), weights=[1 - w, w])
    return R.search(text, depth=d, fusion=f, weights=[1 - w, w])

per_query = {cfg: [ndcg_at(run_cfg(q["text"], *cfg), q["rel"], 3) for q in Q] for cfg in GRID}
mean = lambda xs: sum(xs) / len(xs)
best_all = max(GRID, key=lambda c: mean(per_query))
print(f"{len(GRID)} configurations x {len(Q)} queries; best on all queries: {best_all}, "
      f"nDCG@3 = {mean(per_query[best_all]):.3f}")

rng = random.Random(3)
dev_scores, test_scores = [], []
for _ in range(300):
    idx = list(range(len(Q))); rng.shuffle(idx)
    dev, test = idx[:15], idx[15:]
    pick = max(GRID, key=lambda c: mean([per_query[i] for i in dev]))
    dev_scores.append(mean([per_query[pick][i] for i in dev]))
    test_scores.append(mean([per_query[pick][i] for i in test]))
print(f"tune on 15 random queries, test on the other 45 (300 random splits):")
print(f"  mean dev score of the chosen configuration : {mean(dev_scores):.3f}")
print(f"  mean test score of the same configuration  : {mean(test_scores):.3f}")
print(f"  optimism (dev - test)                      : {mean(dev_scores) - mean(test_scores):.3f}")
print(f"  mean over all 60 configurations, any query : {mean([mean(v) for v in per_query.values()]):.3f}")

60 configurations x 60 queries; best on all queries: ('minmax', 0.5, 20), nDCG@3 = 0.664
tune on 15 random queries, test on the other 45 (300 random splits):
  mean dev score of the chosen configuration : 0.711
  mean test score of the same configuration  : 0.611
  optimism (dev - test)                      : 0.099
  mean over all 60 configurations, any query : 0.572

The best configuration on all 60 queries scores 0.664. The configuration chosen on 15 queries scores 0.711 on those 15 and 0.611 on the 45 it was not chosen on, an optimism of 0.099, and the mean over all 60 configurations is 0.572. The selected configuration is better than average on fresh queries, so tuning found something, but the dev score overstates the result by about a tenth of the metric’s range. The best-on-all number of 0.664 carries the same bias, only smaller. The remedy is the usual one. Fix a dev set and a test set before tuning. Choose weights, constants, and depths on dev. Evaluate the chosen configuration once on test and report that number. Retire the test set, or refresh it, when it has been consulted so often that choices have leaked into it. When the set is small, cross-validate over queries, and keep documents that appear in several queries on the same side of the split.

Judging a difference

A configuration change is credible when it holds up under resampling of the queries. The paired bootstrap used earlier gives the interval for the difference in per-query nDCG@\(k\) between two configurations on the same queries. Report it per slice, since the slices differ in both size and direction. Where a slice has few queries the interval is wide, as in the [+0.125, +0.542] on twelve identifier queries. A wide interval is information: it says how many more queries the slice needs before a decision.

Two more habits keep the benchmark usable over time. Make the run deterministic: stable tie-breaking in every ranking (this article’s code breaks ties by chunk identifier), fixed seeds for any sampled component, and a recorded index version, so that a changed number can be traced to a changed configuration and not to run-to-run noise. And turn the benchmark into a regression gate with per-slice floors, so a change that raises the average and breaks the identifier slice does not ship unnoticed.

What the stage returns, a short ranked list of chunks with their identifiers, scores, and the per-stage record, is the input to answer generation. Whether those chunks support an answer, how to cite them, and what to do when they do not is the next problem, covered in the article on grounded answers and handling missing evidence.

References

• Cormack, G. V., Clarke, C. L. A., and Büttcher, S. “Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.” SIGIR 2009: https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf
• Nogueira, R. and Cho, K. “Passage Re-ranking with BERT.” 2019: https://arxiv.org/abs/1901.04085
• Sun, W., et al. “Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents.” EMNLP 2023: https://arxiv.org/abs/2304.09542
• Thakur, N., et al. “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” NeurIPS 2021: https://arxiv.org/abs/2104.08663
• Gao, L., Ma, X., Lin, J., and Callan, J. “Precise Zero-Shot Dense Retrieval without Relevance Labels” (HyDE). 2022: https://arxiv.org/abs/2212.10496
• Ma, X., Gong, Y., He, P., Zhao, H., and Duan, N. “Query Rewriting for Retrieval-Augmented Large Language Models.” EMNLP 2023: https://arxiv.org/abs/2305.14283
• Zheng, H. S., et al. “Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models.” ICLR 2024: https://arxiv.org/abs/2310.06117
• Anantha, R., et al. “Open-Domain Question Answering Goes Conversational via Question Rewriting” (QReCC). NAACL 2021: https://arxiv.org/abs/2010.04898
• Elastic. Reciprocal rank fusion, Elasticsearch reference: https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion
• OpenSearch. Normalization processor (hybrid search score normalization and combination): https://docs.opensearch.org/2.18/search-plugins/search-pipelines/normalization-processor/