Embeddings and Retrieval Representations

Consider a hypothetical support team that indexes its policy pages. A user types: how do I get my money back? The page that answers it is titled “Reimbursement policy” and never uses the word “refund,” let alone “money back.” A second user pastes an error code, ERR-4021, and expects the one page that mentions that exact string. The first question needs a system that knows two different phrasings mean the same thing. The second needs a system that does not blur a precise token into its neighbors. A retrieval component has to serve both, and the choice of how to represent text decides which of them it serves well.

Retrieval decides which text reaches a model at all. The preceding article on context windows and long-context behavior, the prerequisite for this one, covers what a model does with tokens once they are in the window and why filling the window indiscriminately is costly. Before the window comes a different problem: queries and documents become representations, those representations are trained and compared, they can fail when moved to a corpus unlike the one they were trained on, and they have to be measured so that a change shows up as a number rather than an impression. The encoder is a transformer as in the article on self-attention, and the training math uses the softmax and cross-entropy material in the probability article for deep learning.

Retrieval is ranking by a score that a representation defines

Given a query \(q\) and a collection of documents or chunks \(d_1,\dots,d_N\), a retriever returns the \(k\) items with the highest score \(s(q,d)\). Everything else is a decision about how \(s\) is computed and how the top \(k\) are found without scoring all \(N\) items. Two properties of the representation drive that decision.

• What can match. A representation fixes what two texts can have in common. Word counts can only match identical terms. A learned vector can match texts that share no words but only if training taught the model that they belong together.
• How it is indexed. Sparse term representations fit an inverted index, which maps each term to the documents containing it and scores only documents that share a term with the query. A dense vector is compared with every stored vector, or with an approximate subset found by a nearest-neighbor index.

Four families cover most of what is deployed. The table lists them with the score each one computes and the cost that comes with it.

Representation	Score	Index	Matches	Typical failure
Lexical sparse (BM25)	Weighted term overlap	Inverted index	Identical terms after tokenization	Paraphrase with no shared word
Single-vector dense	Cosine or dot product of two vectors	Vector store, often approximate	Texts the encoder learned to place close	Rare identifiers, unseen jargon, domain shift
Learned sparse (e.g., SPLADE)	Weighted overlap of learned term weights, including added terms	Inverted index	Identical or model-added terms	Expansion the model never learned for the domain
Late interaction (e.g., ColBERT)	Sum over query tokens of the best-matching document-token similarity	Many vectors per document	Token-level alignments	Index size; engineering cost

The first two are the focus here, because they expose the central trade-off and because the others are variations on them. A cross-encoder, which reads query and document together and outputs one relevance score, is a different kind of component: it is accurate but cannot precompute anything per document, so it is used to reorder a short list produced by a representation-based retriever rather than to search a corpus.

Sparse retrieval: what BM25 computes

BM25 scores a document by summing, over the query’s terms, a weight that combines three ideas: rare terms matter more than common ones, repeated occurrences help but with diminishing returns, and the same count means less in a long document. A term is whatever the tokenizer produces, so the choices described in the tokenization article (lowercasing, splitting on punctuation, subwords) decide what BM25 can match. With \(N\) documents, document frequency \(\mathrm{df}(t)\), term frequency \(\mathrm{tf}(t,d)\), document length \(|d|\) and average length \(\mathrm{avgdl}\), one common form is

\[ \mathrm{BM25}(q,d)=\sum_{t\in q}\mathrm{idf}(t)\cdot\frac{\mathrm{tf}(t,d)\,(k_1+1)}{\mathrm{tf}(t,d)+k_1\left(1-b+b\,\frac{|d|}{\mathrm{avgdl}}\right)},\qquad \mathrm{idf}(t)=\ln\!\left(1+\frac{N-\mathrm{df}(t)+0.5}{\mathrm{df}(t)+0.5}\right). \]

Read the second factor as a saturating curve in \(\mathrm{tf}\). With \(k_1=0\) the factor is 1 whenever the term appears, so only presence counts. As \(\mathrm{tf}\) grows the factor approaches \(k_1+1\), so no amount of repetition earns more than that ceiling. The parameter \(b\in[0,1]\) controls length normalization: at \(b=0\) document length is ignored, and at \(b=1\) the denominator grows in proportion to \(|d|/\mathrm{avgdl}\). Manning, Raghavan, and Schütze recommend \(k_1\) between 1.2 and 2 and \(b=0.75\) in their textbook, and those are the defaults below. The idf shown is a smoothed variant that stays positive; the textbook presents \(\log(N/\mathrm{df})\) and a smoothed form, and libraries differ in which they use, so scores from two implementations are not comparable.

A four-document corpus makes the arithmetic concrete. Here \(N=4\), the document lengths are 7, 7, 8, and 10 tokens so \(\mathrm{avgdl}=8\), and the term “refund” appears in two documents. Then \(\mathrm{idf}=\ln(1+2.5/2.5)=\ln 2\approx0.693\). For the first document, \(\mathrm{tf}=1\) and \(|d|=7\), so the denominator term is \(1.2\,(0.25+0.75\cdot7/8)=1.0875\) and the score is \(0.693\cdot2.2/(1+1.0875)\approx0.730\). For the fourth document, which repeats “refund” three times in ten tokens, \(1.2\,(0.25+0.75\cdot10/8)=1.425\) and the score is \(0.693\cdot6.6/(3+1.425)\approx1.034\). Three occurrences earn about 1.4 times the score of one, not three times.

The following program implements the formula with only the standard library, uses the four documents above, and prints the scores for three queries and the effect of \(k_1\).

import math
from collections import Counter

corpus = {
    "d1": "refund requests are processed within five days",
    "d2": "money back guarantee applies to annual plans",
    "d3": "shipping is free for orders over fifty dollars",
    "d4": "refund policy refund window and refund exceptions for annual plans",
}
docs = {k: v.split() for k, v in corpus.items()}
N = len(docs)
avgdl = sum(len(t) for t in docs.values()) / N
df = Counter(w for t in docs.values() for w in set(t))

def idf(term):
    n = df.get(term, 0)
    return math.log(1 + (N - n + 0.5) / (n + 0.5))

def bm25_term(term, tokens, k1=1.2, b=0.75):
    tf = tokens.count(term)
    norm = k1 * (1 - b + b * len(tokens) / avgdl)
    return idf(term) * tf * (k1 + 1) / (tf + norm)

def bm25(query, tokens, **kw):
    return sum(bm25_term(t, tokens, **kw) for t in query.split())

print(f"N={N}, avgdl={avgdl:.2f}, df(refund)={df['refund']}, idf(refund)={idf('refund'):.3f}, "
      f"idf(annual)={idf('annual'):.3f}")
for query in ["refund", "refund annual", "reimbursement"]:
    scores = {k: round(bm25(query, t), 3) for k, t in docs.items()}
    print(f"{query!r:18s}", scores)

print("tf saturation for 'refund' in d4 (tf=3) vs d1 (tf=1):")
for k1 in (0.0, 1.2, 3.0):
    print(f"  k1={k1}: d1={bm25_term('refund', docs['d1'], k1=k1):.3f} "
          f"d4={bm25_term('refund', docs['d4'], k1=k1):.3f}")

N=4, avgdl=8.00, df(refund)=2, idf(refund)=0.693, idf(annual)=0.693
'refund'           {'d1': 0.731, 'd2': 0.0, 'd3': 0.0, 'd4': 1.034}
'refund annual'    {'d1': 0.731, 'd2': 0.731, 'd3': 0.0, 'd4': 1.663}
'reimbursement'    {'d1': 0.0, 'd2': 0.0, 'd3': 0.0, 'd4': 0.0}
tf saturation for 'refund' in d4 (tf=3) vs d1 (tf=1):
  k1=0.0: d1=0.693 d4=0.693
  k1=1.2: d1=0.731 d4=1.034
  k1=3.0: d1=0.746 d4=1.267

The first query matches documents one and four, and document four wins because of its three occurrences, as computed by hand. The second query adds “annual,” which pulls in document two. The third query, “reimbursement,” scores zero everywhere. Document one says “refund requests are processed,” so a human sees the connection immediately, but BM25 sees no shared term. This is the vocabulary mismatch problem, and it is the reason to use learned representations. The last block shows saturation: raising \(k_1\) from 0 to 3 changes the ratio between a three-occurrence document and a one-occurrence document from 1.0 to about 1.7.

BM25 has strengths that matter in practice. It needs no training, so it works the day the corpus exists. It is exact on rare tokens such as error codes, part numbers, and names, because a rare term has high idf and a match is a match. Its scores are explainable term by term. Its weakness is the one just shown: it matches strings, not meaning, and it depends on the tokenizer to decide what a string is.

How an embedding model is trained to make retrieval work

A dense retriever uses an encoder that maps a text to one fixed-length vector, and scores a query-document pair by comparing the two vectors. In the common dual-encoder (or bi-encoder) design the query and the document pass through the encoder independently, which is what allows every document vector to be computed once and stored. Sentence-BERT by Reimers and Gurevych introduced this arrangement for BERT: the cost of finding the most similar pair in a collection of 10,000 sentences falls from about 65 hours with a pair-at-a-time BERT to about 5 seconds with precomputed embeddings. The encoder is a transformer as in the self-attention article; token outputs are reduced to one vector by taking a designated token’s output or averaging over tokens (mean pooling), and the result is usually normalized to unit length. A model that is trained to predict the next token, as described in the article on language-model objectives, does not produce vectors tuned for comparison; embedding models are trained with a separate objective whose goal is that related texts are close.

The contrastive objective

The objective most embedding models use is contrastive. Training data is a set of pairs \((q_i, d_i^+)\) where \(d_i^+\) is a text that should be retrieved for \(q_i\). Within a batch of \(B\) pairs, every other pair’s document serves as a negative for \(q_i\). Let \(s_{ij}\) be the similarity between the vector of \(q_i\) and the vector of \(d_j\), and let \(\tau>0\) be a temperature. The loss for query \(i\) is a softmax cross-entropy over the batch’s documents in which the correct class is \(d_i\):

\[ \mathcal{L}_i=-\log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^{B}\exp(s_{ij}/\tau)}. \]

This is the InfoNCE form that van den Oord, Li, and Vinyals introduced in contrastive predictive coding, and Karpukhin and colleagues use it for dense passage retrieval (DPR) with dot-product similarity. In words: the loss is small when the query is much more similar to its own document than to the other documents in the batch. The loss is a classification over a small candidate set, not a ranking metric; its connection to retrieval is that a model that can pick the right document out of a batch of distractors tends to rank the right document highly in a corpus.

The gradient has a simple form. Treat the scaled scores \(z_{ij}=s_{ij}/\tau\) as logits and let \(P_{ij}\) be the softmax over \(j\). Then \(\partial\mathcal{L}_i/\partial z_{ij}=P_{ij}-\delta_{ij}\), where \(\delta_{ij}\) is 1 when \(j=i\) and 0 otherwise. The positive is pulled closer with a force proportional to how far its probability is from 1, and each negative is pushed away in proportion to the probability the model currently gives it. Negatives that the model already ranks far below the positive contribute almost nothing; the ones that matter are the ones it confuses with the positive.

The temperature controls how much those confusable negatives dominate. A small \(\tau\) sharpens the softmax, so a negative that is only slightly less similar than the positive still receives a large share of the probability. The next program builds a three-pair batch with unit vectors at chosen angles, so cosine similarity is easy to read, and computes the loss at three temperatures. In the second pair, document 0 (cosine 0.707) is closer to the query than the pair’s own positive (cosine 0.643), and in the third pair document 1 (0.985) beats the positive (0.940), so the model gets two of the three pairs wrong.

import numpy as np

def softmax_rows(S):
    S = S - S.max(axis=1, keepdims=True)
    E = np.exp(S)
    return E / E.sum(axis=1, keepdims=True)

def info_nce(Q, D, tau):
    """Q, D: (B, dim) unit vectors; row i of D is the positive for row i of Q."""
    S = Q @ D.T / tau
    P = softmax_rows(S)
    return -np.log(np.diag(P)), P

def at_angle(deg):
    t = np.radians(deg)
    return np.array([np.cos(t), np.sin(t)])

Q = np.stack([at_angle(0), at_angle(60), at_angle(120)])
D = np.stack([at_angle(15), at_angle(110), at_angle(140)])
print("cosine matrix:\n", np.round(Q @ D.T, 3))
for tau in (1.0, 0.1, 0.02):
    per_row, P = info_nce(Q, D, tau)
    print(f"tau={tau:<5} mean loss={per_row.mean():.4f}  per-row={(np.round(per_row, 4) + 0.0)}  "
          f"row-1 probs={np.round(P[1], 3)}")

cosine matrix:
 [[ 0.966 -0.342 -0.766]
 [ 0.707  0.643  0.174]
 [-0.259  0.985  0.94 ]]
tau=1.0   mean loss=0.7378  per-row=[0.3697 0.9903 0.8535]  row-1 probs=[0.396 0.371 0.232]
tau=0.1   mean loss=0.6709  per-row=[0.     1.0687 0.944 ]  row-1 probs=[0.653 0.343 0.003]
tau=0.02  mean loss=1.8702  per-row=[0.     3.2553 2.3554]  row-1 probs=[0.961 0.039 0.   ]

Row 0 is an easy pair and its loss goes to zero as \(\tau\) falls. Rows 1 and 2 are the wrong pairs: the loss of row 1 grows from 0.99 at \(\tau=1\) to 3.26 at \(\tau=0.02\), and that of row 2 from 0.85 to 2.36, because a smaller temperature turns a small similarity deficit into a large probability deficit. The mean loss is therefore not monotone in \(\tau\); it depends on how many pairs the model already gets right. Training with a small temperature concentrates the gradient on the violations. The E5 model card notes that its cosine scores cluster between 0.7 and 1.0 because it was trained with temperature 0.01, and that the order of scores matters rather than their absolute values. That remark has a practical consequence for thresholds, discussed in the similarity section.

Where the positives and negatives come from

The pairs and negatives largely decide what the model treats as similar. Several sources of positives appear in published models.

• Labeled retrieval data. Question and answer-passage pairs, as in DPR, which trains on open-domain question answering data.
• Pairs from natural language inference data. SimCSE, by Gao, Yao, and Chen, uses entailment pairs as positives and contradiction pairs as hard negatives in its supervised variant; its unsupervised variant uses the same sentence twice with different dropout masks as the pair.
• Weakly supervised web pairs. E5, by Wang and colleagues, pretrains contrastively on a curated set of text pairs called CCPairs without labeled relevance data, and the authors report that it is the first model to beat BM25 on the BEIR benchmark in that setting.

Negatives come in three kinds. In-batch negatives add almost no encoding cost, because documents already in the batch are reused for every other query; larger batches give more of them. Hard negatives are documents that look relevant but are not. DPR adds a BM25-retrieved passage that does not contain the answer for each question and reports that this improves results over in-batch negatives alone. The third kind is a hazard: a false negative is a document in the batch that is actually relevant to the query, so the loss pushes the model to separate texts it should keep together. Random in-batch sampling produces false negatives whenever the data has many near-duplicate or same-topic pairs, and mining hard negatives from a retriever makes the problem worse, because the top-ranked non-positive passages are exactly where unlabeled positives hide.

Retrieval models also treat queries and documents asymmetrically. Queries are short and documents are long, so many models are trained with different input prefixes for the two roles. The E5 model card instructs users to prepend “query: ” and “passage: ” for retrieval tasks. A model used without the prefix it was trained with is operating off its training distribution, which is an easy mistake to make when swapping models and one that does not raise an error.

A trainable toy encoder

To see the objective work, the code below builds a small, fully synthetic setting that the rest of the article reuses. There are ten support-style concepts such as refund, shipping, and password. Each concept has four surface words (“refund,” “reimbursement,” “repayment,” “payback”). A text is one concept word plus filler words drawn from a small shared pool. Training sees only the first three surface words of each concept; the fourth is held out. Everything is generated by the script, so the data has no connection to any real corpus.

import random

CONCEPTS = {
    "refund":   ["refund", "reimbursement", "repayment", "payback"],
    "shipping": ["shipping", "delivery", "dispatch", "courier"],
    "password": ["password", "passcode", "credentials", "login"],
    "invoice":  ["invoice", "bill", "statement", "receipt"],
    "cancel":   ["cancel", "terminate", "withdraw", "revoke"],
    "warranty": ["warranty", "guarantee", "coverage", "protection"],
    "address":  ["address", "location", "residence", "destination"],
    "discount": ["discount", "coupon", "voucher", "rebate"],
    "account":  ["account", "profile", "membership", "registration"],
    "delay":    ["delay", "lateness", "postponement", "holdup"],
}
NAMES = list(CONCEPTS)
SEEN = (0, 1, 2)          # surface forms 0-2 appear in training; form 3 never does
FILLER = ["how", "to", "the", "your", "our", "can", "i", "of", "about", "please",
          "policy", "steps", "request", "explains", "details", "for"]

def make_text(rng, forms, n_filler):
    words = list(forms) + [rng.choice(FILLER) for _ in range(n_filler)]
    rng.shuffle(words)
    return words

def training_pairs(rng, n):
    pairs = []
    for _ in range(n):
        c = rng.choice(NAMES)
        a, b = rng.choice(SEEN), rng.choice(SEEN)
        pairs.append((c, make_text(rng, [CONCEPTS[a]], 3),
                         make_text(rng, [CONCEPTS[b]], 6)))
    return pairs

The encoder is deliberately minimal: a table of 16-dimensional word vectors, mean pooling over a text’s words, and normalization to unit length. Words that were never registered during training keep their random initial vector. The gradient is derived by hand so that nothing is hidden: with \(G=(P-I)/(B\tau)\) and the unit vectors stacked as rows of matrices \(Q\) and \(D\), the gradient with respect to the query vectors is \(GD\) and with respect to the document vectors is \(G^\top Q\); each then passes back through the normalization, which removes the component along the vector and divides by its pre-normalization length, and through the mean, which splits it equally over the words.

DIM, TAU = 16, 0.1

class Encoder:
    """Mean-pooled word embeddings, L2-normalized. Unknown words get a fixed random vector."""
    def __init__(self, seed=0):
        self.rng = np.random.default_rng(seed)
        self.vocab = {}
        self.E = np.zeros((0, DIM))
    def ids(self, words, grow=False):
        out = []
        for w in words:
            if w not in self.vocab:
                self.vocab[w] = len(self.vocab)
                self.E = np.vstack([self.E, self.rng.normal(0, 1 / np.sqrt(DIM), DIM)])
            out.append(self.vocab[w])
        return out
    def encode(self, words):
        ids = self.ids(words)
        u = self.E[ids].mean(axis=0)
        return u / np.linalg.norm(u)

def pool(E, id_lists):
    u = np.stack([E[ids].mean(axis=0) for ids in id_lists])
    n = np.linalg.norm(u, axis=1, keepdims=True)
    return u / n, n

def loss_and_grad(E, q_ids, d_ids, tau=TAU):
    q, qn = pool(E, q_ids); d, dn = pool(E, d_ids)
    S = q @ d.T / tau
    S -= S.max(axis=1, keepdims=True)
    P = np.exp(S); P /= P.sum(axis=1, keepdims=True)
    B = len(q_ids)
    loss = -np.mean(np.log(np.diag(P)))
    G = (P - np.eye(B)) / (B * tau)             # dLoss / dScore-before-temperature
    dq, dd = G @ d, G.T @ q                     # gradients w.r.t. the unit vectors
    grad = np.zeros_like(E)
    for x, xn, dx, id_lists in ((q, qn, dq, q_ids), (d, dn, dd, d_ids)):
        du = (dx - x * np.sum(x * dx, axis=1, keepdims=True)) / xn   # through normalization
        for ids, g in zip(id_lists, du):
            for i in ids:
                grad[i] += g / len(ids)                              # through the mean
    return loss, grad

A hand-derived gradient should be checked against a numerical one before it is trusted. The check perturbs each parameter by plus and minus \(10^{-6}\) and compares the central difference of the loss with the analytic gradient on a four-pair batch.

max |analytic - numeric| = 3.14e-10 (max |grad| = 9.65e-01)

The largest disagreement is about \(3\times10^{-10}\) against a largest gradient entry near 1, so the analytic gradient is correct on this batch. The setup uses temperature 0.1 and the Adam optimizer for 600 steps, in the training function below. One design choice in training deserves attention because it reproduces the false-negative problem. The script trains two encoders from the same data generator. One uses batches of ten pairs in which each of the ten concepts appears exactly once, so no in-batch negative is relevant to any query. The other uses ten randomly drawn pairs per batch, so two pairs of the same concept often land in one batch and each becomes the other’s false negative.

def train(batches, steps, lr=0.05, seed=0, enc=None):
    enc = enc or Encoder(seed)
    for b in batches:                       # register the vocabulary first
        for _, qw, dw in b:
            enc.ids(qw); enc.ids(dw)
    m = np.zeros_like(enc.E); v = np.zeros_like(enc.E)
    history = []
    for t in range(1, steps + 1):
        batch = batches[(t - 1) % len(batches)]
        q_ids = [enc.ids(qw) for _, qw, _ in batch]
        d_ids = [enc.ids(dw) for _, _, dw in batch]
        loss, g = loss_and_grad(enc.E, q_ids, d_ids)
        m = 0.9 * m + 0.1 * g; v = 0.999 * v + 0.001 * g * g
        enc.E -= lr * (m / (1 - 0.9 ** t)) / (np.sqrt(v / (1 - 0.999 ** t)) + 1e-8)
        history.append(loss)
    return enc, history

def random_batches(rng, n_batches, size):
    return [training_pairs(rng, size) for _ in range(n_batches)]

def distinct_concept_batches(rng, n_batches, query_forms=SEEN):
    out = []
    for _ in range(n_batches):
        batch = []
        for c in NAMES:
            a, b = rng.choice(query_forms), rng.choice(SEEN)
            batch.append((c, make_text(rng, [CONCEPTS[a]], 3),
                             make_text(rng, [CONCEPTS[b]], 6)))
        out.append(batch)
    return out

The encoders trained on the two batch designs are then evaluated on set A, a paraphrase task described in the next section; both are scored by nDCG@10, defined later.

final loss: distinct-concept batches 0.001, random batches 0.273
== distinct-concept batches
A: paraphrase, seen vocabulary (40 queries, 60 documents)
                 recall@10  MRR@10  nDCG@10
  BM25               0.117   0.224    0.107
  dense              1.000   1.000    1.000
  hybrid (RRF)       0.525   0.674    0.485
== random batches
A: paraphrase, seen vocabulary (40 queries, 60 documents)
                 recall@10  MRR@10  nDCG@10
  BM25               0.117   0.224    0.107
  dense              0.921   0.983    0.929
  hybrid (RRF)       0.454   0.648    0.420

The final training loss is 0.001 with distinct-concept batches and 0.273 with random batches. The random-batch loss cannot reach zero, because for some pairs the loss demands that a relevant document score below the pair’s own positive. The effect shows in retrieval: nDCG@10 on set A is 1.000 with clean batches and 0.929 with random ones. Real data cannot be arranged so that no batch contains a relevant negative, so the practical defenses are deduplicating and clustering the training pairs, and checking mined hard negatives for unlabeled positives.

Similarity: cosine, dot product, and distance

Three similarity functions appear in retrieval systems. Cosine similarity is \(\cos(u,v)=u\cdot v/(\|u\|\|v\|)\), the cosine of the angle between the vectors, as used for word vectors in the word-embeddings article. The dot product \(u\cdot v\) also depends on the lengths. Euclidean distance \(\|u-v\|\) is a distance rather than a similarity, so smaller is closer. The three are linked by the identity \(\|u-v\|^2=\|u\|^2+\|v\|^2-2\,u\cdot v\), which means that for unit vectors \(\|u-v\|^2=2-2\cos(u,v)\), and ranking by dot product, by cosine, and by ascending Euclidean distance give the same order. When an embedding model returns normalized vectors, which one a vector store uses is an implementation detail. When it does not, the three can disagree, and the model’s training objective says which one is right: DPR trains with dot product, and models trained with cosine should be searched with cosine or with normalized vectors.

The program compares one query against two documents. Document 1 points almost the same way as the query but is short. Document 2 points further away but is twice as long.

import numpy as np

def unit(v):
    v = np.asarray(v, dtype=float)
    return v / np.linalg.norm(v)

def at_angle(deg, norm=1.0):
    t = np.radians(deg)
    return norm * np.array([np.cos(t), np.sin(t)])

q = np.array([1.0, 0.0])
docs = {"d1 (10 deg, norm 1)": at_angle(10, 1.0),
        "d2 (40 deg, norm 2)": at_angle(40, 2.0)}

print(f"{'doc':22s} {'dot':>6s} {'cosine':>7s} {'|q-d|^2':>8s} {'2-2cos (unit)':>14s}")
for name, d in docs.items():
    cos = q @ d / (np.linalg.norm(q) * np.linalg.norm(d))
    l2sq = np.sum((q - d) ** 2)
    du = unit(d)
    print(f"{name:22s} {q @ d:6.3f} {cos:7.3f} {l2sq:8.3f} "
          f"{np.sum((q - du) ** 2):6.3f} = {2 - 2 * cos:5.3f}")

doc                       dot  cosine  |q-d|^2  2-2cos (unit)
d1 (10 deg, norm 1)     0.985   0.985    0.030  0.030 = 0.030
d2 (40 deg, norm 2)     1.532   0.766    1.936  0.468 = 0.468

Cosine ranks document 1 first (0.985 against 0.766), and so does Euclidean distance on the raw vectors, with squared distances of 0.030 and 1.936. The dot product ranks document 2 first (1.532 against 0.985) because its length multiplies the score. The last column confirms the identity: after normalizing the document vectors, \(\|q-d\|^2\) equals \(2-2\cos\) in both rows. Whether length is signal or noise depends on the model. A vector’s norm can carry information that cosine discards, and whether that information helps depends on the model and the task. The safe practice is to follow the metric the model was trained with and to normalize explicitly when unsure.

Scores from embedding models are not calibrated probabilities. As the E5 remark above shows, a model trained with a low temperature can return cosine values that are all high, so a fixed cutoff such as “keep results above 0.8” does not transfer between models, and it often does not transfer between corpora with the same model. Use the rank, or calibrate a cutoff on labeled queries from the actual corpus. Embeddings are also not comparable across models: vectors from two different models, or two versions of one model, live in different spaces even when they have the same dimension, which is why changing the embedding model means re-embedding the whole corpus (the operational side is covered in the article on document pipelines for retrieval).

Dimension, cost, and exact versus approximate search

Vector size sets storage and scan cost directly. One million chunks at 768 dimensions in 32-bit floats occupy \(10^6\times768\times4=3.072\) GB, and ten million occupy 30.72 GB before any index overhead. Storing 8-bit integers divides that by four, and keeping only the first 256 dimensions divides it by three. Truncation is only safe for models trained for it: Matryoshka Representation Learning by Kusupati and colleagues trains the leading coordinates to carry a coarse version of the representation, so that prefixes of the vector remain usable at lower cost. Truncating a vector from a model that was not trained this way discards information in an unpredictable way and has to be measured.

Searching every vector gives the exact top \(k\) but costs time proportional to \(N\). Approximate nearest-neighbor indexes, whether graph-based or built on clustering and compression, return a list that is usually, not always, the exact one. DPR uses the FAISS library for this. The cost of approximation is a loss of recall that has to be measured against the exact result on the same queries, and it is separate from the quality of the embedding itself. A retrieval evaluation that mixes the two cannot tell which one to fix.

Dense and sparse retrieval on the same queries

The toy setting can now compare retrievers. The corpus has three kinds of query, each testing one claim from the previous sections. The encoder is the one trained with distinct-concept batches. The sets are:

• A, paraphrase with seen vocabulary. Ten concepts, six documents each, every concept written with a single word in the corpus. Queries use a different seen word for the same concept, so BM25 has no content term in common with the relevant documents. All six documents of the query’s concept count as relevant.
• B, exact identifiers. Forty documents, each containing a unique identifier such as err1007 that never occurs in training, plus a concept word and filler. The query is the identifier and two filler words. One document is relevant.
• C, an unseen query word. The corpus of A, with queries that use the fourth surface word of a concept, which the encoder never saw in training.

The program computes BM25 and dense rankings and a fused ranking from reciprocal rank fusion (RRF). RRF scores each document by \(\sum_r 1/(c+\mathrm{rank}_r(d))\) over the input rankings, using ranks from 1 and \(c=60\), so only positions and not score magnitudes are combined. This matters because BM25 scores and cosine scores have unrelated scales. The metrics, defined fully in the evaluation section, are recall@10, mean reciprocal rank at 10, and nDCG@10. This is the code that computes them:

def ranking(scores):
    return list(np.argsort(-scores, kind="stable"))

def rrf(rankings, c=60):
    fused = {}
    for r in rankings:
        for pos, doc in enumerate(r, start=1):
            fused[doc] = fused.get(doc, 0.0) + 1.0 / (c + pos)
    return sorted(fused, key=lambda d: -fused[d])

# ---------- metrics (binary relevance) ----------
def recall_at(rank, rel, k):  return len(set(rank[:k]) & rel) / len(rel)
def mrr_at(rank, rel, k):
    for pos, d in enumerate(rank[:k], start=1):
        if d in rel: return 1.0 / pos
    return 0.0
def ndcg_at(rank, rel, k):
    dcg = sum(1 / math.log2(pos + 1) for pos, d in enumerate(rank[:k], start=1) if d in rel)
    ideal = sum(1 / math.log2(pos + 1) for pos in range(1, min(len(rel), k) + 1))
    return dcg / ideal

The random initial vectors and the random corpora make the numbers specific to this script and its seeds (Python 3.12, NumPy 2.5.3); they illustrate directions, not benchmark values. The results for the encoder trained with distinct-concept batches are:

A: paraphrase, seen vocabulary (40 queries, 60 documents)
                 recall@10  MRR@10  nDCG@10
  BM25               0.117   0.224    0.107
  dense              1.000   1.000    1.000
  hybrid (RRF)       0.525   0.674    0.485
B: exact identifiers never seen in training (40 queries, 40 documents)
                 recall@10  MRR@10  nDCG@10
  BM25               1.000   0.921    0.941
  dense              0.600   0.322    0.386
  hybrid (RRF)       0.950   0.562    0.655
C: query word never seen in training (40 queries, 60 documents)
                 recall@10  MRR@10  nDCG@10
  BM25               0.150   0.200    0.124
  dense              0.062   0.049    0.042
  hybrid (RRF)       0.142   0.152    0.111

Set A shows what dense retrieval buys. BM25 reaches an nDCG@10 of 0.107, close to what filler-word overlap yields by accident, because the query and the relevant documents share no content term. The dense encoder reaches 1.000, because training placed the synonyms of each concept together. Set B shows the opposite. The identifier is a token that training never touched, so the encoder holds only its random vector, and mean pooling dilutes that vector among the other words. Dense nDCG@10 is 0.386 while BM25 reaches 0.941, because the identifier is rare and its idf is high. Fusion lands between the two on both sets in recall (0.950 on B) and below the better retriever in nDCG.

Set C is the weakest case for both. The encoder has no vector worth anything for the unseen word, and BM25 has no shared term. Random ordering of 60 documents would give a recall@10 near 0.167 (ten of sixty positions), and both retrievers are at or below that level (BM25 0.150, dense 0.062). Hybrid retrieval does not repair a gap that neither leg covers. Only changing the encoder or the corpus can.

Combining retrievers is not automatically better

A single corpus holding both kinds of documents with both kinds of queries is the case for which hybrid retrieval is usually proposed. The program builds it as set D (the documents of A and B together, with the queries of A and B) and compares the fused ranking with each retriever, using a paired bootstrap over queries that is explained in the evaluation section.

D: mixed corpus and mixed queries (80 queries, 100 documents)
                 recall@10  MRR@10  nDCG@10
  BM25               0.527   0.557    0.525
  dense              0.713   0.472    0.546
  hybrid (RRF)       0.613   0.454    0.449
set A: hybrid - BM25  nDCG@10 diff = +0.378  95% CI [+0.326, +0.429]
set A: hybrid - dense nDCG@10 diff = -0.515  95% CI [-0.577, -0.452]
set B: hybrid - BM25  nDCG@10 diff = -0.287  95% CI [-0.390, -0.179]
set B: hybrid - dense nDCG@10 diff = +0.268  95% CI [+0.202, +0.336]
set D: hybrid - BM25  nDCG@10 diff = -0.076  95% CI [-0.172, +0.017]
set D: hybrid - dense nDCG@10 diff = -0.097  95% CI [-0.199, +0.006]

On the mixed set the fused ranking has an nDCG@10 of 0.449 against 0.525 for BM25 and 0.546 for dense, a difference whose 95% interval just includes zero for both comparisons. Equal-weight RRF gave the weak retriever the same vote as the strong one on each half of the queries. Fusion improved recall@10 over BM25 (0.613 against 0.527) but not the ordering. This is a feature of this toy, in which each retriever is weak on one half of the queries, and it shows what to check rather than a general verdict on hybrid retrieval: fusion needs evaluation on the real query mix, and weights, score normalization, or a reranker can change the outcome.

Learned sparse and late-interaction representations

Two other representations sit between the extremes. Learned sparse retrieval keeps the inverted index but lets a model produce the term weights. SPLADE, by Formal, Piwowarski, and Clinchant, uses a masked language model head to give each text weights over the vocabulary, with a sparsity regularizer that keeps most weights at zero and a log-saturation on the weights. A document can then be indexed under words it does not contain, such as “refund” for a page about reimbursement, and exact matches still count. The index stays an inverted index, so its engineering is familiar, but the expansion terms are learned and carry the same domain-shift risk as a dense model.

Late interaction keeps one vector per token rather than per text. ColBERT, by Khattab and Zaharia, encodes queries and documents separately, precomputes the document token vectors offline, and scores a pair by taking, for each query token, the maximum similarity to any document token and summing those maxima (MaxSim). The authors report a large reduction in per-query computation compared with running BERT on every pair while staying competitive in effectiveness. The cost is the index: a document of 200 tokens stores 200 vectors, not one. The small example below shows why the per-token view can distinguish what mean pooling cannot.

import numpy as np

def unit_rows(M):
    M = np.asarray(M, dtype=float)
    return M / np.linalg.norm(M, axis=1, keepdims=True)

def maxsim(Q, D):
    """Late interaction: each query token takes its best-matching document token; sum the maxima."""
    sim = unit_rows(Q) @ unit_rows(D).T        # (query tokens, document tokens)
    return sim.max(axis=1).sum(), sim

Q  = [[1, 0], [0, 1]]                           # two query token vectors
d1 = [[0.9, 0.1], [0.1, 0.9], [0.5, 0.5]]       # covers both query tokens
d2 = [[0.7, 0.7], [0.6, 0.8]]                   # near the diagonal only
for name, D in (("d1", d1), ("d2", d2)):
    score, sim = maxsim(Q, D)
    single_q = unit_rows(Q).mean(axis=0); single_d = unit_rows(D).mean(axis=0)
    cos_pooled = single_q @ single_d / (np.linalg.norm(single_q) * np.linalg.norm(single_d))
    print(f"{name}: MaxSim={score:.3f}  per-token maxima={np.round(sim.max(axis=1), 3)}  "
          f"mean-pooled cosine={cos_pooled:.3f}")

d1: MaxSim=1.988  per-token maxima=[0.994 0.994]  mean-pooled cosine=1.000
d2: MaxSim=1.507  per-token maxima=[0.707 0.8  ]  mean-pooled cosine=0.997

The two documents differ in MaxSim by a wide margin (1.988 against 1.507) because document 1 has a token close to each query token while document 2 has only tokens near the diagonal. The pooled cosine is 1.000 against 0.997, nearly the same, because averaging the token vectors discards which query token is matched by which document token. A single vector has to compress that structure into one point, and late interaction does not.

Domain mismatch: when the representation was trained on someone else’s text

A retrieval model captures the statistics of the data it was trained on. When the target corpus differs, the model’s notion of closeness may not match what the users of the corpus need. The differences that matter in practice fall into a few kinds.

• Vocabulary. Jargon, product names, abbreviations, and new coinages are rare or absent in training, so their vectors are unreliable. Set C reproduces this in a miniature way, with a word the encoder never saw.
• Identifiers. Error codes, SKUs, citation keys, and function names are tokens whose exact form matters. A dense encoder trained to place paraphrases together has no reason to separate ERR-4021 from ERR-4012. Set B reproduces this.
• Query style. A model trained on short questions against encyclopedia passages may do poorly on long pasted error logs, keyword queries, or conversational follow-ups.
• Document form. Tables, code, legal clauses, and multilingual text each need their own handling, and a model that never saw them embeds them by surface features.
• Notion of relevance. “Similar” is task-dependent. Two contract clauses can be topically the same and legally opposite; a model trained to put them together will do exactly that.

The BEIR benchmark by Thakur and colleagues made this measurable. It collects 18 retrieval datasets from different domains and tasks and evaluates models zero-shot, meaning without training on the target data. The authors find that BM25 is a robust baseline, that reranking and late-interaction models achieve the best average zero-shot results at high computational cost, and that dense and sparse retrieval models are cheaper but often underperform. MTEB by Muennighoff and colleagues widened the comparison to eight embedding tasks over 58 datasets and reports that no embedding method dominates across all tasks. A leaderboard score is therefore evidence about the benchmark’s domains, and a model has to be tested on the target corpus before it is trusted.

Remedies and their costs

Four responses are common, and each trades one cost for another.

• Keep a lexical leg. Exact-token behavior comes back, and set B shows why it matters. The cost is a second index and a fusion step that must be evaluated, as set D shows.
• Adapt the encoder. Continue training on in-domain pairs. These can come from click logs, from curated question and passage pairs, or from questions generated from the corpus’s own passages by a language model. Adaptation changes the vector space, so it triggers a full re-embedding, and it can degrade behavior elsewhere if the original distribution is neglected.
• Change the text, not the model. Expanding abbreviations, normalizing identifiers, or adding a title and section path to each chunk before embedding makes the input look more like what the model knows. This is cheap, and its effect is measurable on the evaluation set.
• Rerank. Retrieve a generous candidate set with the cheap representation and reorder it with a more accurate but more expensive scorer. This does not fix candidates that were never retrieved, so recall of the first stage sets its ceiling.

The toy run can show the second remedy. After the first training, the script continues training for 150 steps on batches in which the query word may be the previously unseen fourth word of the concept. Set C is then evaluated again.

== after adaptation (150 steps on pairs whose queries may use the new word)
A: paraphrase, seen vocabulary (40 queries, 60 documents)
                 recall@10  MRR@10  nDCG@10
  BM25               0.117   0.224    0.107
  dense              1.000   1.000    1.000
  hybrid (RRF)       0.504   0.716    0.480
C: query word never seen before adaptation (40 queries, 60 documents)
                 recall@10  MRR@10  nDCG@10
  BM25               0.150   0.200    0.124
  dense              0.642   0.595    0.590
  hybrid (RRF)       0.346   0.496    0.329

Dense nDCG@10 on set C moves from 0.042 to 0.590 after adaptation and recall@10 from 0.062 to 0.642, while set A stays at 1.000. The caveat is part of the lesson. Both the adaptation pairs and the evaluation queries are generated from the same small word pool, so some evaluation texts are likely to coincide with training texts. This measures whether the encoder learned the new word, and it does not measure how well adaptation generalizes to other new words. In a real project, adaptation data and evaluation queries must be split before training, by query and, where documents repeat, by source document. The BM25 numbers do not change, because the lexical index has no training to adapt, and the fused ranking improves only as far as the dense leg does.

Retrieval evaluation

Evaluation needs three things: a set of queries that resemble real ones, a record of which documents are relevant to each (the judgments, often called qrels), and metrics that match how the results are used. Building the set is the larger part of the work, and the metrics are the easier part.

Building the evaluation set

• Queries. Sample from real or realistic traffic and keep the mix: short keyword queries, full questions, identifiers, follow-ups. Reporting one averaged number hides the slices on which a retriever fails. The per-set tables above show this: the averaged result of set D conceals that each retriever is strong on one half.
• Judgments. Relevance can be binary or graded. Judgments are made for the top results of the systems being compared, and a document that was never judged is usually counted as non-relevant. This makes the set incomplete: a new retriever that finds good documents the old ones missed is penalized for it. Where the stakes are high, sample the unjudged documents that a new system retrieves and judge them.
• Granularity. The unit of relevance has to be the unit the system returns. If the index holds chunks, judgments must be at chunk level or mapped from documents by an explicit rule.
• Independence. Queries used to train or tune a model (or to pick fusion weights) cannot be used to evaluate it. The same holds for documents when a model is trained on generated questions about them.

Metrics

For a ranking of documents \(r_1,r_2,\dots\) and a set \(R\) of relevant documents for a query, four metrics are standard, each answering a different question.

• Recall@k \(=|R\cap\{r_1,\dots,r_k\}|/|R|\): the fraction of relevant documents that appear in the top \(k\). It measures whether the evidence is available to whatever consumes the list. If the consumer is a reranker or a language model, recall at the number of candidates it will see is the ceiling on what it can do.
• Precision@k \(=|R\cap\{r_1,\dots,r_k\}|/k\): how much of the list is useful, which matters when every irrelevant item costs tokens or reader attention.
• MRR@k: the mean over queries of \(1/\text{rank of the first relevant document}\), or 0 if none is in the top \(k\). It fits tasks where one good hit is enough.
• nDCG@k: discounted cumulative gain, \(\mathrm{DCG}@k=\sum_{i\le k}\mathrm{rel}_i/\log_2(i+1)\), divided by the best achievable DCG for that query. It rewards putting relevant documents early and handles graded relevance. The regression and ranking metrics article works through NDCG in more detail with a different example.

In the toy runs with binary relevance, the ideal DCG for a query with \(|R|\) relevant documents is \(\sum_{i=1}^{\min(|R|,k)}1/\log_2(i+1)\). For set B, with one relevant document, nDCG@10 equals \(1/\log_2(\text{rank}+1)\) when the document is in the top ten. A relevant document at rank 3 gives 0.5. A query with no relevant documents has no ideal DCG, so it needs an explicit rule (usually exclusion) before averaging.

The choice of \(k\) should come from the system that consumes the results. If the next stage reads the top five chunks, recall@5 describes what that stage can use. A generous \(k\) is not free: every extra chunk uses window space and attention cost and can dilute the evidence. Evaluate the retriever at the \(k\) the application will actually use, and also at a larger \(k\) to learn how much a reranker could recover.

Judging a difference

Averages over a modest number of queries are noisy, so a difference between two systems needs an uncertainty estimate. A paired bootstrap resamples the query set with replacement, recomputes the mean difference of a per-query metric on each resample, and reports the central 95% of the resampled differences. It is paired because both systems are scored on the same resampled queries, which cancels query difficulty. The earlier outputs include these intervals:

set A: hybrid - BM25  nDCG@10 diff = +0.378  95% CI [+0.326, +0.429]
set A: hybrid - dense nDCG@10 diff = -0.515  95% CI [-0.577, -0.452]
set B: hybrid - BM25  nDCG@10 diff = -0.287  95% CI [-0.390, -0.179]
set B: hybrid - dense nDCG@10 diff = +0.268  95% CI [+0.202, +0.336]
set D: hybrid - BM25  nDCG@10 diff = -0.076  95% CI [-0.172, +0.017]
set D: hybrid - dense nDCG@10 diff = -0.097  95% CI [-0.199, +0.006]

On set A the fused ranking is much worse than dense (−0.515, interval well below zero) and on set B much worse than BM25 (−0.287). On set D both intervals include zero, so with 80 queries these data do not establish that fusion is worse overall, although the point estimates are negative. The interval says how far the noise from the sample of queries alone can move the number. It does not cover other sources of error, such as a judgment set that is unrepresentative of real traffic.

def paired_bootstrap(a, b, rng, n=2000):
    a, b = np.asarray(a), np.asarray(b); diffs = []
    for _ in range(n):
        idx = [rng.randrange(len(a)) for _ in range(len(a))]
        diffs.append(np.mean(a[idx] - b[idx]))
    diffs.sort()
    return np.mean(a - b), diffs[int(0.025 * n)], diffs[int(0.975 * n)]

Evaluating the stages separately

Retrieval metrics describe the retriever. They do not describe the final answer produced from retrieved text, and a good retrieval score can coexist with a bad answer, and the other way round. Evaluate the retrieval step with its own labeled set so that an answer failure can be traced to either missing evidence or misuse of present evidence. Keep also the exact-versus-approximate comparison from the similarity section as a separate measurement: the recall of the approximate index against an exact scan with the same vectors tells how much recall the index costs, and recall of the exact scan against the labeled relevant documents tells how good the embedding is.

Public benchmarks have a role as a screen. BEIR and MTEB show how a model behaves across domains and tasks and are a reasonable way to shortlist candidates. They cannot stand in for a test set built from the target corpus, for the reasons in the domain-mismatch section, and a model may have been trained on data that overlaps a benchmark, so a leaderboard difference of a few points is weak evidence.

Failure patterns and what to check

The patterns below follow from the mechanisms above. Each row pairs a symptom with a likely cause and the check that separates it from the alternatives.

Symptom	Likely cause	Check
Exact identifiers are missed	Dense encoder blurs rare tokens; no lexical leg	Compare BM25 and dense on identifier queries; add an exact-match field or a lexical leg
Paraphrases are missed	Lexical-only retrieval; query and document vocabulary differ	Compare against a dense retriever on queries that share no terms with their relevant documents
Quality drops after a model upgrade	Query and document vectors from different models or versions; missing prefix	Confirm one model version for both sides and the prefix convention; re-embed the whole corpus
A fixed similarity cutoff stops working	Score distribution differs by model and corpus	Plot scores of relevant versus non-relevant pairs on labeled queries; choose the cutoff there
Fine-tuned model improves the new domain but regresses elsewhere	Adaptation data does not cover earlier query types	Keep a held-out set per query slice and report each
Results look different between staging and production	Approximate index recall, different normalization	Measure the approximate index against an exact scan on the same vectors
Fusion is worse than the best single retriever	Equal weights give a weak leg the same vote	Evaluate fusion per query slice with confidence intervals; tune weights on a split that is not the test set

An embedding model is a trained function, and it is only as good as the pairs and negatives behind it and the match between its training distribution and the corpus it serves. Its output is a point in a space whose geometry comes from the objective: similarity is whatever the training pairs defined it to be. Sparse term matching has the opposite profile, with exact matches on rare terms and no knowledge of meaning. Choosing between them, or combining them, is a measurement question on the target corpus and its queries, answered with a judged set, per-slice metrics, and an uncertainty estimate.

References

• Manning, C. D., Raghavan, P., and Schütze, H. Introduction to Information Retrieval, chapter on Okapi BM25 (Stanford NLP Group, online edition): https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html
• Karpukhin, V., et al. “Dense Passage Retrieval for Open-Domain Question Answering.” EMNLP 2020: https://arxiv.org/abs/2004.04906
• Reimers, N. and Gurevych, I. “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” EMNLP 2019: https://arxiv.org/abs/1908.10084
• van den Oord, A., Li, Y., and Vinyals, O. “Representation Learning with Contrastive Predictive Coding.” 2018: https://arxiv.org/abs/1807.03748
• Gao, T., Yao, X., and Chen, D. “SimCSE: Simple Contrastive Learning of Sentence Embeddings.” EMNLP 2021: https://arxiv.org/abs/2104.08821
• Wang, L., et al. “Text Embeddings by Weakly-Supervised Contrastive Pre-training” (E5), 2022: https://arxiv.org/abs/2212.03533; model card with usage guidance: https://huggingface.co/intfloat/e5-base-v2
• Kusupati, A., et al. “Matryoshka Representation Learning.” 2022: https://arxiv.org/abs/2205.13147
• Khattab, O. and Zaharia, M. “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.” SIGIR 2020: https://arxiv.org/abs/2004.12832
• Formal, T., Piwowarski, B., and Clinchant, S. “SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.” SIGIR 2021: https://arxiv.org/abs/2107.05720
• Thakur, N., et al. “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” 2021: https://arxiv.org/abs/2104.08663
• Muennighoff, N., et al. “MTEB: Massive Text Embedding Benchmark.” 2022: https://arxiv.org/abs/2210.07316