Chunking
A passage retrieved without its heading can say the opposite of what the document means. How documents are split decides what search can find.
-
-
Representation Shift
Neighbor overlap falls to 0.100 while the unbiased embedding MMD estimate remains near zero, in the nearly rotationally symmetric example with old-index/new-query embeddings.
-
Cosine Similarity
Cosine compares vector directions. Normalization changes distance rankings, coordinate scaling changes meaning, and retrieval thresholds need evaluation.
-
Embedding Similarity
The dot product ties two candidates at 10.0000 that cosine separates at 1.0000 and 0.9806. The metric is not a formatting choice.
-
Natural Language Processing (NLP)
NLP connects language representations to tasks such as classification, extraction, retrieval, and generation. Context, data, and evaluation shape reliability.
-
Building a Retrieval Index with Traceable Evidence
A retrieval index is derived data. Give every chunk an exact source span, name each index build, update approximate indexes without silent recall loss, and test for stale evidence.
-
Embeddings and Retrieval Representations
Retrieval quality is set by the representation: BM25 matches terms, dense embeddings match what training taught them. Learn the objectives, the failure modes, and how to measure them.
-
Case: Data Platform Migration for AI Consumers
A fictional distributor replaces its warehouse while a search index and a prediction model depend on the old outputs. Contracts, dual running, reconciliation, and a coded cutover rule decide when each consumer moves.