Embeddings and Retrieval Representations
Retrieval quality is set by the representation: BM25 matches terms, dense embeddings match what training taught them. Learn the objectives, the failure modes, and how to measure them.
-
-
Context Windows and Long-Context Behavior
A model can accept 200,000 tokens and still miss the clause that matters. Learn position encodings, attention costs, context construction, and how to measure effective length.
-
LLM Decoding and Structured Output
A language model outputs a distribution, and the decoder picks the tokens. Learn temperature and top-p, token masking for structured output, and how to evaluate it.
-
LLM Pretraining and Scaling Laws
One objective, an enormous corpus, and a compute budget you have to allocate correctly the first time.
-
Language Model Objectives and Limits
A language model learns the conditional distribution of the next token. That one objective explains both its range and why its answers are not reliably correct.