Context Windows and Long-Context Behavior

Consider a hypothetical assistant that answers questions about a 300-page contract. The vendor’s documentation says the model accepts 200,000 tokens, and the contract fits. A user asks which party bears the cost of late delivery. The clause is on page 140. The model answers confidently from the indemnity section on page 12 and never mentions the clause that overrides it. Nothing failed in the sense of an error: the request was within limits, the answer is fluent, and the correct text was in the input.

The gap between fits in the window and is used correctly is the subject of this article. A context window is a hard limit with an arithmetic you can plan around, but it is also the outer boundary of a behavior that changes with length, with the position of the evidence, and with how the context was assembled. Four things determine what a long input buys you: how the model represents position, what attention costs as the input grows, how the context is constructed, and whether the evidence should be in the window at all or fetched on demand. The last section turns these into a test you can run on your own model and documents.

The article builds on Decoding: Sampling, Constraints, and Structured Output, which treats generation as repeatedly choosing a next token given everything so far. That framing already implies the first fact used here: each generated token is appended to the sequence and becomes part of the context for the next step, so output and input compete for the same window. Self-attention and causal masking are assumed from Transformers from Scratch: Self-Attention to Encoder–Decoder, and tokens from Tokenization Explained: BPE, WordPiece, and SentencePiece.

What the window limit is, and what it is not

A language model processes a sequence of at most \(N\) tokens, where \(N\) is fixed by the model and its serving configuration. The context window is that maximum. Everything the model conditions on counts against it: the system instructions, retrieved passages, conversation history, tool results, the user’s question, and the tokens the model generates. For a request with \(P\) prompt tokens and a maximum of \(G\) generated tokens, the constraint is

\[P + G \le N.\]

Services often add separate caps, such as a maximum output length that is smaller than \(N – P\), so the binding limit is the smaller of the two. Treat \(G\) as something you reserve in advance rather than discover. If the prompt is built to the window with no reserve, the generation is cut off or the request is rejected, depending on the service.

Token counts are tokenizer-specific. The same contract is a different number of tokens under two tokenizers, and a character or word count only approximates either. Count with the tokenizer of the model you call, and keep the count in the request-building code so that the budget is computed before the call, not inferred from an error afterward.

Three distinctions matter for the rest of the article.

• Architectural limit versus trained length. A model is trained on sequences up to some length. Position representations can often be evaluated at longer positions, but behavior beyond the trained length is not guaranteed. The advertised window is the length the provider supports, usually after extra training for long sequences.
• Supported length versus effective length. A model can accept \(N\) tokens and still use information in them unevenly. The later sections show how this is measured.
• Window versus memory. A chat product that stores a long conversation may send only part of it. What is stored is an application property. What the model sees is the assembled prompt.

How position enters the model

Self-attention treats its input as a set. Without extra information, permuting the input rows permutes the output rows, and the model cannot tell “dog bites man” from “man bites dog” except through whatever else breaks the symmetry. Causal masking does break it somewhat (a token sees only earlier tokens, so the first token’s output depends on one token and the last on all of them), but that is a weak and uneven signal. Real models inject position explicitly. The way they do it determines how they behave at lengths they were not trained on, which is why the design choices below matter for long context.

Absolute positions

The original Transformer of Vaswani et al. adds a fixed sinusoidal vector for each position index to the token embedding. Many later models instead learn a table with one row per position up to a maximum length. Both inject position into the input of the first layer and let the model decode it later. A learned table has no row for position \(N+1\), so it cannot be used past its size without modification, and rows near the end of the table may have been trained on few examples if most training sequences were shorter. Sinusoids can be evaluated at any position, but the model has only seen the values at positions it trained on.

Relative positions and distance biases

What usually matters for language is how far apart two tokens are, not their absolute index. Shaw et al. add learned relative-position terms inside the attention computation, so a score between positions \(i\) and \(j\) depends on \(i – j\). ALiBi (Press et al.) goes further and adds no position embeddings at all. It subtracts a penalty proportional to distance from each attention score:

\[ s_{ij} = \frac{q_i^\top k_j}{\sqrt{d}} – m_h\,(i – j), \qquad j \le i, \]

where \(q_i\) and \(k_j\) are the query and key vectors, \(d\) their width, and \(m_h\) a fixed slope for attention head \(h\), chosen from a geometric sequence so that different heads have different effective reach. Press et al. report that a model trained on sequences of 1,024 tokens extrapolated to 2,048 and matched one trained at the longer length. The penalty is a fixed linear function, so a larger distance simply gets a larger penalty and nothing about it is unfamiliar at test time, but the model still has to use the far tokens well, which is a separate question.

Rotary position embedding

Rotary position embedding (RoPE), introduced by Su et al. in RoFormer, is the position scheme in many current open models, and the one most long-context extension methods modify. It does not add anything to the embeddings. It rotates the query and key vectors by an angle that depends on position.

Split a query or key vector of width \(d\) into \(d/2\) coordinate pairs. For position \(m\), rotate pair \(i\) by the angle \(m\theta_i\), where \(\theta_i = b^{-2i/d}\) and \(b\) is a base, commonly 10,000. Write \(R_m\) for the block-diagonal matrix of these rotations. The rotated vectors are \(\tilde q_m = R_m q\) and \(\tilde k_n = R_n k\). Their dot product is

\[ \tilde q_m^\top \tilde k_n = q^\top R_m^\top R_n\, k = q^\top R_{n-m}\, k . \]

The identity holds because rotations compose by adding angles and \(R_m^\top = R_{-m}\). In words: for fixed content vectors \(q\) and \(k\), the score depends on the two positions only through their difference \(n – m\). The model gets relative-position behavior from an operation applied to absolute positions. Values are not rotated in the usual formulation, so the rotation only shapes which tokens attend to which.

The following program implements the rotation for an 8-dimensional vector and checks the identity. The query and key are arbitrary fixed numbers, not model weights. Its run is Python 3.12 with the standard library only.

import math

def rotate(vec, pos, base=10000.0):
    """Apply rotary embedding: rotate each consecutive coordinate pair by pos * theta_i."""
    d = len(vec)
    out = []
    for i in range(d // 2):
        theta = base ** (-2 * i / d)
        a = pos * theta
        x, y = vec[2 * i], vec[2 * i + 1]
        out.append(x * math.cos(a) - y * math.sin(a))
        out.append(x * math.sin(a) + y * math.cos(a))
    return out

def dot(u, v):
    return sum(a * b for a, b in zip(u, v))

q = [0.5, -1.0, 0.8, 0.3, -0.2, 0.9, 0.4, -0.6]
k = [1.0, 0.2, -0.5, 0.7, 0.3, -0.8, 0.6, 0.1]

print("score(q at m, k at n) for several (m, n)")
for m, n in [(10, 4), (110, 104), (5010, 5004), (10, 7), (110, 107)]:
    print(f"m={m:5d} n={n:5d} offset m-n={m-n}  score={dot(rotate(q, m), rotate(k, n)):.6f}")

print()
print("same content, no rotation: score =", round(dot(q, k), 6))
print("score vs offset (q at 1000):")
for off in [0, 1, 2, 4, 8, 16, 64, 256]:
    print(f"  offset {off:4d}: {dot(rotate(q, 1000), rotate(k, 1000 - off)):.4f}")

score(q at m, k at n) for several (m, n)
m=   10 n=    4 offset m-n=6  score=-0.378019
m=  110 n=  104 offset m-n=6  score=-0.378019
m= 5010 n= 5004 offset m-n=6  score=-0.378019
m=   10 n=    7 offset m-n=3  score=-0.715210
m=  110 n=  107 offset m-n=3  score=-0.715210

same content, no rotation: score = -0.49
score vs offset (q at 1000):
  offset    0: -0.4900
  offset    1: 0.3689
  offset    2: 0.2290
  offset    4: -1.5293
  offset    8: 0.8185
  offset   16: -0.4899
  offset   64: 0.5375
  offset  256: -0.0941

The first block shows that offset 6 gives the same score at positions (10, 4), (110, 104), and (5010, 5004). Moving both tokens by the same amount changes nothing, so the score carries no absolute position. The unrotated dot product is \(-0.49\), which is what the score would be if position were ignored. The second block fixes the query at position 1000 and sweeps the offset. The score oscillates with offset, because each coordinate pair contributes a cosine of its own rotation angle. What a trained model gets from this is a family of oscillations at different frequencies that it can combine into distance-sensitive behavior, which is the reason \(\theta_i\) spans many orders of magnitude.

A model can also work with no explicit position

Kazemnejad et al. compare positional schemes for length generalization on reasoning and arithmetic tasks and find that a decoder with no explicit positional encoding generalized to longer inputs at least as well as the explicit schemes in their setting. A causal decoder can in principle infer some position from the causal mask itself, because a token’s attention is spread over exactly the tokens before it. The result holds for the tasks and model sizes they tested, and it does not mean that production models can drop position. It does show that position handling is a design variable with measurable effects rather than a settled detail.

Extending a trained model to longer inputs

Suppose a RoPE model was trained at 4,096 tokens and you want 16,384. The obvious step, evaluating the rotation at positions up to 16,383, tends to fail. To see why, look at what each coordinate pair experiences. Pair \(i\) rotates with wavelength \(2\pi / \theta_i\) tokens. High-frequency pairs, with wavelengths of a few to a few hundred tokens, complete many full turns within the training length, so every phase they will ever show was seen in training. Low-frequency pairs have wavelengths longer than the training length. They never completed a turn, so positions beyond the training length produce angles the model has never seen on those pairs.

import math

d, base = 64, 10000.0
train, target = 4096, 16384
scale = train / target

print(f"trained length {train}, target length {target}, interpolation factor {scale}")
print("pair  wavelength  turn done in training?  unseen angle at target (rad)  step per token: raw -> interpolated")
for i in [0, 4, 8, 16, 24, 28, 31]:
    theta = base ** (-2 * i / d)
    wavelength = 2 * math.pi / theta
    complete = wavelength <= train
    # Slow pairs never finish a turn in training, so angles beyond train*theta are new.
    unseen = 0.0 if complete else (target - train) * theta
    print(f"{i:4d}  {wavelength:10.1f}  {'yes' if complete else 'no':>20}  {unseen:24.3f}  {theta:.2e} -> {theta*scale:.2e}")

trained length 4096, target length 16384, interpolation factor 0.25
pair  wavelength  turn done in training?  unseen angle at target (rad)  step per token: raw -> interpolated
   0         6.3                   yes                     0.000  1.00e+00 -> 2.50e-01
   4        19.9                   yes                     0.000  3.16e-01 -> 7.91e-02
   8        62.8                   yes                     0.000  1.00e-01 -> 2.50e-02
  16       628.3                   yes                     0.000  1.00e-02 -> 2.50e-03
  24      6283.2                    no                    12.288  1.00e-03 -> 2.50e-04
  28     19869.2                    no                     3.886  3.16e-04 -> 7.91e-05
  31     47117.2                    no                     1.639  1.33e-04 -> 3.33e-05

The program uses a head width of 64 and base 10,000. Pairs 0 to 16 complete a turn inside 4,096 tokens, and the “unseen angle” column is zero for them. Pairs 24, 28, and 31 do not, and at 16,384 tokens they present angles up to 12.3, 3.9, and 1.6 radians beyond anything seen during training. The final column anticipates the next paragraph. The low-frequency pairs are where long-distance information lives, and they are where evaluation at new positions goes out of distribution.

Chen et al. propose position interpolation: instead of evaluating at position \(m\), evaluate at \(m \cdot L/L’\), where \(L\) is the trained length and \(L’\) the target. All angles then stay inside the range seen in training, and the paper reports that a short fine-tuning run, on the order of a thousand steps, extends LLaMA models up to 32,768 tokens. The cost is visible in the last column of the table. With a factor of 0.25, adjacent tokens differ by a quarter of the rotation they used to, including on the high-frequency pairs that the model uses to tell neighbors apart. Squeezing positions makes nearby tokens harder to distinguish, and fine-tuning has to repair that.

Later methods treat frequencies differently. YaRN (Peng et al.) interpolates frequency bands by different amounts, leaving high-frequency components alone and scaling the slow ones, and adjusts the attention logits with a temperature term. The authors report that it reaches a given extension with less fine-tuning than earlier methods. The common structure is a trade among three things: keep angles in the trained range, keep neighbor resolution, and limit how much further training is needed. A model extended this way is a different model from one trained at the long length from the start. It can handle the length, but it may rank behaviors at long range differently, and the benchmark results at the original length should be rechecked after extension.

What attention costs as the window grows

Position schemes decide whether long inputs are meaningful. Attention decides whether they are affordable. The details of the KV cache, FlashAttention, and batching are in LLM Inference Optimization. Here the question is how cost scales with the window and when attention starts to dominate.

Compute

Per layer and per token, the projections and feed-forward layers cost a number of operations independent of sequence length. For a dense layer with model width \(d\) and a feed-forward width of \(4d\), that is about \(24d^2\) floating-point operations (counting a multiply-add as two): \(8d^2\) for the four attention projections and \(16d^2\) for the two feed-forward matrices. Computing scores and the weighted sum of values for one token that attends to \(t\) earlier tokens costs about \(4dt\). During prefill, token \(t\) attends to \(t\) tokens, so the average over a prompt of length \(T\) is about \(2dT\) per token. Setting this equal to \(24d^2\) gives a crossover at \(T = 12d\). Below it, the projections and feed-forward layers dominate prefill compute. Above it, attention does. Total attention work grows with \(T^2\), so doubling the prompt quadruples it.

# Per-layer forward FLOPs per token for a dense decoder, 2 FLOPs per multiply-add.
d = 4096            # model width
ffn = 4 * d         # feed-forward width
proj = 2 * (4 * d * d)          # Q, K, V, O projections
mlp = 2 * (2 * d * ffn)         # up and down projections
linear = proj + mlp             # = 24 d^2
print(f"linear FLOPs per token per layer: {linear/1e6:.1f} M (= 24 d^2)")

def attn(T):
    # average causal position is about T/2: QK^T costs 2*d per key, AV another 2*d per key
    return 4 * d * (T / 2)

print()
print("   T    attention/linear FLOP ratio (prefill average)")
for T in [1_000, 4_000, 16_000, 32_000, 128_000, 1_000_000]:
    print(f"{T:9,d}   {attn(T)/linear:6.2f}")
print(f"ratio reaches 1 at T = {2*linear/(4*d):,.0f} (= 12 d)")

print()
# total prefill attention FLOPs grow with T^2; linear FLOPs grow with T
for T in [8_000, 16_000, 32_000, 64_000]:
    tot_attn = 4 * d * T * T / 2
    tot_lin = linear * T
    print(f"T={T:6,d}: prefill attention {tot_attn/ (tot_attn+tot_lin):5.1%} of per-layer FLOPs; doubling T multiplies attention by 4")

print()
# KV cache per sequence: 2 * layers * kv_heads * head_dim * T * bytes
layers, head_dim, bytes_ = 32, 128, 2
for kv_heads in [32, 8]:
    row = []
    for T in [8_192, 32_768, 131_072]:
        gib = 2 * layers * kv_heads * head_dim * T * bytes_ / 2**30
        row.append(f"T={T:,d}: {gib:5.1f} GiB")
    print(f"KV heads {kv_heads:2d}: " + "   ".join(row))

linear FLOPs per token per layer: 402.7 M (= 24 d^2)

   T    attention/linear FLOP ratio (prefill average)
    1,000     0.02
    4,000     0.08
   16,000     0.33
   32,000     0.65
  128,000     2.60
1,000,000    20.35
ratio reaches 1 at T = 49,152 (= 12 d)

T= 8,000: prefill attention 14.0% of per-layer FLOPs; doubling T multiplies attention by 4
T=16,000: prefill attention 24.6% of per-layer FLOPs; doubling T multiplies attention by 4
T=32,000: prefill attention 39.4% of per-layer FLOPs; doubling T multiplies attention by 4
T=64,000: prefill attention 56.6% of per-layer FLOPs; doubling T multiplies attention by 4

KV heads 32: T=8,192:   4.0 GiB   T=32,768:  16.0 GiB   T=131,072:  64.0 GiB
KV heads  8: T=8,192:   1.0 GiB   T=32,768:   4.0 GiB   T=131,072:  16.0 GiB

For \(d = 4096\) the crossover is at about 49,000 tokens. At 8,000 tokens attention is 14% of per-layer prefill operations, at 32,000 it is 39%, and at 64,000 it is 57%. This counts arithmetic only. Wall-clock time also depends on memory traffic, and the quadratic term is what attention kernels such as FlashAttention (Dao et al.) restructure to reduce memory reads and writes without changing the result. They do not remove the quadratic arithmetic.

Memory

During generation, the keys and values of all previous tokens are kept so that they are not recomputed. This cache grows linearly with the length and with the number of sequences served at once. The last block of the output uses the formula from the inference article, \(2 \cdot \text{layers} \cdot H_{KV} \cdot d_{\text{head}} \cdot T \cdot \text{bytes}\), for a 32-layer model with 128-wide heads and 16-bit entries. With 32 key/value heads the cache for one sequence is 4 GiB at 8,192 tokens and 64 GiB at 131,072. Grouping key/value heads, as in grouped-query attention, divides it by the grouping factor, which is why the second row is a quarter of the first at the same lengths. A window that is cheap to support can be expensive to serve to many users at once, because each concurrent request holds its own cache.

Latency

The time to the first token is dominated by prefill, which grows faster than linearly with prompt length once attention matters. Per-token generation speed slows as the cache grows because each step reads all of it. If your product sends the same long document with every question, the repeated prefill is the cost. Services that offer context caching reuse the processed prefix when later requests begin with identical content, which changes the economics of resending a fixed document; the cost model later in this article shows how much.

Attention variants that cut the quadratic term

Several designs reduce what each token attends to. Longformer (Beltagy et al.) combines local windowed attention with a few globally attending tokens, so cost grows linearly with length. Sliding-window attention in decoders keeps only the most recent tokens’ keys and values. This bounds memory, but a token outside the window has no direct path to the current token; information can reach it only indirectly through the stacked layers. Xiao et al. observe that models assign unusually large attention weight to the first few tokens whatever their content (they call these attention sinks), and that window attention fails when these initial tokens are evicted but recovers if they are kept. Their StreamingLLM keeps the sink tokens plus a recent window and runs indefinitely without retraining. The result is stable generation, not recall of evicted text: a fact that scrolled out of the window is gone.

Why a longer window can make evidence harder to use

Fitting in the window and being used are different properties, and the difference has at least two sources. One is statistical, and one is empirical.

Attention has to be sharper as length grows

Attention weights are a softmax over all visible keys. Suppose one key carries the evidence and receives a score that exceeds the score of every other key by a gap \(\Delta\), while the other \(T-1\) keys all have the same score. The weight on the evidence is

\[ w = \frac{e^{\Delta}}{e^{\Delta} + (T-1)} . \]

Solving \(w = 1/2\) gives \(\Delta = \ln(T-1)\). The score gap needed to keep half the attention mass on the evidence grows with the logarithm of the context length. A model that has learned a gap of 5 is decisive at 100 tokens and nearly blind at 100,000.

import math

def weight_on_needle(T, gap):
    """One key has logit `gap` above T-1 distractors that all have logit 0."""
    e = math.exp(gap)
    return e / (e + (T - 1))

print("gap needed for the needle to get 50% of attention: ln(T-1)")
for T in [100, 1_000, 10_000, 100_000, 1_000_000]:
    print(f"T={T:9,d}  gap for 50% = {math.log(T-1):5.2f}   weight at fixed gap 5: {weight_on_needle(T,5):.4f}")

def entropy_uniform_vs_peaked(T, gap):
    w = weight_on_needle(T, gap)
    rest = (1 - w) / (T - 1)
    return -(w * math.log(w) + (T - 1) * rest * math.log(rest))
print()
for T in [1_000, 100_000]:
    print(f"T={T:,d}: entropy at gap 5 = {entropy_uniform_vs_peaked(T,5):.2f} nats (uniform = {math.log(T):.2f})")

gap needed for the needle to get 50% of attention: ln(T-1)
T=      100  gap for 50% =  4.60   weight at fixed gap 5: 0.5999
T=    1,000  gap for 50% =  6.91   weight at fixed gap 5: 0.1293
T=   10,000  gap for 50% =  9.21   weight at fixed gap 5: 0.0146
T=  100,000  gap for 50% = 11.51   weight at fixed gap 5: 0.0015
T=1,000,000  gap for 50% = 13.82   weight at fixed gap 5: 0.0001

T=1,000: entropy at gap 5 = 6.40 nats (uniform = 6.91)
T=100,000: entropy at gap 5 = 11.51 nats (uniform = 11.51)

At a fixed gap of 5, the weight on the single relevant key falls from 0.60 at \(T = 100\) to 0.13 at \(T = 1{,}000\) and 0.0015 at \(T = 100{,}000\). The gap needed for 50% grows from 4.6 to 11.5 over the same range. The entropy rows say the same thing in another form: at 100,000 keys, the attention distribution under this gap is uniform to two decimals.

This is a toy. It has one head, one layer, and identical distractors. Real models have many heads, can route information over several layers, and learn sharper score distributions when trained on long sequences, so the toy does not predict a particular accuracy. It does explain why long-context ability is something training must supply, not something a larger window provides automatically, and why distractors that resemble the evidence (similar scores, not zero) are harder than random filler.

Position-dependent use of the context

Liu et al. measured how models use information at different positions in the input in two settings: multi-document question answering, where the document that contains the answer is moved among distractor documents, and synthetic key-value retrieval. They report that performance is often highest when the relevant information is at the beginning or end of the input and drops substantially when it is in the middle, producing a U-shaped curve, and that models with larger windows did not remove the effect in their tests. The finding is about the models and settings they tested. Other models and tasks can show different curves, which is why the last section gives a procedure for measuring it on your own system rather than relying on the published shape.

The reasons are not settled. Plausible contributors include the sink-like attention to initial tokens, which makes early positions easy to attend to, recency effects from training on text where nearby tokens are most predictive, and the dilution effect above. For design, the causes matter less than the consequence: the position of a fact relative to other content is an input variable that can change accuracy, and your context-construction code decides it.

Constructing the context

Context construction is the code that turns a question and a pool of candidate material into the prompt. It has four jobs: fit the budget, decide what gets in, decide where it goes, and keep what is stable stable.

Budget accounting

Start from the constraint \(P + G \le N\) and subtract fixed costs before spending on evidence. The budget for retrieved material is \(N – G_{\text{reserve}} – \text{system} – \text{question}\). Two policies need to be explicit. First, what to do when the material exceeds the budget: truncate silently, drop the lowest-ranked items, summarize, or fail with an error. Silent truncation is the worst choice, because the first sign is a wrong answer. Second, what a trimmed item looks like: cutting a passage mid-sentence can remove the exception that changes its meaning, so drop whole passages or trim at structural boundaries.

The following program builds a prompt from scored chunks with a budget, a greedy selection by relevance, and three orderings. It uses a crude token counter (about 1.3 tokens per whitespace-separated word) as a stand-in for a real tokenizer, and the chunks are placeholder text, so only the selection and ordering logic is being demonstrated.

from dataclasses import dataclass

def count_tokens(text):
    # Stand-in tokenizer: roughly 1.3 tokens per whitespace-separated word.
    return int(len(text.split()) * 1.3) + 1

@dataclass
class Chunk:
    id: str
    text: str
    score: float   # retriever relevance, higher is better

def build_context(system, question, chunks, window, reserve_output, order="edges"):
    budget = window - reserve_output - count_tokens(system) - count_tokens(question)
    kept, used = [], 0
    for c in sorted(chunks, key=lambda c: -c.score):      # greedy by relevance
        t = count_tokens(c.text)
        if used + t <= budget:
            kept.append(c); used += t
    if order == "edges":     # best chunks at both ends, weakest in the middle
        left, right = [], []
        for i, c in enumerate(kept):          # kept is sorted best-first
            (left if i % 2 == 0 else right).append(c)
        kept = left + right[::-1]
    elif order == "ascending":   # best chunk last, next to the question
        kept = kept[::-1]
    prompt = system + "\n\n" + "\n\n".join(f"[{c.id}] {c.text}" for c in kept) + "\n\n" + question
    return prompt, kept, budget, used

chunks = [Chunk(f"c{i}", "word " * n, s) for i, (n, s) in
          enumerate([(300, .91), (250, .88), (400, .75), (200, .70), (350, .62), (300, .55)])]
for order in ["relevance", "edges", "ascending"]:
    prompt, kept, budget, used = build_context(
        "You answer from the passages.", "What is the refund window?",
        chunks, window=1500, reserve_output=300, order=order)
    print(f"{order:9s} budget={budget} used={used} order={[c.id for c in kept]} prompt_tokens={count_tokens(prompt)}")

relevance budget=1186 used=978 order=['c0', 'c1', 'c3'] prompt_tokens=992
edges     budget=1186 used=978 order=['c0', 'c3', 'c1'] prompt_tokens=992
ascending budget=1186 used=978 order=['c3', 'c1', 'c0'] prompt_tokens=992

The window is 1,500 tokens with 300 reserved for output, which leaves a 1,186-token budget after the system prompt and question. Chunk c2 has a high score (0.75) but needs about 520 tokens, which would overflow the budget after c0 and c1, so greedy selection skips it and takes the smaller c3. This is a policy decision worth noticing: relevance order was not strictly followed, and whether you prefer the skipped chunk or the small one depends on your task. The three orderings differ only in placement. With “edges”, the weakest selected chunk (c3) sits in the middle and the strongest are at the start and end, which is the arrangement the position findings above would suggest. “Ascending” puts the best chunk last, next to the question. Which ordering is best is an empirical question for your model, and the harness in the last section is how to answer it.

What goes in and where

• Deduplicate and remove near-copies. Repeated passages spend budget and, through the dilution effect, make the evidence harder to find.
• Keep each item self-describing. A passage extracted from a table or a section loses the heading that qualifies it. Label items with source and section so that the model can cite and so a later check can map an answer back to a passage.
• Put instructions where they will be followed. Instructions at the start and a restatement of the task near the question are common practice. Test both positions with long inputs, because the effect depends on the model.
• Manage history deliberately. In a conversation, resending everything grows without bound. Options are a sliding window of recent turns, a running summary, or retrieval over earlier turns. A summary can drop exactly the constraint that later matters, so keep user-stated requirements verbatim, not just in a summary.
• Mark untrusted content as data. Retrieved pages and tool outputs can contain text that tries to give the model instructions. Delimiting content does not make injection impossible. See prompt injection for the attack and Designing Context and Prompt Interfaces for context design as an application interface.

Stable prefixes

Prompt caching works on the beginning of the prompt: a later request benefits only if its start is identical to an earlier one. This pulls in the opposite direction from ordering by relevance, because relevance order changes with every question. A common layout puts the stable material (instructions, a fixed reference document) first and the per-question material after it. If a fixed document must be in the prompt and the question changes, the cacheable prefix is the document, and the ordering experiments apply only to the variable part. The exact matching rules, minimum lengths, lifetimes, and prices are provider-specific and change; read the current documentation of the service you call.

Retrieval or long context

If the contract fits in the window, one option is to send all of it. The other is to retrieve the relevant passages and send only those, the pattern covered in Retrieval-Augmented Generation and Tool-Using Agents, with the representations behind retrieval in Embeddings and Retrieval Representations. The two are not exclusive and neither dominates. They fail differently, and the differences point to which one to use.

Dimension	Put the corpus in the window	Retrieve a few passages
Where recall can fail	Inside the model: the fact is present but not used (position, dilution, distractors)	In the retriever: the right passage is never selected, so the model cannot see it
Questions needing the whole text	Possible when the corpus fits and the model aggregates reliably; summarizing, counting, and comparing across sections are the weak spots at long length	Poor fit: a top-\(k\) selection cannot count or compare what it did not retrieve
Cost per query	Grows with corpus size \(C\); mitigated by prefix caching if the corpus is fixed and requests are close in time	Roughly \(k\) times chunk size plus retrieval overhead; independent of corpus size
Latency	Prefill grows with prompt length	Retrieval step plus a short prefill
Size limit	The corpus must fit in \(N\) with room for output	Corpus size is limited by the index, not the window
Freshness and access control	Whatever is in the prompt is visible; permission filtering happens before assembly	Index must be refreshed; filtering can be applied at retrieval time
Provenance	Citations require the model to point into a large input	Passages come with identifiers and sources

Cost is easy to compute and worth computing before debating. The next program uses hypothetical prices in arbitrary units per million tokens, not any provider’s real rates, to compare stuffing a 400,000-token corpus with retrieving eight 500-token chunks. The cached-prefix multiplier (0.1) and the cache-write multiplier (1.25) are placeholders for the shape of a typical scheme.

# Hypothetical prices in arbitrary units per million input tokens; replace with your provider's real rates.
PRICE_IN = 1.0
PRICE_CACHED_IN = 0.1     # price multiplier for a cached prefix read (hypothetical)
PRICE_OUT = 4.0

def stuff_cost(corpus_tokens, q_tokens, out_tokens, cached=False, first=False):
    price = PRICE_IN if (first or not cached) else PRICE_CACHED_IN
    return (corpus_tokens * price + q_tokens * PRICE_IN + out_tokens * PRICE_OUT) / 1e6

def rag_cost(k_chunks, chunk_tokens, q_tokens, out_tokens, retrieval_fixed=0.0):
    return ((k_chunks * chunk_tokens + q_tokens) * PRICE_IN + out_tokens * PRICE_OUT) / 1e6 + retrieval_fixed

corpus, q, out = 400_000, 200, 300
rag = rag_cost(8, 500, q, out, retrieval_fixed=0.0005)
print(f"corpus {corpus:,} tokens; RAG per query: {rag:.4f}")
print(f"stuffing, no cache:           {stuff_cost(corpus, q, out):.4f}  ({stuff_cost(corpus,q,out)/rag:.0f}x RAG)")
print(f"stuffing, warm cached prefix: {stuff_cost(corpus, q, out, cached=True):.4f}  ({stuff_cost(corpus,q,out,cached=True)/rag:.0f}x RAG)")

# queries needed for a cache write (price 1.25x) to pay back versus no cache
write_mult = 1.25
n = 1
while True:
    total_cached = (corpus*write_mult + (n-1)*corpus*PRICE_CACHED_IN)/1e6
    total_plain = n*corpus*PRICE_IN/1e6
    if total_cached < total_plain: break
    n += 1
print(f"cache write at {write_mult}x pays back after {n} queries on the same prefix (prefix tokens only)")

corpus 400,000 tokens; RAG per query: 0.0059
stuffing, no cache:           0.4014  (68x RAG)
stuffing, warm cached prefix: 0.0414  (7x RAG)
cache write at 1.25x pays back after 2 queries on the same prefix (prefix tokens only)

Under these inputs stuffing costs about 68 times the retrieval pipeline per query without caching and about 7 times with a warm cache, and a cache write at 1.25 times the base price is paid back after two queries on the same prefix. The conclusions that survive changing the numbers are structural: stuffing scales with corpus size and retrieval does not, caching narrows but does not close the gap for large corpora, and caching helps only when many queries share one prefix within the cache lifetime. For a corpus that is queried a few times, or changes between queries, the cache advantage disappears. The computation covers input price only. It does not include the quality difference, which is the thing the evaluation below measures.

A decision sequence follows from the table.

1. If the corpus exceeds the window, or varies per user with permissions, retrieve.
2. If questions need an aggregate over the whole document (every mention, a count, a cross-section comparison), try long context first and measure it, because retrieval misses by construction.
3. If the corpus is small, fixed, and queried often, stuffing with a cached prefix is simple and removes the retriever as a failure source.
4. If neither is clearly right, retrieve generously (more or larger chunks than minimal) and use the window to hold them. This is the middle path: retrieval narrows the corpus, the window absorbs its imprecision. It reintroduces the position and ordering questions from the previous section.

Validating long-context behavior on your own system

Published results describe other models, other tasks, and other dates. What you need to know is how your model behaves on your inputs. The tests below separate limits, position, length, and task difficulty.

Start with the limits

Verify the accepted length with a request near the maximum and check that the service does not truncate silently. Compare your token count with the service’s reported usage on the same text. Test the output reserve by requesting a long generation at a prompt length that leaves little room. These are cheap, deterministic checks that catch integration bugs before any quality question.

The needle-in-a-haystack test, and why it is the floor

The common first probe, the needle-in-a-haystack test, inserts one fact (the needle) into a long filler text (the haystack) and asks for it back, varying the length of the text and the depth of the fact. The result is a grid of accuracy by length and depth. It is a good detector of gross failures such as a window that is not really usable at the advertised length or an obvious position hole. It is a weak test of anything else, because retrieving one verbatim string with no competing information is the easiest long-context task. Hsieh et al. (RULER) extend the idea with multiple needles, variable tracking through a chain of assignments, and aggregation tasks over the whole input. They report that nearly all models in their study, which score near-perfectly on the plain needle test, show large drops as length grows, and that only about half of the models claiming 32K or more kept satisfactory performance at 32K. The claimed length and the length at which a harder task still works are different numbers.

A harness for the sweep

The program below is a complete sweep harness with a stub in place of the model, because a hosted model cannot be called from this environment. The stub is deterministic and its recall curve is invented: it declines with length and dips in the middle, so that the harness has something to detect. The cases are synthetic (a filler sentence repeated, one inserted fact with a random code), which also means the model under test could not have seen them in training. With a real model, replace StubModel.answer with a call that sends the prompt and returns the model’s reply, and keep the rest.

import math, random

FILLER = "The committee reviewed routine items and adjourned without further business."

def make_case(n_filler, depth, secret, rng):
    """Synthetic haystack: filler sentences with one fact inserted at a relative depth."""
    sents = [FILLER] * n_filler
    pos = round(depth * n_filler)
    sents.insert(pos, f"The vault code for room {secret['room']} is {secret['code']}.")
    prompt = " ".join(sents) + f" What is the vault code for room {secret['room']}?"
    return prompt, secret["code"]

class StubModel:
    """Deterministic stand-in for a hosted model: recall probability falls with length and
    dips in the middle. The shape is invented to exercise the harness, not measured."""
    def __init__(self, seed):
        self.rng = random.Random(seed)
    def answer(self, prompt, truth, n_tokens, depth):
        base = 0.99 - 0.25 * (n_tokens / 32_000)
        middle_dip = 0.30 * math.sin(math.pi * depth) * (n_tokens / 32_000)
        p = max(0.0, base - middle_dip)
        return truth if self.rng.random() < p else "unknown"

def wilson(k, n, z=1.96):
    p = k / n
    den = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / den
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / den
    return centre - half, centre + half

def run(model, lengths, depths, trials, rng):
    results = {}
    for L in lengths:
        n_filler = L // 12            # about 12 tokens per filler sentence (rough)
        for d in depths:
            hits = 0
            for _ in range(trials):
                secret = {"room": rng.randint(100, 999), "code": rng.randint(1000, 9999)}
                prompt, truth = make_case(n_filler, d, secret, rng)
                hits += model.answer(prompt, truth, L, d) == truth
            results[(L, d)] = hits
    return results

rng = random.Random(7)
lengths = [2_000, 8_000, 32_000]
depths = [0.0, 0.25, 0.5, 0.75, 1.0]
trials = 40
res = run(StubModel(seed=1), lengths, depths, trials, rng)
print("accuracy (95% Wilson interval), rows = context length, columns = needle depth")
print("length  " + "".join(f"{d:>20.2f}" for d in depths))
for L in lengths:
    cells = []
    for d in depths:
        k = res[(L, d)]; lo, hi = wilson(k, trials)
        cells.append(f"{k/trials:5.2f} [{lo:4.2f},{hi:4.2f}]  ")
    print(f"{L:6,d}  " + "".join(f"{c:>20}" for c in cells))

accuracy (95% Wilson interval), rows = context length, columns = needle depth
length                  0.00                0.25                0.50                0.75                1.00
 2,000    1.00 [0.91,1.00]    0.93 [0.80,0.97]    0.97 [0.87,1.00]    0.97 [0.87,1.00]    1.00 [0.91,1.00]  
 8,000    0.90 [0.77,0.96]    0.82 [0.68,0.91]    0.80 [0.65,0.90]    0.82 [0.68,0.91]    0.97 [0.87,1.00]  
32,000    0.68 [0.52,0.80]    0.50 [0.35,0.65]    0.35 [0.22,0.50]    0.35 [0.22,0.50]    0.78 [0.62,0.88]

Each cell is 40 trials, with a 95% Wilson interval for the proportion. Read it in three steps. First, compare each row’s edges with its middle: at 32,000 tokens the middle cells (0.35) are well below the ends (0.68 and 0.78) and their intervals do not overlap with the right end, so the dip is distinguishable from noise. Second, compare across rows: at 8,000 tokens the cell at depth 0.5 (0.80) and the one at depth 0.0 (0.90) have overlapping intervals, so that difference could be sampling noise, and more trials per cell are needed before acting on it. Third, compare with the short-context control: the 2,000-token row is the baseline, and the useful quantity is the drop from it, not the absolute accuracy. The stub’s shape is invented, so none of these numbers say anything about real models. They show how to read the grid and how many trials are needed before a gap is believable.

What a good validation plan includes

• A short-context control for every task. The same question with only the relevant passage in the prompt measures the model’s ability on the task itself. The difference between that and the long-context score is the long-context penalty. Without it, a task the model cannot do well at any length looks like a long-context failure.
• Position and length as separate axes. Vary the length at fixed depth and the depth at fixed length, and keep a grid when budget allows.
• Hard distractors. Include look-alike facts (another room’s code, an older version of the clause) in addition to random filler, since similar scores are what attention has trouble separating.
• Tasks matched to the product. If the product answers questions that need two passages combined, test two-passage questions. If it summarizes, test coverage of items from different positions. A single-fact retrieval score does not predict these.
• Your real documents with questions whose answers you know. Synthetic cases isolate the variable. Real documents show whether the whole pipeline works, including parsing and chunking upstream. Use only data you are allowed to use.
• Decoding settings fixed. Use the same temperature and constraints across runs, as discussed in the decoding article, and repeat runs where outputs are sampled so that the interval reflects both the item and the sampling variation.
• Interval estimates and paired comparisons. Compare two orderings or two models on the same set of cases and look at the paired differences. Report the number of trials per cell.

From such a grid, you can define an effective length for a task: the longest input at which accuracy stays within an agreed margin of the short-context control, with a stated depth distribution. That number, not the advertised window, is the one to put in a design document. The broader practice of building reproducible test sets is covered in Evaluating AI Applications with Reproducible Test Sets.

When results change

Re-run the grid whenever the model version changes, the prompt template changes, or the retrieval step feeds the window differently. Providers revise models behind a stable name unless a version is pinned. A change in the shape of the position curve after an update is a regression even if average accuracy is unchanged.

References

• Vaswani, A., et al. Attention Is All You Need (2017). Introduces the Transformer and sinusoidal position encodings.
• Shaw, P., Uszkoreit, J., Vaswani, A. Self-Attention with Relative Position Representations (2018).
• Su, J., et al. RoFormer: Enhanced Transformer with Rotary Position Embedding (2021).
• Press, O., Smith, N. A., Lewis, M. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021).
• Kazemnejad, A., et al. The Impact of Positional Encoding on Length Generalization in Transformers (2023).
• Chen, S., Wong, S., Chen, L., Tian, Y. Extending Context Window of Large Language Models via Positional Interpolation (2023).
• Peng, B., Quesnelle, J., Fan, H., Shippole, E. YaRN: Efficient Context Window Extension of Large Language Models (2023).
• Beltagy, I., Peters, M. E., Cohan, A. Longformer: The Long-Document Transformer (2020).
• Xiao, G., et al. Efficient Streaming Language Models with Attention Sinks (2023).
• Dao, T., et al. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022).
• Liu, N. F., et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics (2023).
• Hsieh, C.-P., et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? (2024).