Needle-in-a-Haystack Test

The needle-in-a-haystack test checks whether a language model can use a fact placed somewhere inside a long input. A short statement, the needle, is inserted into a long stretch of unrelated text, the haystack. The model is then asked a question whose answer is the needle. The experiment is repeated across a grid of two variables: the total context length and the depth at which the needle sits, expressed as a percentage from the start (0%) to the end (100%). The public test popularized by Greg Kamradt sweeps exactly this length-by-depth grid and plots accuracy per cell.

What a run looks like

1. Choose a needle with a verifiable answer, for example “The access code for the blue cabinet is 4817.”
2. Build a haystack of filler text up to the target length and insert the needle at the target depth.
3. Ask “What is the access code for the blue cabinet?” and score the reply by exact match.
4. Repeat for every (length, depth) cell, with several needles or filler variants per cell so that each cell has a sample size.

The output is a heat map. A hole at some depth shows a position problem; a hole at a length shows the window is not usable there.

Effective length

From the grid you can define an effective length for this task: the longest input at which accuracy stays within an agreed margin of the short-context accuracy, across the depths you care about. Because each cell has few trials, report an interval. The Wilson interval for a proportion \(k/n\) is cheap to compute. For 20 trials, 20 correct gives an interval of about [0.84, 1.00], 18 correct gives [0.70, 0.97], and 15 correct gives [0.53, 0.89] (computed with the formula below). A cell that shows 90% is compatible with anything from about 70% to 97%, so single-cell differences of a few points mean little.

import math
def wilson(k,n,z=1.96):
    p=k/n; d=1+z*z/n; c=(p+z*z/(2*n))/d
    h=z*math.sqrt(p*(1-p)/n+z*z/(4*n*n))/d
    return c-h,c+h

What the test does not measure

The test is the easiest long-context task there is, because the needle is an exact string, it differs in topic from the filler, and no other passage competes with it. A search routine with no language model passes it. The following program builds haystacks of 100 to 50,000 filler sentences, inserts the needle at five depths, and answers with a regular expression.

import random,re
random.seed(0)
filler=["The committee reviewed the quarterly schedule again.","Rain was forecast for most of the week.",
        "The archive was reorganized by date.","A new printer was installed on the second floor."]
needle="The access code for the blue cabinet is 4817."
def build(n_sent,depth):
    s=[random.choice(filler) for _ in range(n_sent)]
    s.insert(int(depth*n_sent),needle); return " ".join(s)
def grep_model(prompt):
    m=re.search(r"blue cabinet is (\d+)",prompt); return m.group(1) if m else None
for n in (100,1000,10000,50000):
    ok=[grep_model(build(n,dp))=="4817" for dp in (0,.25,.5,.75,1)]
    print(n,"sentences:",sum(ok),"/5 correct")

The result is 5/5 at every size (100, 1,000, 10,000 and 50,000 sentences). A perfect score therefore shows that the model can locate a distinctive string; it does not show that it can reason over, combine, or compare information spread across the input. Tasks that real applications need, such as comparing two clauses, following a chain of references, counting, or answering when several similar passages compete, are not exercised.

Hsieh et al. built RULER to go further. It keeps the retrieval idea but varies the number and type of needles and adds multi-hop tracing and aggregation tasks. They report that models scoring near-perfectly on the plain test show clear drops as length grows, and that only about half of the models claiming 32K tokens or more kept satisfactory performance at 32K on the harder tasks.

Design cautions

• A needle that is odd relative to the filler is easier to find than a fact written in the same style as its surroundings. Varying the needle’s style changes the difficulty.
• Fixed depths such as 0%, 25%, 50% undersample position effects; use several needles per depth.
• Token counts depend on the tokenizer. State the length in the model’s own tokens.
• The test says nothing about answer quality when the prompt also contains instructions, examples, or retrieved passages in a real application format. A test built from that real format measures what the application needs.

References

• Kamradt, G., LLMTest_NeedleInAHaystack — https://github.com/gkamradt/LLMTest_NeedleInAHaystack
• Hsieh, C.-P., et al., RULER: What’s the Real Context Size of Your Long-Context Language Models? — https://arxiv.org/abs/2404.06654