Language Model Objectives and Limits

Consider a hypothetical assistant built on a language model. Asked to write a function that merges overlapping time intervals, it returns correct code with a sensible explanation. Asked the same morning for a citation supporting a claim in a report, it returns an author list, a journal, a year, and a page range, all formatted perfectly. The paper does not exist. Nothing about the second answer looks different from the first: the same fluency, the same confident tone. Two questions follow. What is the model optimized to produce, so that both answers come out of the same mechanism? And what does that imply for how much of its output an engineer can rely on without checking?

Both answers start from the training objective. A language model is a family of conditional distributions, and minimizing cross-entropy rewards some things and not others; the consequences are how context shapes output, why individually plausible samples make long tasks unreliable, and why reasoning and factual recall are where the gap between plausible and correct is widest. Self-attention, masking, and the decoder block are assumed from Transformers from Scratch: Self-Attention to Encoder–Decoder, which is the prerequisite for this article.

A language model is a family of conditional distributions

Take a sequence of tokens \(x_1, x_2, \dots, x_T\), where each token comes from a fixed vocabulary of size \(V\) (how text becomes tokens is covered in Tokenization Explained: BPE, WordPiece, and SentencePiece). The chain rule of probability writes the joint probability of the whole sequence as a product of one-step conditionals:

\[P(x_1,\dots,x_T)=\prod_{t=1}^{T}P(x_t\mid x_{<t})\]

Here \(x_{<t}\) is the prefix \(x_1,\dots,x_{t-1}\). This is an identity, not a modeling assumption: any joint distribution over sequences can be written this way, and the factorization says nothing yet about how well a particular model captures it. A language model is a function that, given any prefix, returns a probability distribution over the \(V\) possible next tokens. A decoder-only Transformer does this by turning the prefix into a vector of \(V\) scores, called logits, and passing them through the softmax function (Softmax and Multiclass Classification Explained derives it). Because the attention mask is causal, a single forward pass over a length-\(T\) sequence produces all \(T\) conditionals at once, one per position. That is what makes training efficient and is the only place where the architecture enters the objective.

Three consequences follow from this definition before any training happens.

• The model never emits a token directly. It emits a distribution, and a separate rule turns the distribution into a token: take the largest probability, draw a sample, or restrict the choice. Everything the model “says” is therefore one draw from a family of possible outputs.
• Each distribution is conditioned on the entire prefix and on nothing else. The prompt, any retrieved documents, earlier turns of a conversation, and the tokens the model itself has already generated are all just prefix.
• Probability is assigned to token strings, not to propositions. The model has no separate representation of “true” in the objective; a true sentence and a false sentence are both sequences whose probability can be high or low.

What the training objective rewards

Training adjusts the parameters \(\theta\) so that the model’s conditional distribution \(q_\theta(x_t\mid x_{<t})\) assigns high probability to the tokens that actually follow in the training text. The loss is the average negative log-probability of the observed next tokens, which is the cross-entropy between the corpus and the model. LLM Pretraining: Objectives, Data, and Scaling Laws covers how targets are built from a corpus, the data pipeline, and scaling; this section only needs the statistical meaning of the loss.

Let \(p(x_t\mid x_{<t})\) denote the true conditional distribution of the text source at a given prefix. For one prefix, the expected loss decomposes as

\[H(p,q)=-\sum_{v}p(v)\log q(v)=H(p)+D_{\mathrm{KL}}(p\,\|\,q)\]

The first term, \(H(p)\), is the entropy of the source: the uncertainty that remains even for a perfect model, because the text really is unpredictable at that position. The second term, the Kullback–Leibler divergence, is zero only when \(q\) equals \(p\) and positive otherwise. Only the second term depends on the model, so minimizing cross-entropy is the same as pulling the model’s distribution toward the source’s distribution at every prefix, weighted by how often that prefix occurs. Three properties of this objective matter for everything later.

• The target is the distribution of the corpus, not the truth. A model that perfectly matched \(p\) would reproduce the corpus’s mixture of correct statements, mistakes, fiction, opinions, and jokes, in the proportions the corpus contains them.
• There is a loss floor. Because \(H(p)\) is positive wherever the text is genuinely open-ended, loss cannot reach zero, and a lower loss is not evidence that the remaining uncertainty can be removed.
• Every position counts through its probability on the observed token. The loss does not distinguish a token that carries a fact (a date, a name) from a token that carries syntax. An error on a rare, factual token costs the same as a comparable error on a function word, and rare tokens are, by definition, seen less often.

A worked example with a known answer

In real corpora \(p\) is unknown, so the decomposition cannot be checked directly. A synthetic source makes it checkable. The program below defines a second-order Markov chain over six tokens, so the true next-token distribution depends on the previous two tokens and its entropy can be computed exactly. It then fits count-based models that look at 0, 1, or 2 previous tokens (add-0.5 smoothing) on training sequences of different lengths and measures cross-entropy on 20,000 fresh test tokens. The program uses Python 3.12 and the standard library only. These count models are not neural networks; they are the smallest example in which “a model is a function from prefix to distribution” is literally true.

import math, random
from collections import Counter, defaultdict

V = 6                      # vocabulary size
rng = random.Random(7)

def dirichlet(alpha, k):
    g = [rng.gammavariate(alpha, 1.0) for _ in range(k)]
    s = sum(g)
    return [x / s for x in g]

# True generator: second-order Markov chain with sparse-ish transitions.
true_p = {(a, b): dirichlet(0.3, V) for a in range(V) for b in range(V)}

def sample(n, seed):
    r = random.Random(seed)
    seq = [r.randrange(V), r.randrange(V)]
    for _ in range(n - 2):
        seq.append(r.choices(range(V), weights=true_p[(seq[-2], seq[-1])])[0])
    return seq

def fit(seq, order, k=0.5):
    counts = defaultdict(Counter)
    for i in range(order, len(seq)):
        counts[tuple(seq[i - order:i])][seq[i]] += 1
    def prob(ctx, tok):
        c = counts.get(tuple(ctx), Counter())
        return (c[tok] + k) / (sum(c.values()) + k * V)
    return prob

def xent(prob, seq, order):
    nll = 0.0; n = 0
    for i in range(2, len(seq)):         # same targets for every order
        nll -= math.log(prob(seq[i - order:i], seq[i])); n += 1
    return nll / n

test = sample(20000, seed=99)
h_true = -sum(math.log(true_p[(test[i-2], test[i-1])][test[i]])
              for i in range(2, len(test))) / (len(test) - 2)
print(f"true conditional entropy (nats/token, estimated on test): {h_true:.3f}")
print(f"uniform baseline log V: {math.log(V):.3f}")
print(f"{'train':>7} {'order':>5} {'xent':>7} {'xent-H':>7}")
for n_train in (300, 3000, 100000):
    train = sample(n_train, seed=n_train)
    for order in (0, 1, 2):
        ce = xent(fit(train, order), test, order)
        print(f"{n_train:>7} {order:>5} {ce:7.3f} {ce - h_true:7.3f}")

Program output:

true conditional entropy (nats/token, estimated on test): 0.849
uniform baseline log V: 1.792
  train order    xent  xent-H
    300     0   1.734   0.885
    300     1   1.519   0.670
    300     2   0.997   0.148
   3000     0   1.726   0.877
   3000     1   1.494   0.644
   3000     2   0.873   0.024
 100000     0   1.726   0.876
 100000     1   1.489   0.639
 100000     2   0.851   0.001

Read the table from the top. The source’s conditional entropy is about 0.849 nats per token, against a uniform guess of \(\ln 6 = 1.792\), so the source is fairly predictable once its context is known. The last column is the excess over that floor, an estimate of the KL term. The order-2 model, which sees exactly the context the source uses, closes the gap as training data grows: 0.148 nats at 300 tokens, 0.024 at 3,000, and 0.001 at 100,000. The order-0 and order-1 models do not. Their gap stays near 0.88 and 0.64 nats however much data they see, because their conditioning information is insufficient: no amount of data lets a model that ignores the second-previous token represent a distribution that depends on it. The order-2 model’s gap at 300 tokens is the opposite problem, too little data for 36 contexts with six outcomes each. Both failure types, missing context and missing data, reappear in the later sections.

A Transformer replaces the count table with a learned function that shares statistical strength across similar prefixes, which lets it condition on thousands of tokens without needing each exact prefix to have occurred. The objective, the floor, and the two sources of excess loss remain the same.

Distribution and context

The distribution \(q_\theta(\cdot\mid x_{<t})\) changes whenever the prefix changes. This is the whole mechanism by which a prompt controls a model, and it explains both why prompting works and why it is fragile.

The prefix is the only input

A prompt shifts probability mass because, in the training text, continuations differ depending on what precedes them. After the prefix “The capital of France is”, text in the corpus overwhelmingly continues with one token. After “Write a poem about”, the continuation is spread over a huge set. The first distribution has low entropy and the second high; the model reproduces that difference because the objective trained it to. Instructions, examples, role descriptions, and retrieved passages work by making some continuations more probable than they would be without them. A model that completes a few input–output examples in the pattern of those examples is conditioning on the examples, which Brown et al. describe as in-context learning in the GPT-3 paper. No parameter changes during this process; the prefix changes, and so does the distribution.

Two practical consequences follow. First, anything that changes the prefix changes the output distribution, including wording, ordering, formatting, and irrelevant text. A reordering of two paragraphs in a prompt is a different input as far as the model is concerned, even when a human reader would call it equivalent. Second, the model has no way to separate instructions from data inside a prefix, because both are tokens, which is why untrusted text placed in a prompt can steer the output (see the glossary entry on prompt injection).

From distributions to text: sampling and greedy choice

Generation repeats one step: compute the distribution for the current prefix, choose a token by some rule, append it, and repeat. With a causal Transformer, the model’s own earlier choices are part of the prefix for every later step. The choice rule is a design decision separate from the model, and the table lists the usual options. Building a Transformer Language Model in PyTorch implements sampling with temperature and top-k; the details of each rule are a separate topic.

Rule	Choice at each step	Typical use and limitation
Greedy	Highest-probability token	Deterministic given a fixed model and numerics; can produce repetitive text and is not the most probable whole sequence
Temperature sampling	Draw from the distribution after dividing logits by a temperature	Higher temperature flattens the distribution and raises diversity and error rate; temperature near zero approaches greedy
Truncated sampling (top-k, top-p)	Draw from the renormalized head of the distribution	Cuts off the unlikely tail that causes rare derailments; leaves the head’s randomness intact

Greedy choice is not the same as finding the most probable sequence, because probabilities multiply along the sequence. Suppose the first token is “A” with probability 0.4 or “B” with probability 0.6. After “A” the model is nearly sure of one continuation (probability 0.9); after “B” its probability is spread evenly over ten continuations (0.1 each). Greedy picks “B” first, then any one of the ten, and ends with a sequence of probability \(0.6\times0.1=0.06\). The path through “A” has probability \(0.4\times0.9=0.36\), six times higher. Greedy decoding maximizes each step and can still miss the sequence the model considers most likely. Conversely, the most probable sequence is also not necessarily a good output, which is part of the reason sampling is used for open-ended text.

What is not in the prefix does not exist

Context also has a hard limit: the model sees a fixed number of tokens, and attention cost grows with sequence length, so a long conversation or document eventually does not fit. Position representations and the cost of long contexts belong to the design of the Transformer block and are covered in the prerequisite article. The modeling point is simpler: whatever is not in the prefix does not exist for the model. If a question depends on a fact the model did not learn in training and that fact is not in the prompt, the distribution is conditioned on nothing relevant and the model will produce what is plausible.

Capabilities versus reliability

The same objective that gives a model broad skill also determines the shape of its failures. To see why, separate two properties that are often blurred in conversation: what a model can do, and what it does on a given attempt.

Two definitions

Call a task within the model’s capability if some prompt and some sample solves it. Call the model reliable on the task if a single attempt, on the prompts a user will actually write, solves it with high probability. For a fixed prompt, let \(p\) be the per-attempt success probability. Capability only needs \(p>0\). Reliability needs \(p\) close to 1 and the same on rephrased and perturbed inputs. A model that solves a problem on one attempt in three can be shown solving it, and still be unusable as a component that must work every time.

Chen et al. define the pass@k metric for code generation: sample \(k\) solutions and count the problem as solved if any passes the unit tests. Because estimating it from exactly \(k\) samples has high variance, they draw \(n\ge k\) samples per problem, count the \(c\) that pass, and use the unbiased estimator (averaged over problems)

\[\widehat{\text{pass@}k}=1-\binom{n-c}{k}\Big/\binom{n}{k}\]

It is the probability that a random subset of \(k\) of the \(n\) samples contains at least one pass. Its mirror image, the probability that all \(k\) attempts succeed, is \(p^k\) for a single problem. Pass@k measures capability under a generous protocol (try again until one works, with a verifier available); \(p^k\) measures reliability under a strict one (it must work every time). The two diverge quickly, as the next program shows on synthetic data: 200 problems whose per-attempt success probabilities are drawn from a Beta(0.4, 0.8) distribution, with 20 attempts simulated per problem.

import math, random
print("per-step success compounding, (1-eps)^n")
print(f"{'eps':>6} " + " ".join(f"n={n:<5}" for n in (1, 5, 10, 20, 50, 100)))
for eps in (0.001, 0.01, 0.05):
    print(f"{eps:>6} " + " ".join(f"{(1-eps)**n:<7.3f}" for n in (1,5,10,20,50,100)))

def pass_at_k(n, c, k):
    if n - c < k: return 1.0
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)

rng = random.Random(3)
# 200 synthetic problems; per-attempt success probability p_i ~ Beta(0.4, 0.8)
P = [rng.betavariate(0.4, 0.8) for _ in range(200)]
print("\nmean per-attempt success rate:", round(sum(P)/len(P), 3))
n = 20
counts = [sum(rng.random() < p for _ in range(n)) for p in P]
print(f"{'k':>3} {'pass@k est':>11} {'pass@k true':>12} {'all-k true':>11}")
for k in (1, 2, 5, 10, 20):
    est = sum(pass_at_k(n, c, k) for c in counts) / len(P)
    true = sum(1 - (1-p)**k for p in P) / len(P)
    allk = sum(p**k for p in P) / len(P)
    print(f"{k:>3} {est:11.3f} {true:12.3f} {allk:11.3f}")
print("\nproblems solved at least once in 20 tries:", sum(c > 0 for c in counts), "of", len(P))
print("problems solved 20 of 20 tries:", sum(c == n for c in counts), "of", len(P))

Program output:

per-step success compounding, (1-eps)^n
   eps n=1     n=5     n=10    n=20    n=50    n=100  
 0.001 0.999   0.995   0.990   0.980   0.951   0.905  
  0.01 0.990   0.951   0.904   0.818   0.605   0.366  
  0.05 0.950   0.774   0.599   0.358   0.077   0.006  

mean per-attempt success rate: 0.356
  k  pass@k est  pass@k true  all-k true
  1       0.360        0.356       0.356
  2       0.481        0.478       0.234
  5       0.627        0.621       0.123
 10       0.721        0.711       0.068
 20       0.795        0.783       0.034

problems solved at least once in 20 tries: 159 of 200
problems solved 20 of 20 tries: 6 of 200

The first table is the compounding calculation discussed below. In the second, the mean per-attempt success rate is 0.356, and pass@1 matches it. Pass@20 reaches 0.795, and 159 of 200 problems were solved at least once in 20 tries, so by the pass@k view this synthetic model “can do” four out of five problems. The all-k column tells the other story: the chance that 5 consecutive attempts all succeed is 0.123 averaged over the problems, and only 6 of 200 problems succeeded on all 20 tries. The estimator and the exact value agree to within about 0.01, which is the sampling error from using 20 attempts per problem. Any evaluation of a model that reports only the best of several tries is measuring capability, and the right number for a system that acts on each output is the one a single try gives.

Why small per-step error becomes large task error

The first table in the program output shows the second mechanism. Many real tasks are sequences of dependent steps: a long answer, a multi-step calculation, an agent that calls tools in a chain. If each step independently succeeds with probability \(1-\varepsilon\), the whole sequence succeeds with probability \((1-\varepsilon)^n\). At \(\varepsilon=0.01\), a 20-step task succeeds 81.8% of the time and a 100-step task 36.6% of the time. At \(\varepsilon=0.05\), a 20-step task succeeds only 35.8% of the time. The numbers in that table are arithmetic on assumed error rates, not measurements of any model.

Real errors are not independent. They correlate, because a wrong value early in a derivation becomes part of the prefix and conditions every later token. A model that has written an incorrect intermediate result tends to continue consistently with it, which is exactly what the objective rewards: coherence with the prefix. Errors can also be corrected when a later step recovers, so the product is neither a bound nor a prediction. It shows the shape of the problem: reliability requirements multiply across steps, so a system with many steps needs a much higher per-step success rate than a single question does, or it needs checks between steps.

Training and generation also differ in a way that feeds this effect. During training the model always sees the true prefix from the corpus (teacher forcing). During generation it sees its own previous output. A prefix containing the model’s own slip may look unlike anything in the training text, so the conditionals there are less well trained. This mismatch is known as exposure bias in the sequence-modeling literature. Whatever the cause, a slip early in a long output conditions every later token.

Confidence is available, but only partly trustworthy

A softmax distribution seems to offer an obvious reliability signal: the probability of the chosen token. If the model assigns 0.95 to its answer, it should be right about 95% of the time. That property is called calibration, and it holds in some settings and not others. Kadavath et al. report that larger pretrained models are well calibrated on multiple-choice and true/false questions in suitable formats, and that models can be trained to estimate whether they know an answer. Calibration measured on one question format does not transfer automatically to free-form generation, where the answer is a sequence of tokens and probability is split across many phrasings of the same claim, or to a model whose output distribution was reshaped by later training stages. Treat token probabilities as a signal to be measured against labeled outcomes on the task, not as a guarantee. Probability Calibration and Uncertainty Estimation and Model Calibration in Deep Learning give the measurement methods (reliability curves, expected calibration error) and the recalibration techniques.

Sources of unreliability to expect

• Sampling variance. Two runs on the same prompt can differ, and a single run can pass or fail by luck. An evaluation needs enough samples that the confidence interval is narrow relative to the effect being measured.
• Input sensitivity. Because the prefix is the only input, equivalent prompts that differ in wording or layout give different distributions. A test set built from one phrasing per case tests one point in a region of inputs.
• Task difficulty is not uniform. The same model can be reliable on common patterns and unreliable on rare variations of the same task, a distribution-shift effect that benchmarks that average over cases hide.
• Post-training changes the distribution. Instruction tuning and preference training move the model away from the raw corpus distribution toward preferred behaviors; this changes style, refusal, and calibration without changing the architecture. Fine-Tuning and Alignment: SFT, LoRA, RLHF, and DPO explains these methods.

Reasoning limits

A model trained only to predict text can solve many problems that look like reasoning, and it can also fail on small variations of problems it solves at other times. The first point is what the objective explains: if reasoning appears in the text, then predicting the text well requires modeling its steps. The second point comes from how a Transformer spends computation.

Fixed computation per token, and tokens as working memory

A Transformer with \(L\) layers performs a fixed amount of sequential computation to produce each token: information flows through \(L\) attention and feed-forward blocks and then through the output softmax. A question whose answer requires many dependent steps, such as composing a long chain of operations where each step needs the previous result, cannot be solved by extending the computation within one forward pass past what the depth allows, unless the network finds a shortcut that computes the answer in fewer sequential steps. The model can extend its computation in a different way: by writing intermediate results as tokens and conditioning on them. Each generated token becomes part of the prefix, so the text itself serves as working memory.

Wei et al. showed that prompting a large model with examples that include intermediate reasoning steps improves its performance on arithmetic, commonsense, and symbolic tasks. The mechanism is visible in a small case. Asked for \(47\times86\) in a single token, the model must produce all of “4042” from the prompt with one pass for the first digit and one for each next digit, each of which must internally contain parts of the whole calculation. Written step by step, the calculation becomes a sequence of easy conditionals:

47 x 86
47 x 80 = 3760
47 x 6  = 282
3760 + 282 = 4042

Each line is a short lookup-and-combine step whose result is then in the prefix. The reasoning tokens are generated by the same distribution as any other text, so the earlier compounding argument applies to them: a slip in “47 x 6 = 272” is conditioned on by every later line. Chain-of-thought text improves accuracy when it makes each step easier than the whole, and it can lower accuracy when the added steps introduce errors of their own.

What the failures look like

Several published results show specific patterns, each with a connection to the objective.

• Compositional depth. Dziri et al. tested multi-digit multiplication, logic grid puzzles, and dynamic programming tasks and report that accuracy falls as the number of dependent steps grows, even for models that perform well on small instances. They interpret the pattern as matching of subcomputations seen in training rather than execution of a general procedure.
• Explanations that do not reflect the cause of the answer. Turpin et al. add biasing features to prompts, such as reordering multiple-choice options, and find that models’ chain-of-thought explanations rationalize the biased answer without mentioning the bias. Generated reasoning is text that continues the prefix plausibly; it is not a readout of the computation that selected the answer.
• Self-correction without new information. Huang et al. find that asking a model to review and revise its own reasoning, with no external feedback, often fails to improve and sometimes lowers accuracy on reasoning tasks. A critique is another continuation from the same distribution, so it is no more grounded than the first answer unless it brings in a check the first lacked, such as running the code or consulting a source.
• Direction of the learned association. Berglund et al. fine-tuned models on statements of the form “A is B” and found they do not reliably answer the reversed question “B is ?”, although the reversed relation is a logical consequence. They name this the reversal curse and note that when the “A is B” statement is in the prompt, the model can answer the reversed question.

A toy for the reversal asymmetry

The reversal result follows from the objective in a direct way. The loss rewards probability on the next token given the prefix, so a statement “X wrote Y” teaches the conditional of Y given “X wrote” and says nothing about the conditional of X given “Y was written by”, which is a different prefix that does not appear in the training text. A learned model generalizes across prefixes, so the asymmetry in a Transformer is a finding about learned behavior and not a theorem. The count-based toy below shows the logic in its cleanest form: a model that conditions on the full prefix, trained on four synthetic “author wrote title” sentences with invented names.

import math
from collections import Counter, defaultdict

facts = [("Mira Lund", "Salt Orchard"), ("Tomas Vey", "Glass Meridian"),
         ("Ines Okafor", "Paper Lantern"), ("Jun Halvard", "Quiet Engine")]
fwd = [f"{a} wrote {b} ." for a, b in facts]          # only one direction
both = fwd + [f"{b} was written by {a} ." for a, b in facts]

def train(sentences):
    counts = defaultdict(Counter)
    for s in sentences:
        toks = s.split()
        for i in range(1, len(toks)):
            counts[tuple(toks[:i])][toks[i]] += 1       # condition on full prefix
    return counts

def next_dist(counts, prefix, vocab):
    c = counts.get(tuple(prefix.split()))
    if not c:                                           # unseen prefix: no evidence
        return {w: 1/len(vocab) for w in vocab}, False
    t = sum(c.values()); return {w: n/t for w, n in c.items()}, True

for name, corpus in (("forward-only", fwd), ("both directions", both)):
    cnt = train(corpus)
    vocab = sorted({w for s in corpus for w in s.split()})
    print(f"[{name}] vocab size {len(vocab)}")
    d, seen = next_dist(cnt, "Mira Lund wrote Salt", vocab)
    print("  P(Orchard | 'Mira Lund wrote Salt') =", round(d.get("Orchard", 0), 3), "context seen:", seen)
    d, seen = next_dist(cnt, "Salt Orchard was written by", vocab)
    print("  P(Mira | 'Salt Orchard was written by') =", round(d.get("Mira", 0), 3), "context seen:", seen)

Program output:

[forward-only] vocab size 18
  P(Orchard | 'Mira Lund wrote Salt') = 1.0 context seen: True
  P(Mira | 'Salt Orchard was written by') = 0.056 context seen: False
[both directions] vocab size 21
  P(Orchard | 'Mira Lund wrote Salt') = 1.0 context seen: True
  P(Mira | 'Salt Orchard was written by') = 1.0 context seen: True

With forward-only training, the model assigns probability 1.0 to “Orchard” after “Mira Lund wrote Salt” and has no evidence at all for the reversed prefix, so it falls back to a uniform guess (1/18 = 0.056). Adding the reversed sentences to the training text fixes the lookup completely. A neural model interpolates between prefixes instead of falling back to uniform, so the toy overstates how absolute the failure is; Berglund et al. report that in real models the effect persists across sizes and families. The engineering consequence is the same: a fact the model was trained on in one phrasing is not guaranteed to be retrievable under another, so retrieval-based or tool-based lookups need to be tested for each direction of use that the application requires.

Factuality limits

Return to the citation in the opening scenario. The objective does not penalize a fluent, specific, wrong answer more than a fluent, specific, right one unless the training data makes the right one more probable. This section follows that observation to where it is measurable.

What the model does when it does not know

Suppose the prompt asks for the birthday of a person who appears once in the training corpus, or in no document. The conditional distribution over the next token, given the prefix “…was born on”, is shaped by what birth-date text looks like in general: a month name, then a number, then a year. If the model has no stored association for this person, the distribution is spread over plausible dates. Sampling from it, or even choosing its highest-probability item, yields a date-shaped answer. The model has no separate mechanism that says “no association found, abstain”; abstention is a continuation like any other, and it is rare in training text that is followed by a fact.

This is one account of what people call hallucination: fluent output that is not supported by any source the model was given or by reliable stored knowledge. It is not the only cause. A corpus also contains errors, outdated statements, and conflicting claims, and a model that matches the corpus will reproduce them in proportion. Stale knowledge is a related case: facts after the training cutoff are absent, and facts that changed are represented by both old and new versions.

Rare facts are the hard case

Mallen et al. report that models answer questions about popular entities well and struggle with long-tail entities, and that increasing model size does not appreciably improve memorization of the long tail. The statistics behind this can be stated precisely. Suppose a corpus mentions facts with a heavy-tailed frequency distribution, so many facts appear exactly once and many never appear. A model cannot know a fact it never saw, and a fact seen once is hard to distinguish from a mistake in the corpus. The Good–Turing estimator from statistics says that the probability that the next draw is something never seen before is approximately the fraction of draws so far that were singletons. Kalai and Vempala use this to argue that a language model that is calibrated in a particular technical sense must hallucinate on arbitrary facts at a rate close to the fraction of facts that occur exactly once in training. They note that the argument does not apply to facts that are systematic, such as arithmetic, or that occur many times.

The program below checks the statistical half of this on synthetic data. It defines 20,000 facts with Zipf-distributed mention probabilities (probability proportional to 1/rank), draws N training mentions, and measures three quantities, averaged over 5 random seeds. The singleton rate is the number of facts mentioned exactly once divided by N. The unseen mass is the total probability of facts that were never mentioned, which is the chance that a fresh query concerns an unseen fact. The last column is the error rate of an idealized model that recalls every seen fact correctly and guesses uniformly among 100 candidate answers for unseen ones. This idealized model is a thought experiment, not a language model.

import random
from collections import Counter

M = 20000                                  # distinct facts (e.g. person -> birthday)
weights = [1 / (r + 1) for r in range(M)]  # Zipf(1) mention frequencies
total = sum(weights)
probs = [w / total for w in weights]
print(f"{'mentions N':>10} {'singleton rate':>15} {'unseen mass':>12} {'fresh-query error':>18}")
for N in (2000, 20000, 200000):
    sing, miss, err = [], [], []
    for seed in range(5):
        r = random.Random(seed)
        seen = Counter(r.choices(range(M), weights=probs, k=N))
        sing.append(sum(1 for c in seen.values() if c == 1) / N)
        miss.append(sum(p for i, p in enumerate(probs) if i not in seen))
        # model recalls every seen fact, guesses uniformly among 100 candidates otherwise
        err.append(sum(p for i, p in enumerate(probs) if i not in seen) * (1 - 1/100))
    avg = lambda x: sum(x) / len(x)
    print(f"{N:>10} {avg(sing):>15.3f} {avg(miss):>12.3f} {avg(err):>18.3f}")

Program output:

mentions N  singleton rate  unseen mass  fresh-query error
      2000           0.393        0.390              0.386
     20000           0.180        0.177              0.176
    200000           0.023        0.023              0.022

The singleton rate and the unseen mass track each other closely at every corpus size: 0.393 against 0.390 at 2,000 mentions, 0.180 against 0.177 at 20,000, and 0.023 against 0.023 at 200,000. That agreement is the Good–Turing estimate. It also shows the way out and its cost. The fresh-query error falls from 0.386 to 0.022 as the corpus grows by a factor of 100, because more mentions cover more of the tail, and it falls slowly. Within this model, no training improvement removes the error for facts the data never contains. What reduces it is changing where the answer comes from.

Design responses to factual limits

The analysis points to responses that act on the source of the answer instead of the model’s fluency.

Limit	Mechanism	Response in a system
Rare or unseen fact	No stored association; distribution is spread over plausible continuations	Put the evidence in the prefix with retrieval (see RAG), and measure whether answers follow the evidence
Fact stored in one phrasing only	Conditionals are learned per prefix; reversed or paraphrased queries may miss	Test each query form the application will use; use structured lookups for key-value facts
Multi-step computation	Fixed depth per token; errors condition later steps	Offload arithmetic and logic to code or tools; check intermediate results
Confident wrong output	Plausibility is not correctness; self-review reuses the same distribution	Verify against an external source or executable check; calibrate confidence on labeled data and abstain below a threshold
Stale knowledge	Training cutoff; old and new versions coexist	Supply current data in context; record the data version used
Prompt variation	Different prefix gives a different distribution	Evaluate on several phrasings; track variance across samples

Each response has costs and failure modes of its own. Retrieval adds a component that can return the wrong passage or an outdated one, and the model can still ignore or misread what it is given. Tool use moves correctness to the tool but adds interface errors. Verification works well where an executable or lookup check exists, such as unit tests for code or schema validation for structured output, and poorly where “correct” is a judgment. Abstention thresholds trade coverage for accuracy, and their value depends on how well the confidence signal is calibrated for the task. The questions of how to build and evaluate these pieces are taken up in the articles on AI application engineering, and they all start from the constraints established here.

What the objective cannot be asked to deliver

Next-token prediction is a general-purpose training signal. It pulls in syntax, world knowledge, code, and reasoning patterns because all of them help predict text, which is why one model can write a merge function and draft a report. The same signal contains no term for verification, abstention, or consistency across phrasings. Each limit in this article follows from one of those absent terms: sampled output that is plausible but not checked, long chains whose steps are generated rather than verified, facts whose evidence is too sparse to determine the continuation, and answers whose confidence is not tied to correctness.

The boundaries are not fixed. Larger models, better data, and post-training change where on the capability–reliability curve a task sits, and tools and retrieval change what the model has to know. What does not change with scale alone is the structure of the problem: the model produces a distribution over continuations conditioned on a prefix, and a correct answer is one of the plausible continuations. A system built on that component gets reliability from what surrounds the model: the evidence placed in the prefix, the checks applied to the output, the measurement of per-attempt success on the inputs that matter, and the choice of tasks where an occasional wrong answer is affordable.

References

• Vaswani et al., Attention Is All You Need (arXiv:1706.03762)
• Brown et al., Language Models are Few-Shot Learners (arXiv:2005.14165)
• Chen et al., Evaluating Large Language Models Trained on Code (arXiv:2107.03374)
• Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv:2201.11903)
• Kadavath et al., Language Models (Mostly) Know What They Know (arXiv:2207.05221)
• Mallen et al., When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (arXiv:2212.10511)
• Turpin et al., Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (arXiv:2305.04388)
• Dziri et al., Faith and Fate: Limits of Transformers on Compositionality (arXiv:2305.18654)
• Berglund et al., The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A” (arXiv:2309.12288)
• Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet (arXiv:2310.01798)
• Kalai and Vempala, Calibrated Language Models Must Hallucinate (arXiv:2311.14648)