LLM Pretraining: Objectives, Data, and Scaling Laws

This article focuses on autoregressive, decoder-only language-model pretraining: learning to predict tokens from preceding text. The objective connects the model to a corpus, while data selection, architecture, optimization, and available compute shape the result. Scaling laws help plan that allocation by fitting empirical relationships between loss and training resources. They do not determine the quality of a corpus or guarantee a model’s performance on every downstream task.

How next-token prediction supplies targets

\[\mathcal{L}=-\sum_{t}\log P_\theta(x_t\mid x_{<t})\]

Here \(x_t\) is the token at position \(t\), \(x_{<t}\) is its preceding context, and \(\theta\) denotes the model parameters. Minimizing negative log-probability rewards assigning more probability to the observed continuation. Training examples can contain syntax, facts, code, and arguments, so the objective can support learning useful patterns in each. Low average loss does not establish reliable factual knowledge, arithmetic, or a particular internal world model.

Causal language modeling supplies targets at each included next-token position; padding and other excluded positions do not contribute. A causal attention mask prevents a position from seeing future tokens. BERT’s original masked-language-model objective selected 15% of positions as targets, and only 80% of those were replaced with a mask token; the rest were randomly replaced or left unchanged. Its bidirectional context and training task differ, so counting prediction targets alone does not establish which objective is more compute-efficient.

BOS and EOS mark the beginning and end of a sequence. For the sequence [BOS, a, b, EOS], feed [BOS, a, b] and predict [a, b, EOS]. This one-position shift matters: predicting the token already supplied at the same position would permit a trivial copying task. Training uses the observed preceding tokens, sometimes called teacher forcing, while generation feeds back sampled or selected tokens. The example below supplies logits directly to isolate the loss calculation; a real model produces them with causal attention. The examples require PyTorch and should be run in order.

import math
import torch
import torch.nn.functional as F

def next_token_loss(logits, targets):
    """Already aligned logits (B, T, V), targets (B, T); -100 excludes padding."""
    if logits.ndim != 3 or logits.shape[:2] != targets.shape:
        raise ValueError("Expected aligned (B, T, V) logits and (B, T) targets")
    if not (targets != -100).any():
        raise ValueError("At least one target is required")
    return F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1),
                           ignore_index=-100)

ids = torch.tensor([[0, 1, 2, 3]])
inputs, targets = ids[:, :-1], ids[:, 1:]
probabilities = torch.tensor([[[0.1, 0.5, 0.3, 0.1],
                               [0.1, 0.2, 0.25, 0.45],
                               [0.25, 0.25, 0.25, 0.25]]])
logits = probabilities.log().detach().requires_grad_()
loss = next_token_loss(logits, targets)
loss.backward()
print("input IDs", inputs.tolist(), "target IDs", targets.tolist())
print(f"mean NLL {loss.item():.4f}; perplexity {loss.exp().item():.4f}")
# input IDs [[0, 1, 2]] target IDs [[1, 2, 3]]
# mean NLL 1.1552; perplexity 3.1748

The observed targets receive probabilities 0.5, 0.25, and 0.25. Their negative log-probabilities average to about 1.1552 nats; exponentiating gives perplexity about 3.1748. The equation above is a sum, while the code averages over included targets. For validation across batches, divide total negative log-likelihood by the total number of included tokens, rather than averaging unequal-sized batch means. Loss and perplexity comparisons require the same tokenizer and comparable evaluation data.

Self-attention, masking, and the full block are derived in Transformers from Scratch: Self-Attention to Encoder–Decoder.

Scaling laws

Within a controlled training setup, held-out language-model loss can often be fitted by a power-law relationship. Let \(N\) denote dense model parameters, \(D\) the number of training tokens processed, and \(C\) the training FLOPs. One fitted form is:

\[L(N,D)\approx E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}\]

The positive constants \(A,B,\alpha,\beta\) describe how excess loss decreases with scale; \(E\) is a fitted asymptotic floor for this model of the data and training procedure. These are empirical fit parameters, not known universal constants. Small runs at several parameter/token budgets can estimate them. Hold out some runs to check prediction error before extrapolating, and refit when the tokenizer, data mixture, architecture, or training regime changes. A smooth loss forecast is not a direct forecast of a particular benchmark score.

For a dense transformer when parameter-matrix operations dominate, \(C\approx6ND\) is a useful training-compute approximation. It excludes some costs, including context-dependent attention work and recomputation, and is not a wall-clock estimate. Substitute \(D=C/(6N)\) into the fitted loss: increasing \(N\) reduces its parameter term but increases its data term. Setting the derivative with respect to \(N\) to zero yields \(\alpha A N^{-\alpha}=\beta B D^{-\beta}\), and \(N\) scales as \(C^{\beta/(\alpha+\beta)}\). Equal parameter and token scaling therefore requires approximately equal exponents; it is not automatic from the equation.

Hoffmann et al. found approximately balanced parameter/token scaling in their experiments. The often-used 20 tokens per parameter is a planning heuristic drawn from that setting. Chinchilla’s 70B parameters and 1.4T tokens were compared at a budget comparable to Gopher’s 280B parameters, not GPT-3’s 175B on 300B tokens. Under the same 6ND approximation, those Chinchilla and GPT-3 configurations cost about 5.88×10²³ and 3.15×10²³ FLOPs respectively. Exercise 1 compares allocations at the same estimated GPT-3 budget.

def allocation_at_ratio(flops, tokens_per_param=20):
    """Budget allocation under C=6ND and an assumed D/N ratio; no fitted optimum."""
    if not all(math.isfinite(v) and v > 0 for v in (flops, tokens_per_param)):
        raise ValueError("Compute and token/parameter ratio must be positive and finite")
    n = math.sqrt(flops / (6 * tokens_per_param))
    return n, tokens_per_param * n

for flops in [1e21, 1e22, 1e23, 1e24]:
    n, d = allocation_at_ratio(flops)
    print(f"{flops:.0e} FLOPs -> {n/1e9:.1f}B params, {d/1e12:.2f}T tokens")
# 1e+21 FLOPs -> 2.9B params, 0.06T tokens
# 1e+22 FLOPs -> 9.1B params, 0.18T tokens
# 1e+23 FLOPs -> 28.9B params, 0.58T tokens
# 1e+24 FLOPs -> 91.3B params, 1.83T tokens

Training-compute allocation is only one planning objective. A smaller dense model can reduce serving memory and per-token computation, so expected deployment volume can justify spending more pretraining tokens on fewer parameters. The resulting choice also depends on achievable quality, context length, batching, latency, and hardware. No single tokens-per-parameter ratio minimizes all of those costs. Likewise, repeated tokens need not provide the same benefit as additional unique text even though both count toward compute.

Constructing the training corpus

A corpus pipeline determines what information the model repeatedly encounters. Its stages may be interleaved, and their effects should be measured on the intended domains:

1. Extraction and language identification — converting raw crawl data into text, discarding boilerplate and navigation.
2. Quality filtering — checks on extraction errors, length, repetition, and source quality. A classifier or heuristic filter can also remove useful minority-domain or multilingual material, so inspect what it excludes.
3. Deduplication — exact hashing, approximate document matching such as MinHash, and repeated-substring detection address different forms of repetition.
4. Benchmark decontamination — detecting overlap with held-out evaluations and documenting removal rules. Missed overlap can inflate scores; overly broad matching can remove legitimate material.
5. Mixing — choosing source sampling weights for the intended tasks. Upweighting a source increases its exposure and can make it repeat before other sources are exhausted.

Removing duplicates can reduce memorization and train/test overlap, and published experiments have found benefits for training efficiency and evaluation. The size of the benefit depends on the corpus and duplicate definition. Repetition is not intrinsically useless, and deduplication is not guaranteed to improve every score: removing contamination can lower an inflated score while making the measurement more trustworthy. Near-duplicate detection also does not certify that an evaluation set is uncontaminated.

Code is a useful source when code-related behavior is desired; its effect on other tasks should be tested rather than assumed. Compare data mixtures at a matched compute or token budget and evaluate each target domain. A mixture that improves one group of tasks can reduce another group’s exposure.

After filtering and mixing, tokenize the text, mark document boundaries, and pack it into training sequences. Padding targets must be excluded from the loss. Packing independent documents together also requires a policy for attention across boundaries: an EOS token marks a boundary but does not itself block attention. Track both the token count in the deduplicated corpus and tokens actually processed, plus the per-source sampling counts. Here “unique data” refers to distinct retained text, not the number of token types in the vocabulary. Recording these counts distinguishes new text from repeated exposure.

The training run

One pass over the stored corpus does not rule out memorization, distribution mismatch, or a validation gap. The corpus may already contain duplicates, and weighted sampling can repeat selected sources. Experiments on data-constrained language models found that several passes can remain useful under their conditions, with diminishing returns as repetition grows. Four passes is not a universal boundary at which quality starts to decline. Monitor held-out performance and the composition of the reused data.

When loss spikes, inspect batch contents, sequence lengths, nonfinite activations or gradients, learning-rate changes, and distributed failures before choosing a recovery. Restoring a checkpoint may help, but silently skipping a valid difficult batch changes the training distribution and can hide a recurring problem. A resumable checkpoint includes model and optimizer states, scheduler state, random-number state, and the data-stream position; record any discarded batches and the reason.

For fixed-length sequences without padding, a microbatch of 2 sequences × 2,048 tokens × 8 accumulation steps × 4 data-parallel replicas gives 131,072 tokens per optimizer update. Tensor-parallel ranks share the same examples and do not add another independent batch factor. With variable-length examples, count included targets and weight accumulated losses accordingly.

Parallelism

A common memory estimate for mixed-precision Adam is 16 bytes per parameter: 2 for low-precision weights, 2 for gradients, 4 for a float32 master copy, and 8 for two float32 moments. For 70B parameters this is 1.12 trillion bytes, or about 1.12 decimal TB (1,043 GiB), before activations, temporary buffers, or communication storage. Other precision and optimizer implementations have different budgets. Sharding and parallel execution divide this state and computation:

• Data parallel — different examples per replica. Replicated training averages gradients; sharded variants also partition parameters, gradients, or optimizer state and use operations such as reduce-scatter and all-gather.
• Tensor parallel — individual matrix operations split across devices, with communication during layer execution. Fast links are valuable, but the split is not restricted to a single node.
• Pipeline parallel — layer groups placed on different devices. Microbatches improve overlap, though startup, drain, and imbalance can leave idle intervals.
• Sequence/context parallel — sequence parallelism can shard selected token-wise activations alongside tensor parallelism; context parallelism partitions long sequences while arranging attention communication. Implementations use these terms differently, so specify which operations and tensors are distributed.

Model FLOPs utilization (MFU) compares estimated useful model arithmetic per second with the selected hardware peak. Using the same dense approximation, \(\mathrm{MFU}\approx6N\times\text{global tokens/second}/(\text{device count}\times\text{peak FLOPs/second per device})\). Specify the precision and peak convention, and whether the model FLOP estimate includes attention. MFU excludes extra recomputation by definition; hardware utilization can count it. A low MFU does not identify the cause: communication, memory bandwidth, small kernels, input stalls, and recomputation all warrant profiling. There is no universal 30% cutoff that selects a better parallel layout.

The token model and a small training loop are implemented in Building a Transformer Language Model in PyTorch. That article does not implement the distributed pretraining system described here.

Emergence, and how to read it

Some reported task scores improve abruptly between measured model scales. Metric choice, finite evaluation sets, sparse sampling of scales, and decoding settings can contribute to that appearance. Exact match gives each answer a binary score, but its expected value can vary smoothly with model behavior. Analyses of apparent emergence have demonstrated metric-dependent effects; they do not establish that every capability change has the same explanation.

Evaluate several scales where feasible, keep prompts and decoding settings comparable, and report uncertainty. Pair a task’s operational score with graded or probabilistic measurements when those are meaningful. A failed small model does not prove a larger one will fail, and a smooth language-model loss curve does not promise that a particular capability will appear.

Pretraining learns a distribution of text continuations. A base model may already imitate conversations or follow some instructions present in its training distribution, but this behavior can be unreliable. Post-training aims to make instruction-following and other desired behaviors more consistent; it does not supply guarantees by itself.

Turning a pretrained model into an assistant is covered in Fine-Tuning and Alignment: SFT, LoRA, RLHF, and DPO.

Sparse expert layers that distinguish total parameter capacity from the parameters activated per token are covered in Mixture of Experts: Sparse Models Explained.

Exercises

1. A matched-compute allocation. Use \(C=6ND\) to estimate the budget of 175B parameters trained on 300B tokens. Reallocate that budget under an assumed 20 tokens per parameter. How does this differ from proving a loss-minimizing allocation?

gpt3_n, gpt3_d = 175e9, 300e9
compute = 6 * gpt3_n * gpt3_d
n, d = allocation_at_ratio(compute, 20)
print(f"estimated budget {compute:.3e} FLOPs")
print(f"original tokens/parameter {gpt3_d/gpt3_n:.2f}")
print(f"ratio-constrained allocation {n/1e9:.2f}B parameters, {d/1e12:.3f}T tokens")
print(f"new tokens/parameter {d/n:.2f}; budget ratio {6*n*d/compute:.3f}")
# estimated budget 3.150e+23 FLOPs
# original tokens/parameter 1.71
# ratio-constrained allocation 51.23B parameters, 1.025T tokens
# new tokens/parameter 20.00; budget ratio 1.000

2. What does a 30% duplicate fraction identify? Suppose the raw corpus contains ten equal-length documents: A appears four times and B through G once each. Deduplication retains one of each. Compare one raw pass with uniform sampling of ten documents from the deduplicated corpus.

from collections import Counter
raw = ["A", "A", "A", "A", "B", "C", "D", "E", "F", "G"]
counts = Counter(raw)
removed_fraction = (len(raw) - len(counts)) / len(raw)
budget = len(raw)
print(f"removed fraction {removed_fraction:.2f}")
print(f"raw passes {budget/len(raw):.2f}; unique-corpus equivalents {budget/len(counts):.2f}")
print(f"raw exposures: A={counts['A']}, B={counts['B']}")
print(f"uniform deduplicated sampling: expected exposures per document {budget/len(counts):.2f}")
other_raw = ["A", "A", "B", "B", "C", "C", "D", "E", "F", "G"]
print("same unique count", len(set(other_raw)) == len(counts))
print("other raw maximum multiplicity", max(Counter(other_raw).values()))
# removed fraction 0.30
# raw passes 1.00; unique-corpus equivalents 1.43
# raw exposures: A=4, B=1
# uniform deduplicated sampling: expected exposures per document 1.43
# same unique count True
# other raw maximum multiplicity 2

3. Smooth exact match and thresholded reporting. Assume five independent token-correctness events, each with probability p. Compute the expected whole-answer exact match as p rises. Separately report whether that expectation is at least 0.10. Which quantity has a discontinuity?

for n in [1e8, 1e9, 1e10, 1e11, 1e12, 1e13]:
    p_correct = 0.30 + 0.12 * (math.log10(n) - 8)
    expected_exact = p_correct ** 5
    above_threshold = expected_exact >= 0.10
    print(f"N={n:.0e} token={p_correct:.3f} expected_exact={expected_exact:.4f}"
          f" threshold_met={above_threshold}")
# N=1e+08 token=0.300 expected_exact=0.0024 threshold_met=False
# N=1e+09 token=0.420 expected_exact=0.0131 threshold_met=False
# N=1e+10 token=0.540 expected_exact=0.0459 threshold_met=False
# N=1e+11 token=0.660 expected_exact=0.1252 threshold_met=True
# N=1e+12 token=0.780 expected_exact=0.2887 threshold_met=True
# N=1e+13 token=0.900 expected_exact=0.5905 threshold_met=True

References

• Kaplan et al. (2020). Scaling Laws for Neural Language Models. arXiv preprint.
• Brown et al. (2020). Language Models Are Few-Shot Learners. NeurIPS.
• Lee et al. (2022). Deduplicating Training Data Makes Language Models Better. ACL.
• Hoffmann et al. (2022). Training Compute-Optimal Large Language Models.
• Wei et al. (2022). Emergent Abilities of Large Language Models.
• Penedo et al. (2023). The RefinedWeb Dataset for Falcon LLM.
• Schaeffer, Miranda, and Koyejo (2023). Are Emergent Abilities of Large Language Models a Mirage? NeurIPS.
• Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint.
• Chowdhery et al. (2023). PaLM: Scaling Language Modeling with Pathways. JMLR.
• Muennighoff et al. (2023). Scaling Data-Constrained Language Models.
• Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.