Fine-Tuning Experiments and Failure Analysis

Consider a hypothetical team that fine-tunes a language model on 2,000 support tickets and reports that the tuned model scores 71% on its evaluation set while the original scores 64%. Before anyone ships it, a series of questions decides whether the seven points mean anything. Was the 64% produced with a prompt written as carefully as the training data? Would a different random seed give 68% or 74%? Is the gain spread across ticket types or concentrated in one? Did the model get worse at something the team never measured, such as following the output format or declining unsafe requests? And if the gain is real, which part of the recipe produced it?

A fine-tuning experiment answers those questions by design, not by inspection afterward. This article covers the design: which baselines a result has to be compared with, how to tell a real difference from run-to-run noise, how to treat parameter-efficient methods as experimental arms, how to run ablations that isolate one factor, how to recognize overfitting and catastrophic forgetting in the measurements, and how to analyze the failures that remain. It assumes the dataset and split discipline from Designing Fine-Tuning Datasets and Evaluation Splits (task data, labels, leakage, and a held-out evaluation set that was never used for choices), and the LoRA and SFT mechanics from Fine-Tuning and Alignment: SFT, LoRA, RLHF, and DPO.

Hosted-model fine-tuning cannot be run in this setting, so every number below comes from a small NumPy model that fine-tuning can be run on in seconds. The model is a stand-in, not a language model: it lets the experimental procedure be shown end to end, with real outputs, and the section on each procedure says how it maps to an LLM. The ranking of methods that the toy produces is a property of the toy and should not be carried over to other models.

What a fine-tuning experiment has to establish

A result such as “adapted beats base” is a claim with parts, and each part needs its own evidence.

• The gain exists. On held-out data the adapted model scores higher than the starting model by more than run-to-run and sampling noise.
• The gain is caused by the adaptation. A comparable cheaper alternative (a better prompt, a simpler adaptation, more data of the same kind) does not deliver the same gain.
• The gain comes from the part of the recipe you credit. Removing or changing that part (rank, target layers, data amount, data mixture) moves the result in the expected direction.
• The cost is known. Capabilities outside the target task, such as general behavior, format following, and refusals, did not degrade beyond what you accept.
• The remaining errors are understood. Failures are grouped by cause, so the next experiment is chosen from evidence.

Each claim maps to a section below. The mapping also fixes what the experiment must record: the exact data, split, prompt or template, decoding settings, hyperparameters, seed, and checkpoint for every row of the results table, because a number that cannot be tied back to those cannot be compared with another number. Practices for reproducible runs (seeds, configuration, checkpoints) are covered in Reproducible Training Pipelines: Seeds, Configs, and Checkpoints.

A testbed with a known answer

The testbed is a classifier with a hidden layer, \(\text{logits}=V\,\mathrm{relu}(Wx+b_1)+b_2\), with 64 hidden units and 5 output classes. Inputs have 19 content features plus one “mode” feature. A fixed random two-layer teacher network assigns each content vector a general label; the mode feature (+3 or −3) is ignored by the teacher. The base model is pretrained on 20,000 examples of both modes with the general labels, so it knows the general task and has never needed the mode feature.

The adaptation target is a domain that relabels the general answer: in domain mode (+3), the correct label for general classes 0 and 1 is swapped, and classes 2, 3, and 4 keep their general labels. This stands in for a behavior change that conflicts with the base model’s habits on part of the input space, such as a house style that reverses a convention. General mode (−3) keeps the general labels and plays the role of the capabilities the adapted model must retain. The fine-tuning set contains 600 domain-mode examples, 10% of whose labels are replaced by a random wrong class to imitate annotation noise. The dev set (400 examples) and test set (2,000 examples) are clean and drawn from domain mode; two further sets of general-mode examples (400 for dev, 2,000 for test) measure retention. All data are synthetic and generated by the code.

The excerpt below defines the data generator and the model. The code for backpropagation, the Adam optimizer, and pretraining (about 40 lines of standard code) is not reproduced; the listings here are the parts that define the experiment. Environment: Python 3.12 and NumPy 2.2; all outputs shown in this article are the printed output of these programs.

import numpy as np

D_CONTENT, D, H, K = 19, 20, 64, 5     # content features, +1 mode feature, hidden units, classes
PERM = np.array([1, 0, 2, 3, 4])       # the domain's relabeling of the general answer

def make_teacher(seed=0):
    r = np.random.default_rng(seed)
    P = r.normal(size=(32, D_CONTENT)) / np.sqrt(D_CONTENT)
    U = r.normal(size=(K, 32))
    return lambda xc: np.argmax(np.maximum(xc @ P.T, 0) @ U.T, axis=1)

TEACHER = make_teacher()

def sample(n, mode, rng):
    """mode=+1: domain inputs, mode=-1: general inputs. Returns X (n, D) and general-task labels."""
    xc = rng.normal(size=(n, D_CONTENT))
    X = np.hstack([xc, np.full((n, 1), 3.0 * mode)])
    return X, TEACHER(xc)

def domain_labels(y_general): return PERM[y_general]

def noisy(y, rate, rng):
    y = y.copy(); flip = rng.random(len(y)) < rate
    y[flip] = (y[flip] + rng.integers(1, K, flip.sum())) % K
    return y

def forward(p, X, lora=None):
    W, V = p["W"], p["V"]
    if lora:
        s = lora["alpha"] / lora["r"]
        if "W" in lora["targets"]: W = W + s * lora["BW"] @ lora["AW"]
        if "V" in lora["targets"]: V = V + s * lora["BV"] @ lora["AV"]
    Z = X @ W.T + p["b1"]; Hh = np.maximum(Z, 0)
    return Hh @ V.T + p["b2"], (Z, Hh, W, V)


def make_lora(rng, r, targets, alpha=None):
    lo = {"r": r, "alpha": alpha or r, "targets": targets}   # alpha=r -> scale 1
    if "W" in targets: lo["AW"] = rng.normal(size=(r, D)) * 0.1; lo["BW"] = np.zeros((H, r))
    if "V" in targets: lo["AV"] = rng.normal(size=(r, H)) * 0.1; lo["BV"] = np.zeros((K, r))
    return lo

The LoRA parameterization follows Hu et al.: the frozen matrix \(W\) is augmented with \(\tfrac{\alpha}{r}BA\), where \(B\) starts at zero so the adapted model equals the base model at step 0. With \(\alpha=r\) the scale is 1. The two matrices a LoRA can adapt here are the first-layer weight \(W\) (64×20) and the output weight \(V\) (5×64). For a matrix with \(d_\text{out}\) rows and \(d_\text{in}\) columns, rank \(r\) costs \(r(d_\text{out}+d_\text{in})\) trainable entries, so rank 4 on \(V\) costs \(4\times(5+64)=276\), against 1,669 for full fine-tuning of this model.

def make_eval_sets(seed=999, n_dev=400, n_test=2000):
    rng = np.random.default_rng(seed)
    out = {}
    Xd, yd = sample(n_dev, +1, rng);  out["dev"] = (Xd, domain_labels(yd))
    Xt, yt = sample(n_test, +1, rng); out["test"] = (Xt, domain_labels(yt))
    Xg, yg = sample(n_test, -1, rng); out["retain"] = (Xg, yg)       # general-mode inputs, general labels
    Xr, yr = sample(n_dev, -1, rng);  out["retain_dev"] = (Xr, yr)   # a small general-mode dev set for choosing checkpoints
    return out

def finetune(method, lr, epochs, seed, n_train=200, noise=0.1, replay=0, r=4, targets=("W", "V"), evals=None, batch=32, shuffle=False):
    """method: 'full' | 'head' | 'lora'. Returns per-epoch history of train/dev/test/retain metrics."""
    rng = np.random.default_rng(1000 + seed)
    p = base_params()
    X, yg = sample(n_train, +1, rng); y_clean = domain_labels(yg); y = noisy(y_clean, noise, rng)
    if shuffle: y = rng.permutation(y)
    if replay:
        Xr, yr = sample(replay, -1, rng); X, y = np.vstack([X, Xr]), np.concatenate([y, yr])
    lora = make_lora(rng, r, tuple(targets)) if method == "lora" else None
    trainable = {"full": ["W", "b1", "V", "b2"], "head": ["V", "b2"]}.get(method)
    opt = Adam(lr); hist = []
    def snapshot(ep):
        row = {"epoch": ep, "train": loss_acc(p, X, y, lora)}
        for k, (Xe, ye) in evals.items(): row[k] = loss_acc(p, Xe, ye, lora)
        hist.append(row)
    snapshot(0)
    for ep in range(1, epochs + 1):
        idx = rng.permutation(len(X))
        for i in range(0, len(X), batch):
            b = idx[i:i + batch]; g = grads(p, X[b], y[b], lora)
            if method == "lora":
                tr = {k: lora[k] for k in g if k in lora}; opt.step(tr, {k: g[k] for k in tr})
                for k in tr: lora[k] = tr[k]
            else:
                opt.step(p, {k: g[k] for k in trainable})
        snapshot(ep)
    return hist, (p, lora), (X[:n_train], y[:n_train], y_clean)

Each call of finetune returns the train, dev, test, and retention loss and accuracy after every epoch. Every later experiment is a different way of reading these histories.

Baselines: what the adapted model must beat

A comparison with the untouched base model is the weakest test, because it measures only that training on task data changed something. A useful baseline ladder starts with the cheapest alternatives and ends with the strongest, and the adapted model is credited only for what it adds beyond the last rung that is practical to deploy.

Rung	What it is	What beating it shows
Trivial	Majority class, a keyword rule, or the previous production system	The task is not solved by a shortcut
Base, as-is	The pretrained or instruction-tuned checkpoint with its default prompt	Training changed behavior (the weakest claim)
Base, tuned prompt	The same checkpoint with task instructions, a format specification, and a few examples, all written with a budget comparable to the data effort	Fine-tuning beats the cheapest adaptation that needs no training
Base plus retrieval or tools	The base model given the relevant documents or functions at inference time	The behavior needs to be in the weights, not in the context
Simple adaptation	Training only a small part (an output head, a few biases, a small adapter) with the same data	The extra capacity of the heavier method is used
Previous tuned model	The last model you shipped, evaluated on the same current test set	The new recipe improves on the incumbent

Three rules keep the ladder honest. First, every rung is scored by the same harness: the same evaluation examples, prompt template, decoding settings (a fixed temperature and maximum length, ideally greedy decoding), parser, and metric. A base model scored with a different output parser than the tuned model is a different experiment. Second, rungs that involve prompt writing get a tuning budget comparable to what training received, and the tuning is done on dev data, not on the test set. Third, compute and data are reported next to the score, since a baseline that is 1.5 points worse at one-tenth the cost may be the better decision. The reasoning about when retrieval, prompting, or fine-tuning serves a need is developed in Fine-Tuning and Alignment: SFT, LoRA, RLHF, and DPO; the general discipline of deciding what to try next from measured gaps is covered in A Systematic Strategy for Improving Machine Learning Models.

The toy comparison

The first experiment evaluates the base model, a majority-class guess, and four adapted variants on the domain test set. The variants are head-only training (the output layer \(V\) and \(b_2\) are trained, \(W\) is frozen), LoRA of rank 4 on \(V\), LoRA of rank 4 on both \(W\) and \(V\), and full fine-tuning. Each method gets its own learning rate from the grid 3e-4 to 1e-1 and its own epoch count (at most 40), chosen by mean dev accuracy over 5 seeds. Choosing the learning rate per method matters because methods that update few parameters usually need different step sizes than full updates; comparing all methods at one learning rate favors whichever method that rate happens to suit.

import numpy as np
from ftlib import *

ev = make_eval_sets()
SEEDS, EPOCHS, N_TRAIN = range(5), 40, 600
METHODS = {"head only": ("head", {}), "LoRA r=4 on V": ("lora", {"r": 4, "targets": ("V",)}),
           "LoRA r=4 on W,V": ("lora", {"r": 4, "targets": ("W", "V")}), "full": ("full", {})}
GRID = (3e-4, 1e-3, 3e-3, 1e-2, 3e-2, 1e-1)

base = base_params()
y_test = ev["test"][1]
print("majority class on domain test:", round(np.bincount(y_test).max() / len(y_test), 3))
print("base model: domain test", round(loss_acc(base, *ev["test"])[1], 3), " general retain", round(loss_acc(base, *ev["retain"])[1], 3))

print(f"{'method':18s}{'trainable':>10s}  lr    epoch  dev    test (mean ± sd)   retain (mean ± sd)")
chosen = {}
for name, (m, kw) in METHODS.items():
    runs = {lr: [finetune(m, lr, EPOCHS, s, n_train=N_TRAIN, evals=ev, **kw)[0] for s in SEEDS] for lr in GRID}
    # choose (lr, epoch) by mean dev accuracy over seeds
    best = max(((np.mean([h[e]["dev"][1] for h in runs[lr]]), lr, e) for lr in GRID for e in range(1, EPOCHS + 1)))
    dev, lr, e = best; chosen[name] = (lr, e)
    te = np.array([h[e]["test"][1] for h in runs[lr]]); re = np.array([h[e]["retain"][1] for h in runs[lr]])
    print(f"{name:18s}{count_trainable(m, **{k: v for k, v in kw.items()}):>10d}  {lr:<6g}{e:<6d} {dev:.3f}  {te.mean():.3f} ± {te.std(ddof=1):.3f}      {re.mean():.3f} ± {re.std(ddof=1):.3f}")
print(chosen)

majority class on domain test: 0.242
base model: domain test 0.554  general retain 0.926
method             trainable  lr    epoch  dev    test (mean ± sd)   retain (mean ± sd)
head only                325  0.03  12     0.734  0.737 ± 0.013      0.454 ± 0.036
LoRA r=4 on V            276  0.003 25     0.810  0.825 ± 0.020      0.578 ± 0.015
LoRA r=4 on W,V          612  0.01  11     0.734  0.751 ± 0.020      0.533 ± 0.040
full                    1669  0.03  2      0.662  0.673 ± 0.012      0.502 ± 0.035
{'head only': (0.03, 12), 'LoRA r=4 on V': (0.003, 25), 'LoRA r=4 on W,V': (0.01, 11), 'full': (0.03, 2)}

The base model scores 55.4% on the domain test set and a majority-class guess scores 24.2%, so the base model already knows part of the domain behavior: it is right on the examples whose correct class is unchanged. All four adapted variants improve on it. LoRA on \(V\) is best at 82.5% ± 2.0 (mean and standard deviation over 5 seeds), then LoRA on \(W,V\) at 75.1%, head-only at 73.7%, and full fine-tuning at 67.3%. The selected learning rate for every method lies inside the grid and not at its edge, which is the check that the grid was wide enough; when the best value sits at an edge, extend the grid before comparing.

The last column carries a warning. On general-mode examples, where the correct behavior is the unchanged general label, base accuracy is 92.6% and every adapted variant is far below it (45% to 58%). The adapted models learned the domain relabeling and applied it in general mode too, because the fine-tuning set contained no general-mode example that would penalize that. An experiment that reported only the domain column would have called all four methods successes. The retention measurement is developed in the section on forgetting.

The ordering among methods is specific to this setup. Full fine-tuning is last here because the noisy labels let it memorize, a behavior examined below, and because the toy has only 600 examples. In the paper “LoRA Learns Less and Forgets Less”, Biderman et al. report the opposite learning outcome for large models on code and mathematics: standard-rank LoRA underperformed full fine-tuning on the target task while forgetting less of the base model’s behavior. Both findings can hold, since the regime (data volume, label quality, rank, training length) differs; this is why the comparison has to be run in the regime you care about.

Is the difference larger than the noise?

A single run of each method yields one number per method. Two sources of variation sit between that number and a conclusion. Training randomness (the seed that sets adapter initialization and data order) changes the model; Dodge et al. found that different seeds with identical hyperparameters can give substantially different fine-tuning results for BERT on GLUE tasks, and that weight initialization and data order contributed about equally. Evaluation sampling changes the estimate even for a fixed model: accuracy on \(n\) examples has a binomial standard error of approximately

\[\mathrm{SE}=\sqrt{\frac{p(1-p)}{n}}\]

where \(p\) is the true accuracy. At \(p=0.8\) this is 0.040 for \(n=100\), 0.020 for \(n=400\), and 0.009 for \(n=2000\). A 2-point difference on a 400-example dev set is within one standard error of nothing. The standard-error formula treats each example as an independent coin flip; correlated examples (many tickets from one customer, near-duplicates) make the effective sample smaller than \(n\).

When two models are scored on the same examples, a paired analysis is tighter than comparing two independent accuracies, because the examples that are easy for both models cancel. The paired bootstrap resamples examples (with replacement), recomputes the accuracy difference on each resample, and reads a 95% interval from the 2.5th and 97.5th percentiles. The pairing is preserved by using the same resampled indices for both models. The statistical background is in Comparing Models: Uncertainty and Significance.

def paired_bootstrap(a, b, n_boot=5000, seed=0):
    """a, b: 0/1 correctness on the SAME examples. Resample examples, keep the pairing."""
    rng = np.random.default_rng(seed); n = len(a)
    idx = rng.integers(0, n, size=(n_boot, n))
    d = (a[idx] - b[idx]).mean(1)
    return (a - b).mean(), np.percentile(d, [2.5, 97.5])

L, Hd, B = correct["LoRA r=4 on V"][0], correct["head only"][0], correct["base"]
for label, a, b in [("LoRA(seed 0) - base", L, B), ("head(seed 0) - base", Hd, B), ("LoRA(seed 0) - head(seed 0)", L, Hd)]:
    d, ci = paired_bootstrap(a.astype(float), b.astype(float))
    print(f"{label:28s} diff {d:+.3f}  95% CI [{ci[0]:+.3f}, {ci[1]:+.3f}]")

# unpaired width for comparison: treat the two accuracy estimates as independent
pa, pb, n = L.mean(), Hd.mean(), len(L)
se = np.sqrt(pa * (1 - pa) / n + pb * (1 - pb) / n)
print(f"unpaired normal-approx 95% half-width for LoRA-head: {1.96 * se:.3f}")
d_seed = acc["LoRA r=4 on V"] - acc["head only"]
print("seed-matched LoRA - head differences:", np.round(d_seed, 3), "mean", round(d_seed.mean(), 3), "sd", round(d_seed.std(ddof=1), 3))

per-seed test accuracy
  base            0.554
  head only       [0.715 0.74  0.74  0.742 0.748]
  LoRA r=4 on V   [0.819 0.794 0.836 0.831 0.844]
seed sd: {'head only': 0.0127, 'LoRA r=4 on V': 0.0196}
LoRA(seed 0) - base          diff +0.265  95% CI [+0.240, +0.289]
head(seed 0) - base          diff +0.161  95% CI [+0.131, +0.190]
LoRA(seed 0) - head(seed 0)  diff +0.104  95% CI [+0.084, +0.123]
unpaired normal-approx 95% half-width for LoRA-head: 0.026
seed-matched LoRA - head differences: [0.104 0.053 0.097 0.088 0.096] mean 0.088 sd 0.02
binomial standard error of one accuracy near 0.8 at n=100: 0.040
binomial standard error of one accuracy near 0.8 at n=400: 0.020
binomial standard error of one accuracy near 0.8 at n=2000: 0.009
dev accuracy by seed [0.795 0.775 0.822 0.81  0.848] -> picked seed 4 test 0.844 vs seed-mean 0.825

The per-seed rows show the training-randomness part: head-only test accuracy ranges from 71.5% to 74.8% across seeds (sd 1.3 points) and LoRA on \(V\) from 79.4% to 84.4% (sd 2.0). The bootstrap intervals show the evaluation-sampling part for one fixed model per method: LoRA on \(V\) beats the base model by 26.5 points with an interval of ±2.5 points, and beats head-only by 10.4 points with an interval from 8.4 to 12.3. The unpaired normal approximation for that last difference has a half-width of 2.6 points, wider than the paired half-width of about 2.0, which shows the gain from pairing. Matching seeds across methods gives per-seed differences from 5.3 to 10.4 points, all positive.

Both sources count. The bootstrap interval says nothing about seed variation, since it uses one trained model, and the seed spread says nothing about how well the test set represents the task. A defensible claim needs a difference that is large relative to both, and a decision rule fixed before looking: for example, report the mean over at least 3 to 5 seeds with the spread, and call a difference real only when the paired interval and the seed-matched differences agree in sign. Selecting the best seed after the fact is a form of tuning on the test set; here the dev-selected seed scored 84.4% on test while the 5-seed mean was 82.5%, a 1.9-point flattering of the result.

For LLM fine-tuning the same structure applies with costlier runs. Each run is a training job, so the budget often allows 3 seeds, not 30. Use more evaluation examples instead: the evaluation cost is cheaper than the training cost, and it removes one of the two sources. If the score comes from a model judge or a human rubric, the judge’s own variability is a third source and needs repeated judging of a sample.

Parameter-efficient methods as experimental arms

Parameter-efficient fine-tuning (PEFT) trains a small set of added or selected parameters and freezes the rest. The family has several members with different mechanics, and an experiment that lists “PEFT” as one arm hides choices that change the outcome.

Method	What is trained	Typical knobs
Full fine-tuning	All weights	Learning rate, schedule, epochs, weight decay
Output-head or layer-subset training	The last layer or last few blocks	Which layers; learning rate
Adapters (Houlsby et al.)	Small bottleneck modules inserted into each layer; base weights frozen	Bottleneck size, insertion points
LoRA (Hu et al.)	Low-rank factors \(B,A\) added to selected weight matrices	Rank \(r\), scale \(\alpha\), target matrices, dropout, learning rate
Prompt and prefix methods	Learned vectors prepended to the input or to attention keys and values	Number of virtual tokens, initialization

Houlsby et al. report that adapters on BERT came within 0.4% of full fine-tuning on the GLUE benchmark while adding about 3.6% parameters per task. Hu et al. report that LoRA matched or exceeded full fine-tuning on several benchmarks with a large reduction in trainable parameters. Those are results under particular models, tasks, and budgets; they motivate PEFT as an arm in the experiment, not as a foregone conclusion.

Four design rules apply to PEFT arms.

• Tune each arm separately. The learning rate that suits full updates is generally not the one that suits adapters. The grid and the budget should be the same size for each arm, and the selected value should not sit at a grid edge.
• Count what you compare. Report trainable parameters, but also the training memory, the time per step, and the size of what must be stored and served per task. Trainable parameters are a proxy: LoRA on \(V\) in the toy has 276 trainable entries yet outperforms an 1,669-entry full update, while LoRA on \(W,V\) with 612 entries is worse than LoRA on \(V\) alone.
• Vary where as well as how much. Target matrices are a design choice with an effect as large as rank. In the toy, putting the adapter on the output matrix (the place where the behavior change lives) beat putting it on the first layer.
• Keep the base checkpoint exact. An adapter is meaningful only relative to the weights it was trained against. Record the checkpoint hash and tokenizer version with every adapter.

The toy results from the first experiment show how these rules change the reading. The method with the fewest trainable parameters among the competitive arms wins on the target task, while the arm with the most trainable parameters loses, and a larger adapter (W and V) is worse than a smaller one. Without per-arm tuning and several seeds, each of those observations could have been produced by a badly chosen learning rate or one lucky run.

Ablations: changing one thing at a time

An ablation takes the complete recipe, changes one factor, and measures what that factor contributed. Three conditions make it informative. Everything else is held fixed, including the seeds and the evaluation set. Each arm is tuned in the way the baseline was, since removing a component often requires retuning the rest (the best epoch count changes when data size changes). And the measured quantity is the one the claim is about, which includes retention if the claim is that the recipe is safe.

The factors worth ablating in a fine-tuning recipe are those that cost money or carry a hidden assumption: adapter rank and target matrices, the amount of training data, label noise, the data mixture, the prompt or chat template, the loss mask (only assistant tokens or all tokens), the learning rate and schedule, and the number of epochs. Ablate in order of cost and doubt: the most expensive component and the one you are least sure about come first.

import numpy as np
from ftlib import *

ev = make_eval_sets(); SEEDS, EPOCHS = range(5), 40

def arm(label, method, lr, **kw):
    """One ablation arm: everything fixed except what is in kw. The epoch is chosen on dev (mean over seeds)."""
    hs = [finetune(method, lr, EPOCHS, s, evals=ev, **kw)[0] for s in SEEDS]
    e = max(range(1, EPOCHS + 1), key=lambda e: np.mean([h[e]["dev"][1] for h in hs]))
    te = np.array([h[e]["test"][1] for h in hs]); re = np.array([h[e]["retain"][1] for h in hs])
    print(f"{label:34s} epoch {e:2d}  test {te.mean():.3f} ± {te.std(ddof=1):.3f}   retain {re.mean():.3f}")

print("-- rank (LoRA on V, n=600, lr 3e-3); the update to V has shape 5 x 64, so rank cannot exceed 5")
for r in (1, 2, 4, 8): arm(f"r={r}", "lora", 3e-3, n_train=600, r=r, targets=("V",))
print("-- target modules (r=4)")
for t in (("W",), ("V",), ("W", "V")): arm("targets=" + ",".join(t), "lora", 3e-3, n_train=600, r=4, targets=t)
print("-- training-set size")
for n in (100, 200, 600, 2000):
    arm(f"LoRA r=4 on V, n={n}", "lora", 3e-3, n_train=n, r=4, targets=("V",))
    arm(f"full,            n={n}", "full", 3e-2, n_train=n)
print("-- negative control: labels shuffled across examples")
arm("LoRA r=4 on V, shuffled labels", "lora", 3e-3, n_train=600, r=4, targets=("V",), shuffle=True)
print("-- label noise in the training set")
for nz in (0.0, 0.1, 0.3):
    arm(f"LoRA r=4 on V, noise={nz}", "lora", 3e-3, n_train=600, r=4, targets=("V",), noise=nz)

-- rank (LoRA on V, n=600, lr 3e-3); the update to V has shape 5 x 64, so rank cannot exceed 5
r=1                                epoch 40  test 0.833 ± 0.011   retain 0.597
r=2                                epoch 36  test 0.835 ± 0.006   retain 0.590
r=4                                epoch 25  test 0.825 ± 0.020   retain 0.578
r=8                                epoch 16  test 0.816 ± 0.014   retain 0.580
-- target modules (r=4)
targets=W                          epoch 19  test 0.712 ± 0.014   retain 0.621
targets=V                          epoch 25  test 0.825 ± 0.020   retain 0.578
targets=W,V                        epoch 22  test 0.755 ± 0.020   retain 0.565
-- training-set size
LoRA r=4 on V, n=100               epoch 39  test 0.649 ± 0.018   retain 0.675
full,            n=100             epoch 40  test 0.577 ± 0.032   retain 0.515
LoRA r=4 on V, n=200               epoch 40  test 0.752 ± 0.028   retain 0.605
full,            n=200             epoch 21  test 0.608 ± 0.028   retain 0.470
LoRA r=4 on V, n=600               epoch 25  test 0.825 ± 0.020   retain 0.578
full,            n=600             epoch  2  test 0.673 ± 0.012   retain 0.502
LoRA r=4 on V, n=2000              epoch 13  test 0.848 ± 0.015   retain 0.567
full,            n=2000            epoch  3  test 0.725 ± 0.014   retain 0.453
-- negative control: labels shuffled across examples
LoRA r=4 on V, shuffled labels     epoch  2  test 0.560 ± 0.004   retain 0.920
-- label noise in the training set
LoRA r=4 on V, noise=0.0           epoch 38  test 0.887 ± 0.006   retain 0.576
LoRA r=4 on V, noise=0.1           epoch 25  test 0.825 ± 0.020   retain 0.578
LoRA r=4 on V, noise=0.3           epoch 12  test 0.682 ± 0.030   retain 0.600

Rank. Test accuracy is 83.3%, 83.5%, 82.5%, and 81.6% for ranks 1, 2, 4, and 8, with seed standard deviations between 0.6 and 2.0 points. The differences are within noise, so the toy offers no evidence that a higher rank helps. The arm for rank 8 also demonstrates a structural limit: the update to \(V\) has shape 5×64, so its rank cannot exceed 5, and rank 8 only adds redundant parameters. Rank can never exceed the smaller dimension of the adapted matrix; in a transformer, the useful range depends on the matrix shapes and the task.

Target modules. Adapting only \(W\) reaches 71.2%, only \(V\) 82.5%, and both 75.5%. Adding the first-layer adapter to the output adapter lowers the target score by 7 points. The intuition that more adaptable parameters cannot hurt does not hold here: under the same budget, the arm that adds the first-layer adapter to the output adapter is the worse one.

Data amount. LoRA on \(V\) scores 64.9%, 75.2%, 82.5%, and 84.8% with 100, 200, 600, and 2,000 examples, and full fine-tuning scores 57.7%, 60.8%, 67.3%, and 72.5%. Both improve with data and the adapter stays ahead at every size, but the gains diminish: going from 200 to 600 examples adds 7.3 points for LoRA and going on to 2,000 adds 2.3. Several arms selected epoch 39 or 40, at the edge of the allowed range (the small-data LoRA arms, rank 1, and the noise-free arm), so those arms may be undertrained and their scores are lower bounds on what the recipe can reach.

Label noise. With clean labels, LoRA on \(V\) reaches 88.7%; with 10% noise, 82.5%; with 30%, 68.2%. The score drops steeply with noise, and the selected epoch moves earlier (38, 25, 12), so the model has to be stopped earlier as the noise grows. Label quality is often the cheapest factor to improve, and this ablation measures how much a cleaning pass would be worth before paying for it.

Negative control. The last arm trains on the same inputs with labels shuffled across examples, so the labels carry no information about the inputs. If the experimental pipeline were leaking the answer, for instance through the evaluation set, the shuffled arm would still score well. It scores 56.0%, indistinguishable from the base model’s 55.4%, and the dev-selected epoch is 2, early in training, where the model has barely moved. A control like this is cheap and detects a class of bugs (split leakage, labels attached to the wrong examples) that no ablation of the real recipe can reveal. Leakage and contamination of the fine-tuning and evaluation data are treated in Designing Fine-Tuning Datasets and Evaluation Splits.

Two cautions about reading ablations. Effects can interact: the right rank depends on the target matrices, and the right epoch depends on the data size, so one-factor-at-a-time tables miss combinations. When an interaction is suspected, run the small grid over the two factors. And each ablation arm adds another comparison to the pile: with many arms and few seeds, some difference will look significant by chance. Fix the list of arms before the run and report all of them.

Overfitting: reading the curves

Overfitting in fine-tuning means that the model’s improvement on the training examples stops transferring to unseen examples of the task. It is diagnosed from the gap between training and held-out measurements as training proceeds, and the signal is usually visible in the loss before it is visible in accuracy. Fine-tuning sets are small compared with pretraining sets, often hundreds to tens of thousands of examples, so a model with millions or billions of parameters can fit each of them. Zhang et al. showed that standard deep networks can fit even randomly labeled training data, so a high training score alone carries no information about generalization. The general diagnosis and the regularization tools are covered in Diagnosing and Preventing Overfitting in Deep Learning.

import numpy as np
from ftlib import *

ev = make_eval_sets(); EPOCHS = 60
RUNS = {"full (lr 3e-2)": ("full", 3e-2, {}), "LoRA r=4 on V (lr 3e-3)": ("lora", 3e-3, {"r": 4, "targets": ("V",)})}
SHOW = (0, 1, 2, 4, 9, 20, 40, 60)

for name, (m, lr, kw) in RUNS.items():
    h, model, (Xtr, ytr, yclean) = finetune(m, lr, EPOCHS, 0, n_train=600, evals=ev, **kw)
    print(name)
    print("  epoch  train-loss  train-acc  dev-loss  dev-acc")
    for e in SHOW:
        r = h[e]; print(f"  {e:5d}  {r['train'][0]:10.3f}  {r['train'][1]:9.3f}  {r['dev'][0]:8.3f}  {r['dev'][1]:7.3f}")
    best_loss = min(range(EPOCHS + 1), key=lambda e: h[e]["dev"][0]); best_acc = max(range(EPOCHS + 1), key=lambda e: h[e]["dev"][1])
    print(f"  dev loss lowest at epoch {best_loss}, dev accuracy highest at epoch {best_acc}")
    p, lora = model; pred = forward(p, Xtr, lora)[0].argmax(1)
    flipped = ytr != yclean
    print(f"  at epoch {EPOCHS}: fits {np.mean(pred[~flipped] == ytr[~flipped]):.3f} of correct labels and {np.mean(pred[flipped] == ytr[flipped]):.3f} of the {flipped.sum()} flipped labels")

print("optimism of picking the best dev checkpoint (5 seeds, best epoch per run)")
for name, (m, lr, kw) in RUNS.items():
    gaps, tests = [], []
    for s in range(5):
        h = finetune(m, lr, 40, s, n_train=600, evals=ev, **kw)[0]
        e = max(range(1, 41), key=lambda e: h[e]["dev"][1])
        gaps.append(h[e]["dev"][1] - h[e]["test"][1]); tests.append(h[e]["test"][1])
    print(f"  {name:24s} best-dev minus test at that checkpoint: {np.mean(gaps):+.3f}  (test {np.mean(tests):.3f})")

full (lr 3e-2)
  epoch  train-loss  train-acc  dev-loss  dev-acc
      0       7.404      0.502     6.426    0.535
      1       1.794      0.700     1.874    0.655
      2       1.072      0.740     1.448    0.670
      4       0.504      0.847     1.132    0.640
      9       0.186      0.953     1.140    0.647
     20       0.026      1.000     1.619    0.642
     40       0.004      1.000     1.993    0.635
     60       0.002      1.000     2.196    0.627
  dev loss lowest at epoch 5, dev accuracy highest at epoch 13
  at epoch 60: fits 1.000 of correct labels and 1.000 of the 62 flipped labels
LoRA r=4 on V (lr 3e-3)
  epoch  train-loss  train-acc  dev-loss  dev-acc
      0       7.404      0.502     6.426    0.535
      1       7.270      0.503     6.302    0.535
      2       6.738      0.517     5.779    0.547
      4       4.429      0.548     3.467    0.570
      9       2.254      0.703     1.212    0.735
     20       1.610      0.768     0.634    0.800
     40       1.093      0.783     0.630    0.765
     60       0.923      0.782     0.633    0.755
  dev loss lowest at epoch 24, dev accuracy highest at epoch 21
  at epoch 60: fits 0.855 of correct labels and 0.145 of the 62 flipped labels
optimism of picking the best dev checkpoint (5 seeds, best epoch per run)
  full (lr 3e-2)           best-dev minus test at that checkpoint: +0.009  (test 0.670)
  LoRA r=4 on V (lr 3e-3)  best-dev minus test at that checkpoint: -0.014  (test 0.829)

The table lists one seed per method, with 600 training examples of which 62 have flipped labels. The base model’s loss on this task is high (6.4 on dev) because it makes confident wrong predictions on the domain’s relabeled classes. Full fine-tuning drops the training loss to 0.026 by epoch 20 and to 0.002 by epoch 60, with training accuracy 1.000: it fits all 62 flipped labels, which are noise by construction. Meanwhile the dev loss reaches its minimum of about 1.13 near epoch 5 and then climbs to 2.20 by epoch 60. Dev accuracy peaks at epoch 13 and later slips from about 65% to 63%. This is the classic pattern: training loss falls to zero while dev loss turns upward. The rise in dev loss is the earlier and clearer symptom, since the model grows more confident on its wrong answers before its accuracy changes.

The LoRA run on \(V\) shows the other pattern. Its training accuracy plateaus near 78% and it fits only 14.5% of the flipped labels at epoch 60, because a rank-4 update of a 5×64 matrix cannot memorize 600 examples. The dev loss has a flat minimum (0.63 from epoch 20 onward) and degrades little afterward. Restricting capacity acts as a regularizer here, which is consistent with, but does not prove, the ranking observed in the first experiment. Note also that the two losses rank the methods differently from accuracy at some epochs, which is why the selection criterion should be stated: the epoch with the lowest dev loss (5 for full, 24 for LoRA) differs from the epoch with the highest dev accuracy (13 and 21).

In an LLM run the same curves come from the training loss on response tokens against the loss and task score on held-out examples, logged every few hundred steps. Two additional memorization checks are cheap. Prompt the tuned model with the beginning of training examples and test whether it reproduces the continuation verbatim at a rate well above that of held-out examples. And check whether outputs on held-out inputs contain strings that appear only in the training set (names, identifiers), which signals both overfitting and a privacy problem.

Choosing the checkpoint without fooling yourself

The remedy for overfitting is to select the stopping point on dev data, which is early stopping, plus lower learning rates, fewer epochs, regularization, more or cleaner data, and lower-capacity adaptation. The selection itself carries a bias. When you pick the best of many candidate checkpoints by a noisy dev score, the winner’s dev score is optimistic (the winner’s curse), because noise that happened to favor it was selected along with its quality. The size of that optimism depends on the dev set size and the number of candidates; Hyperparameter Tuning and Model Selection develops the general case. The next program measures it: it trains LoRA on \(V\) for 40 epochs, picks the best epoch using only the first \(n_\text{dev}\) dev examples, and compares that dev score with the test accuracy at the chosen epoch, over 20 seeds.

import numpy as np
from ftlib import *

ev = make_eval_sets(); Xd, yd = ev["dev"]; Xt, yt = ev["test"]

def epoch_correctness(seed, lr=3e-3, epochs=40, n_train=600):
    """LoRA r=4 on V; after every epoch, record per-example correctness on dev and accuracy on test."""
    rng = np.random.default_rng(1000 + seed)
    p = base_params(); X, yg = sample(n_train, +1, rng); y = noisy(domain_labels(yg), 0.1, rng)
    lora = make_lora(rng, 4, ("V",)); opt = Adam(lr); rows = []
    for _ in range(epochs):
        idx = rng.permutation(n_train)
        for i in range(0, n_train, 32):
            b = idx[i:i + 32]; g = grads(p, X[b], y[b], lora)
            tr = {k: lora[k] for k in g if k in lora}; opt.step(tr, {k: g[k] for k in tr})
        rows.append(((forward(p, Xd, lora)[0].argmax(1) == yd), (forward(p, Xt, lora)[0].argmax(1) == yt).mean()))
    return rows

runs = [epoch_correctness(s) for s in range(20)]
print("dev examples  mean(best-epoch dev accuracy - test accuracy at that epoch), 20 seeds, 40 candidate epochs")
for n_dev in (25, 50, 100, 400):
    gaps = []
    for rows in runs:
        dev_scores = .mean() for c, _ in rows]          # choose the epoch on the first n_dev dev examples
        e = int(np.argmax(dev_scores)); gaps.append(dev_scores[e] - rows[e][1])
    print(f"  {n_dev:4d}   {np.mean(gaps):+.3f}")

dev examples  mean(best-epoch dev accuracy - test accuracy at that epoch), 20 seeds, 40 candidate epochs
    25   +0.100
    50   +0.060
   100   +0.034
   400   -0.016

With 25 dev examples the best-dev score exceeds the test accuracy at that epoch by 10.0 points on average. With 50 it is 6.0, with 100 it is 3.4, and with 400 the gap is −1.6, a value within noise of zero. The consequence is a rule: the dev score of the selected checkpoint is not a performance estimate, and the number that goes in the report is the test score measured once, after every choice (checkpoint, hyperparameters, prompt) is frozen. A dev set of a few dozen examples can choose among a handful of options but cannot support a fine-grained search.

Catastrophic forgetting: measuring what the adapted model lost

Catastrophic forgetting is the loss of previously learned behavior when a network is trained on new data. The phenomenon was studied in sequential task learning, where Kirkpatrick et al. proposed slowing learning on weights that were important for earlier tasks. In LLM fine-tuning it appears as degraded performance on tasks the fine-tuning data did not cover: general knowledge, reasoning, instruction following, formatting, safety behavior, or other languages. Luo et al. studied continual instruction tuning of models from 1B to 7B parameters and observed forgetting across that range, with the larger models in it forgetting more. The cause is straightforward: gradient updates are computed only on the fine-tuning data, so nothing in the objective rewards keeping behavior the data does not exercise.

Forgetting is measured with a retention set, held-out examples of the behaviors to preserve, scored before and after adaptation with the same harness. For an LLM this means a general benchmark or internal evaluation suite, format and schema checks, a safety or refusal suite, and, if relevant, a sample of earlier-task examples. The retention set needs the same split discipline as the target set. Two scores matter: the absolute retention score after adaptation, and the drop relative to the base model. Report both per slice, since an average can hide a collapse in one behavior.

base = base_params()
print("base model: general-mode accuracy", round(loss_acc(base, *ev["retain"])[1], 3))

print("\ngeneral-mode test accuracy (mean of 5 seeds) while fine-tuning on domain-only data")
SHOW = (0, 1, 2, 5, 10, 20, 40)
print("  epoch          " + "".join(f"{e:7d}" for e in SHOW))
for name, (m, lr, kw) in CFG.items():
    hs = [finetune(m, lr, EPOCHS, s, n_train=600, evals=ev, **kw)[0] for s in SEEDS]
    print(f"  {name:15s}" + "".join(f"{np.mean([h[e]['retain'][1] for h in hs]):7.3f}" for e in SHOW))

base model: general-mode accuracy 0.926

general-mode test accuracy (mean of 5 seeds) while fine-tuning on domain-only data
  epoch                0      1      2      5     10     20     40
  head only        0.927  0.679  0.582  0.515  0.477  0.432  0.366
  LoRA r=4 on V    0.927  0.925  0.915  0.738  0.614  0.586  0.555
  LoRA r=4 on W,V  0.927  0.853  0.643  0.595  0.551  0.467  0.434
  full             0.927  0.530  0.502  0.463  0.428  0.416  0.409

Each row averages 5 seeds, with each method at the learning rate it was tuned to for the target task. General-mode accuracy starts at 92.7% for every method at epoch 0 and falls as training proceeds, at different speeds. Full fine-tuning has lost 40 points after one epoch (53.0%) and keeps declining to 40.9%. Head-only training falls to 67.9% after one epoch. LoRA on \(V\) holds at 92.5% through the first epoch and 91.5% through the second, then falls to 73.8% by epoch 5 and 55.5% by epoch 40. The adapter starts as an exact copy of the base model (\(B=0\)) and its learning rate is small, so it leaves the base behavior more slowly, but it does leave it. The speed of departure depends on the learning rate, the method, and the checkpoint, so retention has to be tracked along the training curve and measured at the checkpoint that will be shipped.

Biderman et al. found, for large models trained on code and mathematics, that LoRA maintained the base model’s performance outside the target domain better than full fine-tuning and better than common regularizers such as weight decay and dropout. The toy agrees in direction for the first epochs and the final epoch (LoRA on \(V\) retains more than full fine-tuning at every epoch after the first), but LoRA on \(W,V\) in the same table retains less than LoRA on \(V\), so the adapter’s placement matters as much as the choice of LoRA over full updates.

Intended change and unintended change

Not every change to a behavior is forgetting. Behavior that the fine-tuning data intentionally changes (the domain relabeling here, a new house style or refusal policy in practice) is a target of the adaptation, and its drop on the base model’s old evaluation is expected. The retention set must therefore consist of behaviors where the old answer should survive. Building it requires deciding, per behavior, whether the new model should differ from the base model, which is a design decision and not a measurement.

Replay and how to read its result

The standard mitigation in the toy’s setting is to include examples of the behaviors to preserve in the training mixture, often called replay or rehearsal. In the toy this means general-mode examples with their general labels, mixed with the 600 domain examples. Replay supplies the contrast the domain-only set lacked: it penalizes applying the relabeling in general mode. The experiment varies the number of replay examples and, because the target and retention scores now compete, selects the learning rate and epoch for each row by the mean of dev accuracy on the domain and general dev sets.

print("\nreplay: general-mode examples added to the 600 domain examples")
print("(lr and epoch chosen per row by the mean of domain-dev and general-dev accuracy)")
print(f"  {'method':16s}{'replay':>7s}  lr     epoch  domain test      general test")
for name in ("LoRA r=4 on V", "LoRA r=4 on W,V", "full"):
    m, _, kw = CFG[name]
    for replay in (0, 60, 200, 600):
        best = None
        for lr in (1e-3, 3e-3, 1e-2, 3e-2):
            hs = [finetune(m, lr, EPOCHS, s, n_train=600, replay=replay, evals=ev, **kw)[0] for s in SEEDS]
            for e in range(1, EPOCHS + 1):
                score = np.mean([(h[e]["dev"][1] + h[e]["retain_dev"][1]) / 2 for h in hs])
                if best is None or score > best[0]: best = (score, lr, e, hs)
        _, lr, e, hs = best
        dt = np.array([h[e]["test"][1] for h in hs]); gt = np.array([h[e]["retain"][1] for h in hs])
        print(f"  {name:16s}{replay:7d}  {lr:<6g}{e:5d}  {dt.mean():.3f} ± {dt.std(ddof=1):.3f}   {gt.mean():.3f} ± {gt.std(ddof=1):.3f}")

replay: general-mode examples added to the 600 domain examples
(lr and epoch chosen per row by the mean of domain-dev and general-dev accuracy)
  method           replay  lr     epoch  domain test      general test
  LoRA r=4 on V         0  0.003     2  0.564 ± 0.003   0.915 ± 0.005
  LoRA r=4 on V        60  0.003     2  0.566 ± 0.001   0.911 ± 0.004
  LoRA r=4 on V       200  0.001     4  0.563 ± 0.002   0.918 ± 0.003
  LoRA r=4 on V       600  0.003     1  0.560 ± 0.002   0.922 ± 0.002
  LoRA r=4 on W,V       0  0.001     3  0.562 ± 0.003   0.922 ± 0.002
  LoRA r=4 on W,V      60  0.001     3  0.564 ± 0.002   0.922 ± 0.003
  LoRA r=4 on W,V     200  0.003    25  0.755 ± 0.025   0.736 ± 0.039
  LoRA r=4 on W,V     600  0.003    21  0.751 ± 0.021   0.820 ± 0.015
  full                  0  0.001     1  0.567 ± 0.001   0.922 ± 0.002
  full                 60  0.001     1  0.567 ± 0.002   0.921 ± 0.002
  full                200  0.001     1  0.566 ± 0.003   0.921 ± 0.002
  full                600  0.003    16  0.671 ± 0.005   0.821 ± 0.017

Without enough replay, no method has a checkpoint that beats the starting point on the combined criterion. The rows with LoRA on \(V\) at every replay size, and the rows for full fine-tuning at 0, 60, and 200 replay examples, select a checkpoint after one to four epochs whose scores (domain about 56%, general about 92%) are those of the base model. For LoRA on \(V\) at zero replay this is a trade the criterion refused: the earlier table shows 82.5% on the domain at 57.8% general, whose mean is lower than that of the unchanged base model. The selection rule, not the training, produced these rows, and a team that applied a retention floor would see the same: the adaptation is not worth shipping at those settings.

With 200 replay examples (25% of the mixture), LoRA on \(W,V\) reaches 75.5% on the domain and 73.6% general; with 600 (50%) it reaches 75.1% and 82.0%. Full fine-tuning with 600 replay examples reaches 67.1% and 82.1%. So at an equal replay budget, LoRA on \(W,V\) has the better domain score at the same retention, and the ordering of the methods differs from the one in the domain-only comparison, where LoRA on \(V\) was best by 7 points. LoRA on \(V\) never learns to apply the relabeling selectively. A plausible reason is that its update acts on hidden features computed by a frozen first layer, which do not separate the two modes well enough for a low-rank change to treat them differently, while an adapter on \(W\) lets training build such features. The experiment shows the outcome, not the mechanism; the mechanism would be tested by examining the hidden features or adapting only the mode column of \(W\).

The lesson for experiment design is that the winner depends on the metric. A recipe chosen on the target score alone can be the one that cannot be made safe, and a recipe chosen on a combination can be a different one. Decide the retention requirement before the comparison and treat the target score and the retention score as a pair; a constraint such as “the retention score may drop by at most 2 points” turns the choice into a constrained one and makes failing recipes visible. For LLMs the replay data is typically a sample of general instruction data, earlier task data, or the model’s own outputs on general prompts; other mitigations include lower learning rates, fewer steps, lower-rank adapters, and regularizing toward the base weights, as in the method of Kirkpatrick et al. Their effects must be measured on the retention set, and the pretrained-to-adapted transfer setting in general is covered in Data Mismatch, Transfer Learning, and Multitask Learning.

Failure analysis: from a score to a cause

An aggregate score says how much is wrong and nothing about why. Failure analysis breaks the errors down until the groups point at different fixes. Three cuts are useful for any fine-tuning experiment. First, compare per-example outcomes of the base and adapted models, which sorts the examples into both right, gained, lost, and both wrong; the lost examples are the cost of the adaptation, and they are invisible in the totals. Second, slice the results by attributes of the input (category, length, source, language) and by the correct answer. Third, read the errors themselves, because some of them turn out to be wrong labels or ambiguous cases, not model failures.

import numpy as np
from ftlib import *

ev = make_eval_sets(); Xt, yt = ev["test"]; Xg, yg = ev["retain"]
def preds(model, X): p, lora = model; return forward(p, X, lora)[0].argmax(1)

base = (base_params(), None)
models = {"LoRA r=4 on V": finetune("lora", 3e-3, 25, 0, n_train=600, evals=ev, r=4, targets=("V",))[1],
          "full": finetune("full", 3e-2, 2, 0, n_train=600, evals=ev)[1]}

for name, model in models.items():
    b_ok, a_ok = preds(base, Xt) == yt, preds(model, Xt) == yt
    print(f"== {name}: domain-mode test (n={len(yt)})")
    print(f"  both right {np.sum(b_ok & a_ok):5d}   gained {np.sum(~b_ok & a_ok):5d}   lost {np.sum(b_ok & ~a_ok):5d}   both wrong {np.sum(~b_ok & ~a_ok):5d}")
    print("  accuracy by correct domain label:   " + "  ".join(f"{c}: {b_ok[yt == c].mean():.2f}->{a_ok[yt == c].mean():.2f}" for c in range(K)))
    g_pred, b_pred = preds(model, Xg), preds(base, Xg)
    wrong = g_pred != yg
    leaked = wrong & (g_pred == PERM[yg]) & (PERM[yg] != yg)
    print(f"  general-mode errors: {wrong.sum()} of {len(yg)}; of these, {leaked.sum()} give the domain's answer instead of the general one ({leaked.sum() / wrong.sum():.2f})")
    print("  general-mode accuracy by class:      " + "  ".join(f"{c}: {np.mean(b_pred[yg == c] == c):.2f}->{np.mean(g_pred[yg == c] == c):.2f}" for c in range(K)))

== LoRA r=4 on V: domain-mode test (n=2000)
  both right   997   gained   641   lost   112   both wrong   250
  accuracy by correct domain label:   0: 0.03->0.76  1: 0.01->0.79  2: 0.90->0.87  3: 0.94->0.84  4: 0.90->0.83
  general-mode errors: 870 of 2000; of these, 584 give the domain's answer instead of the general one (0.67)
  general-mode accuracy by class:      0: 0.94->0.14  1: 0.94->0.04  2: 0.90->0.91  3: 0.94->0.85  4: 0.91->0.80
== full: domain-mode test (n=2000)
  both right   800   gained   531   lost   309   both wrong   360
  accuracy by correct domain label:   0: 0.03->0.59  1: 0.01->0.65  2: 0.90->0.86  3: 0.94->0.50  4: 0.90->0.71
  general-mode errors: 1026 of 2000; of these, 456 give the domain's answer instead of the general one (0.44)
  general-mode accuracy by class:      0: 0.94->0.14  1: 0.94->0.20  2: 0.90->0.85  3: 0.94->0.40  4: 0.91->0.78

LoRA on \(V\). Of the 2,000 domain-mode test examples, 997 are right under both models, 641 are gained, 250 are wrong under both, and 112 are lost. Per class, the two swapped classes rise from 3% and 1% to 76% and 79%. The unchanged classes 2, 3, and 4 drop from 90%, 94%, and 90% to 87%, 84%, and 83%: the adaptation took 3 to 10 points from classes it was not supposed to touch. In general mode, 870 of 2,000 examples are wrong, and 584 of those (67%) are exactly the domain’s answer in place of the general one. The per-class view shows where: classes 0 and 1 fall from 94% to 14% and 4%, and classes 2 to 4 hold at 91%, 85%, and 80%. Most of the failure is one mechanism, the relabeling leaking into general mode, and a smaller part is collateral damage on classes 3 and 4.

Full fine-tuning fails differently. It gains fewer examples (531) and loses more (309). Class 3 is not part of the domain rule yet falls from 94% to 50% in domain mode and to 40% in general mode. Only 44% of its general-mode errors are the domain’s answer. This model is damaged beyond the behavior the data targets, a signature of a large, unconstrained update, whereas the LoRA model’s errors are concentrated in the one behavior the data left unconstrained.

The two signatures call for different next experiments. For the LoRA model, hyperparameters are not the issue; the training data lacks contrast examples, and the replay experiment above is the right next step. For the full-fine-tuning model, the update is too destructive, so lower learning rates, fewer steps, or a lower-capacity adapter come first. A single aggregate retention number would have called both models similarly forgetful (49% and 56% general-mode accuracy on these seed-0 models).

For an LLM the corresponding inspection is on generations. Sample lost examples (base correct, adapted wrong) and gained examples, and read them in pairs. Look at output length and format, refusal wording, repetition, and whether the failure follows a pattern a prompt could reveal. Label a few dozen errors by cause and count the labels; the counts, not the loudest anecdote, rank the fixes. The table maps common symptoms to likely causes and to the experiment that tests each.

Observation	Likely cause	Next experiment
Training loss falls, dev loss rises	Memorization of the training set, noisy labels	Earlier checkpoint, fewer epochs, lower rank, label audit; ablate epochs and noise
Target score barely moves and training loss stays high	Learning rate or capacity too low, inconsistent labels, wrong loss mask or template	Decode a few formatted training examples and check the masked tokens; widen the learning-rate grid; add target modules
Target up, behavior outside the data down, errors follow the new rule	The data never shows when not to apply the new behavior	Add contrast or replay examples; add the behavior to the retention set
Target up, unrelated behaviors down	Update too large or too long	Lower learning rate, fewer steps, lower-rank adapter, regularize toward base weights
Gain concentrated in one slice	Training data covers that slice only	Slice the training data like the evaluation data; add data for weak slices
Many “errors” look correct on reading	Label errors or ambiguous items in the evaluation set	Re-label a sample; report a score on the audited subset
Gain disappears with another seed or prompt wording	The difference is within noise or depends on the prompt	More seeds; evaluate with several prompt templates
Shuffled-label control scores well above base	Leakage between training and evaluation data	Audit splits and duplicates before any other run

The pattern of the exercise is the point more than the specific fixes: every row connects an observation that can be computed from the saved per-example outputs to an experiment that changes one thing. Save the per-example predictions of every run, since each of the analyses above needs them and recomputing them means retraining.

Turning runs into evidence

A results table is evidence for a claim only if the claim, the comparison, and the uncertainty are stated together. A claim that fits the toy’s evidence reads: “On the domain test set (2,000 examples), LoRA of rank 4 on the output matrix, tuned on dev, scores 82.5% ± 2.0 (5 seeds) against 55.4% for the base model and 73.7% ± 1.3 for head-only training; its paired difference from head-only is 10.4 points with a 95% bootstrap interval from 8.4 to 12.3 for one seed, and positive for all five seeds matched. On general-mode examples, accuracy falls from 92.6% to 57.8%.” Each clause can be traced to a saved artifact, and the last sentence is the part most often left out.

Claim	Evidence that supports it	Evidence that would refute it
The adapted model is better	Paired difference on the held-out test set, interval and seed-matched differences agree in sign	Interval includes zero; sign flips with seed
Fine-tuning beats the cheaper alternative	Gain over the tuned-prompt and simple-adaptation rungs with equal tuning effort	A tuned prompt matches it on the same harness
Component X matters	Ablation of X with other factors fixed and each arm tuned, difference beyond noise	Difference within seed spread, or reverses after retuning
The pipeline is sound	Negative control (shuffled labels) scores at base level; no overlap between train and test	Shuffled arm scores well; duplicates across splits
The model is safe to ship	Retention suite within the agreed floor, per slice; failure analysis shows no unexplained cluster	Any slice below the floor; a cluster of lost examples with a common cause

The run record should hold, for every row, the data version and split identifiers, template and decoding settings, base checkpoint, method and hyperparameters, seed, selected checkpoint and the rule that selected it, and the per-example predictions. With these, any number in the table can be recomputed and any comparison can be repeated with a new baseline. Negative and null results belong in the record too: a failed arm or an ablation that showed no effect is the reason the recipe has its current form, and without it the next person repeats the experiment.

References

• Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
• Houlsby et al. (2019). Parameter-Efficient Transfer Learning for NLP. ICML.
• Biderman et al. (2024). LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research.
• Dodge et al. (2020). Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. arXiv preprint.
• Zhang, Bengio, Hardt, Recht, and Vinyals (2017). Understanding Deep Learning Requires Rethinking Generalization. ICLR.
• Kirkpatrick et al. (2017). Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences.
• Luo, Yang, Meng, Li, Zhou, and Zhang (2023). An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning. arXiv preprint.