Reproducible Training Pipelines: Seeds, Configs, and Checkpoints

A checkpoint can preserve the weights while losing the conditions that produced them: code changes, data revisions, augmentation, and optimizer history. This article separates three goals: repeat a run in a controlled environment, continue it after interruption, and check whether an improvement persists across training seeds. Each requires different evidence. Run the Python body blocks in order; the resume demonstration uses CPU training and checkpoints at epoch boundaries.

Where randomness enters

Initialization, shuffling, augmentation, and dropout can consume randomness. They are not necessarily independent: several operations may advance the same generator, so inserting an extra random draw can change later results. Python, NumPy, and PyTorch maintain separate random state, and explicitly constructed generators maintain their own state too. A seed chooses a starting state; saving the current state lets a run continue from its current position.

import os, random
import numpy as np
import torch

def set_seed(seed: int, deterministic: bool = False):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = deterministic
    torch.use_deterministic_algorithms(deterministic, warn_only=False)
    return np.random.default_rng(seed)

def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2 ** 32
    np.random.seed(worker_seed)
    random.seed(worker_seed)

np_rng = set_seed(0, deterministic=True)
# For worker processes: DataLoader(dataset, worker_init_fn=seed_worker, generator=g).
g = torch.Generator().manual_seed(0)

The helper seeds Python, legacy NumPy calls, and PyTorch, and returns a separately seeded NumPy Generator. It also explicitly sets the deterministic flag on or off, avoiding a previous call silently leaving it enabled. Strict mode raises when a supported operation is known to lack a deterministic implementation; warn_only=True would allow it to continue. Runtime cost depends on the operations and hardware, so there is no fixed 10–20% penalty or universal recommendation to turn it off.

Matching seeds alone does not guarantee matching outputs or a representative result. Record the software, hardware, algorithms, data order, and precision settings. Exact equality across PyTorch releases or devices is not guaranteed; that is different from saying it can never occur. CUDA operations may require additional settings such as CUBLAS_WORKSPACE_CONFIG before CUDA initialization. Set PYTHONHASHSEED in the launching environment before Python starts; assigning it inside a running script does not change that interpreter’s existing hash seed.

The worker initializer seeds libraries from the worker’s PyTorch seed. Pass it together with a seeded generator when using worker processes. With num_workers=0, loading runs in the main process and the worker initializer is not called. Explicit NumPy Generator objects used by transforms need their own policy; seeding the legacy NumPy generator does not reset them. Persistent workers also keep state across epochs, so recreating them on resume can change augmentation draws.

The training loop this code slots into is built in Building a PyTorch Training Loop with nn.Module, Dataset, and DataLoader.

Seed variance is a measurement, not a nuisance

A fixed seed helps reproduce a trajectory under controlled conditions; it does not make that trajectory representative. A 0.3-percentage-point mean difference is not automatically indistinguishable just because individual-run standard deviation is 0.5 points. Precision depends on the number of runs and on the variability of the paired differences. Compare candidates on the same data split and a planned set of seeds, report individual scores plus a mean and spread, and add repetitions when the uncertainty matters. Three to five seeds can be a starting check, not a universal threshold. Repeated seeds on one split measure training variability, not uncertainty from drawing another dataset.

Configuration as data

Literals in versioned source can be recorded and compared, but a resolved configuration makes a run easier to inspect and repeat. Save the values actually used after defaults and command-line overrides are applied. Include architecture, loss, optimizer, schedule, augmentation, and data/split identifiers. The small configuration below exposes the CPU example’s editable settings. Other choices, including its eight-unit hidden layer, squared-error loss, and scheduler decay, are fixed in the accompanying code and must travel with it.

from dataclasses import dataclass, asdict
import json, subprocess, platform, sys

@dataclass
class Config:
    seed: int = 0
    lr: float = 0.01
    batch_size: int = 4
    epochs: int = 4
    weight_decay: float = 0.01
    dropout: float = 0.2
    in_dim: int = 4
    model: str = "linear_dropout_linear"
    dataset: str = "synthetic_v1_seed123"

def git_info():
    try:
        commit = subprocess.check_output(["git", "rev-parse", "HEAD"],
                                         text=True, stderr=subprocess.DEVNULL).strip()
        status = subprocess.check_output(["git", "status", "--porcelain"],
                                         text=True, stderr=subprocess.DEVNULL)
        return {"commit": commit, "dirty": bool(status.strip())}
    except (OSError, subprocess.CalledProcessError):
        return {"commit": None, "dirty": None}

def run_metadata(cfg, device):
    return {"config": asdict(cfg), "git": git_info(),
            "python": platform.python_version(), "numpy": np.__version__,
            "torch": str(torch.__version__), "cuda_build": torch.version.cuda,
            "platform": platform.platform(), "device": str(device),
            "deterministic": torch.are_deterministic_algorithms_enabled()}

cfg = Config()
metadata = run_metadata(cfg, torch.device("cpu"))
print("config round-trip", json.loads(json.dumps(metadata))["config"] == asdict(cfg))
# config round-trip True

A commit identifies committed code, not local modifications. Record the dirty status and preserve a source snapshot or a patch plus any required untracked files. A dirty working tree can be reproduced when its contents are archived; a dirty flag alone is insufficient. Outside a Git repository, the metadata reports missing version information explicitly. Also preserve dependency versions, the launch command, and the dataset manifest; the helper collects a useful subset rather than a complete environment archive.

Checkpoints that can actually resume

A model state dict supplies weights and buffers, but inference also needs the matching architecture and input processing. To continue training along the same trajectory, restore optimizer and scheduler state, progress counters, and every random stream used by the run. A seed reset would restart the random sequence rather than continue it. The functions below support CPU training with no AMP, one explicit loader generator, one NumPy Generator, and checkpoints after a complete epoch with no pending accumulated gradients.

from pathlib import Path
import tempfile

def require_cpu(model):
    if any(t.device.type != "cpu" for t in list(model.parameters()) + list(model.buffers())):
        raise ValueError("this checkpoint example supports CPU models only")

def save_checkpoint(path, model, optimizer, scheduler, next_epoch, cfg, metrics, g, np_rng):
    require_cpu(model)
    state = {"model": model.state_dict(), "optimizer": optimizer.state_dict(),
             "scheduler": scheduler.state_dict(), "next_epoch": next_epoch,
             "config": asdict(cfg), "metrics": metrics,
             "rng": {"python": random.getstate(), "numpy": np.random.get_state(),
                     "torch": torch.get_rng_state(), "loader": g.get_state(),
                     "numpy_generator": np_rng.bit_generator.state}}
    path = Path(path)
    path.parent.mkdir(parents=True, exist_ok=True)
    with tempfile.NamedTemporaryFile(dir=path.parent, delete=False) as f:
        temporary = Path(f.name)
    try:
        torch.save(state, temporary)
        os.replace(temporary, path)
    finally:
        temporary.unlink(missing_ok=True)

def load_checkpoint(path, model, optimizer, scheduler, g, np_rng):
    require_cpu(model)
    # Load only a checkpoint you trust: Python/NumPy RNG state needs this mode.
    ckpt = torch.load(path, map_location="cpu", weights_only=False)
    model.load_state_dict(ckpt["model"])
    optimizer.load_state_dict(ckpt["optimizer"])
    scheduler.load_state_dict(ckpt["scheduler"])
    random.setstate(ckpt["rng"]["python"])
    np.random.set_state(ckpt["rng"]["numpy"])
    torch.set_rng_state(ckpt["rng"]["torch"])
    g.set_state(ckpt["rng"]["loader"])
    np_rng.bit_generator.state = ckpt["rng"]["numpy_generator"]
    return ckpt

The saved next_epoch is the next epoch to execute. Construct the model, optimizer, and scheduler before loading, then restore random state last and do not reseed afterward. Restore the loader generator before creating its next iterator. CUDA runs additionally need relevant device RNG states and possibly an AMP scaler; distributed runs need per-rank and sampler state. Mid-epoch resumption may require the current permutation, cursor, pending gradients, and worker/prefetch state. This CPU example does not implement those cases.

The save function writes a temporary file in the destination directory and replaces the destination only after serialization finishes. If writing fails before replacement, the previous checkpoint remains available. This reduces partial-file replacement risk; it is not a guarantee of durability under power loss or every filesystem. Keep a recent recovery checkpoint and a validation-selected checkpoint; retain extra snapshots when debugging, auditing, or later analysis warrants the storage.

Check the continuation, not just the load call

The following small regression model uses dropout, shuffled batches, and noise from Python and both NumPy random APIs. One run trains for four epochs. A second stops after two epochs, saves, constructs fresh training objects, restores the checkpoint, and finishes. We compare the remaining batch losses and final parameters with exact equality on this CPU run. A weights-only restart is included to show what changes when optimizer, schedule, and random state are not restored.

from torch.utils.data import TensorDataset, DataLoader
from tempfile import TemporaryDirectory

def setup(cfg):
    np_rng = set_seed(cfg.seed, deterministic=True)
    g = torch.Generator().manual_seed(cfg.seed + 1)
    data_rng = torch.Generator().manual_seed(123)
    X = torch.randn(16, cfg.in_dim, generator=data_rng)
    y = 0.5 * X.sum(dim=1, keepdim=True)
    loader = DataLoader(TensorDataset(X, y), batch_size=cfg.batch_size,
                        shuffle=True, generator=g, num_workers=0)
    model = torch.nn.Sequential(torch.nn.Linear(cfg.in_dim, 8),
                                torch.nn.Dropout(cfg.dropout), torch.nn.Linear(8, 1))
    optimizer = torch.optim.AdamW(model.parameters(), lr=cfg.lr,
                                   weight_decay=cfg.weight_decay)
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.8)
    return model, optimizer, scheduler, loader, g, np_rng

def train_epochs(bundle, start, stop):
    model, optimizer, scheduler, loader, g, np_rng = bundle
    losses = []
    model.train()
    for epoch in range(start, stop):
        for xb, yb in loader:
            noise = 0.01 * (random.random() + np.random.rand() + np_rng.random())
            optimizer.zero_grad(set_to_none=True)
            loss = ((model(xb + noise) - yb) ** 2).mean()
            losses.append(loss.item())
            loss.backward()
            optimizer.step()
        scheduler.step()
    return losses

cfg = Config()
full = setup(cfg)
full_losses = train_epochs(full, 0, cfg.epochs)
partial = setup(cfg)
train_epochs(partial, 0, 2)
with TemporaryDirectory() as directory:
    path = Path(directory) / "latest.pt"
    save_checkpoint(path, partial[0], partial[1], partial[2], 2, cfg, {}, partial[4], partial[5])
    resumed = setup(cfg)
    ckpt = load_checkpoint(path, resumed[0], resumed[1], resumed[2], resumed[4], resumed[5])
    assert ckpt["config"] == asdict(cfg)
    resumed_losses = train_epochs(resumed, ckpt["next_epoch"], cfg.epochs)
    offset = 2 * len(full[3])
    print("continuation losses equal:", full_losses[offset:] == resumed_losses)
    print("final parameters equal:", all(torch.equal(a, b) for a, b in
          zip(full[0].parameters(), resumed[0].parameters())))
    weights_only = setup(cfg)
    weights_only[0].load_state_dict(ckpt["model"])
    other_losses = train_epochs(weights_only, 2, cfg.epochs)
    print("weights-only losses equal:", full_losses[offset:] == other_losses)
# continuation losses equal: True
# final parameters equal: True
# weights-only losses equal: False

Matching continuation losses and parameters checks more than successful deserialization. The weights-only path starts with the same saved parameters but follows a different subsequent trajectory in this example. It need not have a larger first loss or larger updates in every problem. Exact equality here is a result for this controlled test; repeat the continuation test on the actual training pipeline before relying on its resume support.

Data versioning

A data revision can change what a comparison measures. Keep immutable dataset versions or manifests linking example IDs to content checksums and labels, and preserve the split assignments. A file-list hash detects changes in membership but misses edited contents and labels under unchanged names. Counts per class and preprocessing versions help explain changes, but do not replace the data itself.

A hash of a stable identifier can keep an existing example in the same split when new examples are added. Keep the hash rule, thresholds, and ID meaning fixed. Hash a patient, user, or other group ID when related rows must stay together; for future prediction, chronological splits may be more appropriate. Hashing individual rows does not prevent duplicate or group leakage, guarantee class balance, or produce exact split sizes. A saved shuffle-based split is also stable if you preserve existing assignments rather than recomputing them.

import hashlib

def split_of(example_id: str, val_frac=0.1, test_frac=0.1):
    if not (np.isfinite(val_frac) and np.isfinite(test_frac)
            and 0 <= val_frac <= 1 and 0 <= test_frac <= 1
            and val_frac + test_frac <= 1):
        raise ValueError("split fractions must be nonnegative and sum to at most 1")
    h = int.from_bytes(hashlib.sha256(example_id.encode()).digest()[:8], "big")
    scale = 2 ** 64
    if h < int(test_frac * scale):
        return "test"
    if h < int((test_frac + val_frac) * scale):
        return "val"
    return "train"

print(split_of("img_00042.jpg"), split_of("img_00043.jpg"))
# train val

Experiment tracking

Give each run a unique ID and link its resolved configuration, code snapshot, data manifest, metrics, resource use, and checkpoints. Runs sharing the same hyperparameters can use different seeds or environments. Even an exact repeat of a resolved configuration needs its own run ID. A hosted tracker and a directory of structured files can both work if artifacts remain connected and retrievable.

Try recovering a run from its saved record: can you locate the code, dependencies, data, preprocessing, command, and checkpoint? Then rerun an evaluation or a short continuation and compare the outputs. Recovering a command is useful, but it cannot compensate for missing data or changed dependencies.

How to allocate a tuning budget across these choices is the subject of A Hyperparameter Tuning Strategy That Fits Your Budget.

Before relying on a saved run

• Record the intended reproducibility goal and seed every random stream the pipeline uses.
• Archive the resolved configuration, code changes, environment, data version, and split assignments.
• Save and restore the state needed at the chosen checkpoint boundary, then compare a resumed continuation.
• Compare candidate models across planned repetitions and report uncertainty at the level actually tested.

Compressing and serving the finished model is covered in Model Compression and Deployment: Distillation, Pruning, and Serving.

Exercises

1. Which stream was reset? Initialize all three global generators, then draw twice while resetting only PyTorch inside each draw. Compare the components. Explain why the PyTorch values repeat while the Python and legacy NumPy sequences advance.

The two calls share initial setup, but only one stream is reset between them.

import random, numpy as np, torch
random.seed(0)
np.random.seed(0)
def draw():
    torch.manual_seed(0)
    return random.random(), float(np.random.rand()), torch.rand(1).item()
a, b = draw(), draw()
print([round(x, 6) for x in a])
print([round(x, 6) for x in b])
print("same values:", [x == y for x, y in zip(a, b)])
# [0.844422, 0.548814, 0.496257]
# [0.757954, 0.715189, 0.496257]
# same values: [False, False, True]

2. What a checkpoint must contain. Use the body’s continuation experiment. Compare the uninterrupted, full-resume, and weights-only paths. Identify which saved state controls parameter updates, batch order, and augmentation, and explain why a weights-only restart does not necessarily cause an immediate loss jump.

The complete resume matches this CPU trajectory; the weights-only path diverges later in the comparison.

3. Stable split assignments. Hash 1,000 IDs, add 500 new IDs, and count how many old assignments change. Compare with recomputing a shuffled split using the same seed and new list length. Explain how preserving an existing split manifest differs from either recomputation.

The unchanged hash rule preserves old assignments. Measure how many change under the specific reshuffle below.

import numpy as np
ids = [f"id{i:05d}" for i in range(1000)]
ids2 = ids + [f"id{i:05d}" for i in range(1000, 1500)]
before = {i: split_of(i) for i in ids}
after = {i: split_of(i) for i in ids2}
print("hash moved:", sum(before[i] != after[i] for i in ids))

def reshuffle(items):
    order = np.random.default_rng(0).permutation(len(items))
    n = len(items)
    return {items[index]: ("test" if rank < n // 10 else
                           "val" if rank < n // 5 else "train")
            for rank, index in enumerate(order)}

before_random, after_random = reshuffle(ids), reshuffle(ids2)
print("reshuffle moved:", sum(before_random[i] != after_random[i] for i in ids))
# hash moved: 0
# reshuffle moved: 321

Implementation references: PyTorch reproducibility notes, DataLoader worker randomness, and Python’s PYTHONHASHSEED setting.