Epochs

An epoch is one complete pass of the training set through the model. It is a unit of exposure, not of progress: it says how many times the data has been shown, and nothing about whether showing it again helps.

For a minibatch optimizer, a batch is the group of examples used for one gradient update, and a step usually means one such update. With gradient accumulation, several microbatches contribute to an update. Sample-wise SGD instead updates after each example. Count optimizer updates from the implementation, not simply from the number of data-loading calls.

Counting passes and updates

With 10,000 training examples and batches of 100, one pass consists of 100 batches. If the optimizer updates once per batch, ten epochs mean 1,000 updates and 100,000 example presentations. Those are repeated presentations of 10,000 examples, not 100,000 distinct training observations. Shuffling changes their order; augmentation can change the view presented on each visit. Neither turns the epoch count into a direct measure of information learned.

The last batch matters when the dataset size is not divisible by the batch size. For 1,050 examples and a batch size of 100, retaining the partial batch gives 11 batches, the last with 50 examples. Dropping it gives 10 batches and leaves 50 examples unused in that pass. Which examples are omitted depends on the sampling order. State this policy when a reported epoch is not literally a visit to every example.

Accumulation and replication change the update count. Suppose four data-parallel workers each process microbatches of 25 examples and accumulate four microbatches before a synchronized update. The effective global batch is 4 × 25 × 4 = 400 examples, assuming disjoint samples and consistent gradient averaging. A 10,000-example global pass then takes 25 optimizer updates. It is still one epoch across the global dataset, not four epochs because four workers participated. Padding, repeated samples, and partial accumulation at the end can change this arithmetic in an implementation.

Why another pass can help or hurt

The model sees the same example under updated parameters on a later pass, so its loss and gradient generally differ from the previous visit. Repetition lets optimization continue without requiring a new dataset. Training can reduce useful errors, but it can also fit noise or idiosyncrasies that do not generalize. Neither training nor validation loss must decrease smoothly; stochastic updates and evaluation noise can produce fluctuations.

Consider hypothetical checkpoint losses: training loss falls from 0.6 to 0.3 to 0.1, while validation loss changes from 0.65 to 0.50 to 0.58. The last checkpoint fits training observations better but performs worse on this validation sample. That is evidence for selecting an earlier checkpoint under the chosen metric, not proof that every later epoch must worsen every unseen population. Early stopping makes that selection using a declared metric, improvement rule, and patience. Keep final evaluation separate because choosing an epoch already uses the validation data.

How learning rate and epoch count interact

Learning rate and epoch count are interacting hyperparameters, not one quantity. For gradient descent on a quadratic, the final error depends on a power of the update factor, not only on learning rate times step count. Larger steps can follow a different path or become unstable, and stochastic gradients, regularization and learning-rate schedules add further differences. Hyperparameter tuning should therefore consider their interaction rather than treating either as redundant.

A generous epoch ceiling with early stopping can avoid spending the entire budget on each candidate. Set the validation metric, patience and checkpoint-restoration rule explicitly, and compare candidates under a suitable tuning budget. The ceiling remains relevant when training reaches it; reducing a nonbinding ceiling changes neither the selected checkpoint nor the cost.

Where the unit stops being useful

There is no dataset-size rule that determines the right epoch count. Optimizer, learning-rate schedule, batch size, regularization, model capacity, and validation objective all affect how long useful training lasts. Equal epoch counts across two datasets need not mean equal compute; equal update counts need not mean equal exposure. Report the quantities needed for the comparison being made.

Large language-model training is often budgeted in tokens rather than complete passes through a fixed dataset. GPT-3 trained on 300 billion tokens with non-proportional sampling across sources: Table 2.2 reports about 2.9 effective epochs for WebText2, 1.9 for Books1 and 3.4 for Wikipedia. A token budget therefore does not imply that training data is never repeated.

For a continuing stream, sampling with replacement, or a mixture whose sources are oversampled unequally, a literal full pass may not exist. A training system may define an epoch as a fixed number of steps or draws; name that convention explicitly. Learning-rate schedules tied to epochs also change their update timing when batch size changes. Estimate FLOPs from the work actually executed, not a nominal ceiling. Halving an epoch ceiling halves training work only when it halves the executed passes and per-pass work stays fixed.

References

• scikit-learn documentation, sklearn.linear_model.SGDClassifier (partial_fit, learning_rate, eta0). scikit-learn.org
• scikit-learn documentation, sklearn.metrics.log_loss. scikit-learn.org
• Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165, Section 2.2. arxiv.org