Early Stopping

Early stopping trains for as long as it helps and no longer. After each pass over the data, a monitored metric is computed on a validation set; when it has not improved for a set number of passes — the patience — training halts. Halting and keeping the best model are separate steps: the weights from the best epoch are available only if the procedure saves a checkpoint of them during training and restores it afterward. Without that, the model you hold is the one from the last epoch, patience epochs past the best point.

Early stopping can regularize a model and save training compute when later epochs fit the training data while validation performance deteriorates. It still requires validation data and monitoring, and its benefit depends on the learning curve.

What it buys, and what its imprecision costs

This illustration runs all 400 epochs and compares the full curves afterward. It chooses the validation minimum retrospectively, rather than executing a prespecified patience rule or restoring a saved model during training.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import log_loss
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=60, n_informative=6, flip_y=0.15, random_state=0)
xtr, rest, ytr, rest_y = train_test_split(X, y, train_size=200, random_state=0)
xval, xte, yval, yte = train_test_split(rest, rest_y, train_size=300, random_state=0)
"""200 rows to fit, 300 to decide when to stop, 5,500 held back. A noisy problem
   with 60 features and only 200 training rows, where continued fitting can overfit."""

model = SGDClassifier(loss="log_loss", learning_rate="constant", eta0=0.0002,
                      penalty=None, random_state=0)
history = []
for _ in range(400):
    model.partial_fit(xtr, ytr, classes=[0, 1])
    history.append([log_loss(a, model.predict_proba(b)[:, 1], labels=[0, 1])
                    for b, a in ((xtr, ytr), (xval, yval), (xte, yte))])
history = np.array(history)

chosen = int(history[:, 1].argmin())
ideal = int(history[:, 2].argmin())
print(f"{'epoch':>36}  {'train':>7} {'validation':>11} {'test':>7}")
for label, e in ((f"chosen by the validation set", chosen), (f"the best epoch for the test set", ideal),
                 (f"no stopping at all", len(history) - 1)):
    print(f"{label + f' ({e + 1})':>36}  {history[e, 0]:>7.4f} {history[e, 1]:>11.4f} {history[e, 2]:>7.4f}")
print(f"\n  stopping where the validation set says costs"
      f" {history[chosen, 2] - history[ideal, 2]:+.4f} of test loss against the best available,")
print(f"  and running to the end costs {history[-1, 2] - history[ideal, 2]:+.4f}")
#                                epoch    train  validation    test
#    chosen by the validation set (41)   0.4757      0.5625  0.5318
# the best epoch for the test set (50)   0.4640      0.5633  0.5308
#             no stopping at all (400)   0.3676      0.6668  0.6273
#
#   stopping where the validation set says costs +0.0010 of test loss against the best available,
#   and running to the end costs +0.0965

Two numbers at the bottom set the scale of the problem.

Not stopping costs 0.0965 of log loss. By epoch 400 the training loss has fallen to 0.3676 while the test loss has risen to 0.6273 — the model has spent three hundred and fifty epochs memorizing a two-hundred-row training set. This is what early stopping exists to prevent, and the saving is large.

Stopping at the wrong epoch costs 0.0010. The validation set nominates epoch 41; epoch 50 would have been marginally better. That is a hundredth of the other error, on a validation set of only 300 rows. The curve is flat near its minimum, so approximate is good enough.

Which is the useful conclusion: do not agonize over the precision of the stopping rule, and do not skip it.

The stopping epoch is a hyperparameter

This is the part that gets skipped. Choosing the number of epochs from a validation curve is hyperparameter selection, performed on whichever data the curve was computed from. Everything that follows from that applies here.

• Never early-stop on the test set. If the test set chose the epoch, the test score is a selection score and is optimistic. It is a quiet form of leakage because no data was copied anywhere — a single number crossed the boundary, four hundred times.
• The validation score at the chosen epoch is also optimistic, for the same reason: it is the minimum of four hundred noisy estimates. Report the test loss at that epoch, not the validation loss that selected it.
• Keep the stopping data separate from each evaluation fold. Splitting early-stopping validation from the training portion of every outer fold is a common design. A dedicated validation set reserved before cross-validation can also avoid test leakage if those rows are excluded from every later training and test fold. It uses data differently, and repeated adaptation to it still needs to be considered.

scikit-learn’s estimators handle this by taking the early-stopping validation set out of the training data with validation_fraction, which keeps the boundary intact by construction.

Setting the parameters

• Patience is itself a choice. Too short and a single noisy epoch ends training; too long and you pay for epochs you will discard. Scale it to how noisy the curve is, which means looking at the curve.
• Monitor a loss, not accuracy. Accuracy moves in steps as predictions cross the threshold, so it plateaus and jumps; log loss responds to every change in confidence and gives a smoother signal.
• Checkpoint the best weights and restore them. Stopping alone leaves you with the weights from patience epochs past the best point, which is the wrong end of the curve. Keep the best-so-far weights during training and evaluate or deploy those, as the retrospective choice of the validation minimum does above.
• Remember it interacts with the other regularizers. A larger weight decay or a smaller learning rate moves the stopping point, so tuning them and the patience independently finds a worse combination than tuning them together.

And the validation set needs to be large enough that its curve is not mostly noise. Three hundred rows was enough here because the minimum is flat; on a problem with a sharper optimum, a small validation set will stop somewhere arbitrary and report a confident number about it.

References

• scikit-learn API reference, SGDClassifier
• scikit-learn, Common pitfalls and recommended practices