Early stopping

A neural network learns in epochs: one epoch is one pass over all the training examples. After each epoch the training loss usually falls. The question from lesson 3.1 comes back in a new form: how many epochs are too many? Early stopping answers it with the validation set: training stops when the validation loss stops falling, and the model goes back to its best epoch.

Try it

• The lab opens on clean labels. Switch to the random labels and keep a patience of 5. In which epoch does training stop, which weights are kept, and is that the best epoch of the whole run?
• Raise the patience to 10. What changes in the kept epoch and in the test accuracy?
• Go back to clean labels. What does training to the end cost now?

Once you have checked it in the lab, let's name what you see.

The idea in plain words

At the start of training the network learns what many images share: the shape of a zero, an eight, a seven. The training and validation losses fall together. Then the network starts fitting individual examples, including the ones with a wrong label. The training loss keeps falling, while the validation loss starts to rise. That is the overfitting of lesson 3.1, only along the axis of epochs instead of the number of parameters.

Early stopping watches the validation loss. It remembers the epoch with the lowest loss and counts the epochs without an improvement. When there are as many as the patience, training stops and the network goes back to the weights of the best epoch. In the lab, with a patience of 10, training stops in epoch 38 and the weights of epoch 28 are kept: 86.1% on the test instead of 76.3% after 100 epochs.

Too little patience hurts too. The validation curve wobbles from epoch to epoch. With a patience of 5 training stops as early as epoch 23 and keeps the weights of epoch 18 (85.0% on the test), although the best was epoch 28.

Jak to policzono

• Digits: Optical Recognition of Handwritten Digits, UCI, CC BY 4.0 licence. 1797 images of 8×8 pixels.
• Stratified split: 299 images to learn from, 501 for validation, 997 for the test. In the noisy version each training label is replaced, with probability 20%, by a random digit: 51 labels changed (a random digit sometimes hits the true one).
• Network 64 → 32 → 10, ReLU, cross-entropy, SGD with momentum 0.9, step 0.05, batches of 32 images, 100 epochs. Validation loss computed after every epoch.
• Validation and test have the true labels. The noise is in the training data only.

Why it works

An example with a wrong label is hard for the network: the image looks like a three, but the label says "seven". To lower the loss on such an example, the network has to learn a rule that fits only that example. It usually does so late, once the easy, shared patterns are learned. In the lab the network repeats 10 of the 51 wrong labels in epoch 28, and all 51 after 100 epochs. On its own training data it then has 99.7% accuracy: it agrees even with the labels that are false.

Rules fitted to wrong labels spoil the answers on new images. The validation set has the true labels, so its loss rises exactly when the network starts learning the noise. That is why the validation loss, not the training loss, tells you when to stop.

This is not a quirk of one run. In the code below the same experiment runs in scikit-learn on 10 other splits and noise draws: the best epoch falls between 13 and 23, and training to 100 epochs lowers the test accuracy in every one of the 10 runs, on average from 87.2% to 76.4%. The lab’s best epoch (28) falls outside that range because it also depends on the starting weights and the batch order: the lab’s own network with 9 other random seeds does best between epochs 12 and 25, and training to 100 epochs lowers the test accuracy every time.

Common mistake

True or false: "training longer always gives a better model". False. Training longer always fits the training data better, but not new data. With random labels, 100 epochs give 99.7% accuracy on the training set and 76.3% on the test, while stopping at epoch 28 gives 86.1% on the test.

A note the other way round: with clean labels training to the end spoils almost nothing. The lowest validation loss comes in epoch 30 (93.9% on the test), and after 100 epochs the accuracy is 94.5%. Early stopping does not always save accuracy, but it always saves time and guards against data that are worse than you think.

import copy
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import log_loss

X, y = load_digits(return_X_y=True)
X = X / 16
def run(seed):  # 300 digits to learn from, 500 for validation, the other 997 for the test
    used, te = train_test_split(np.arange(len(y)), train_size=800, stratify=y, random_state=seed)
    tr, va = train_test_split(used, test_size=500, stratify=y[used], random_state=seed)
    rng = np.random.default_rng(seed)
    noisy = y[tr].copy()
    flip = rng.random(len(tr)) < 0.2  # 20% of the training labels replaced by a random digit
    noisy[flip] = rng.integers(0, 10, flip.sum())
    net = MLPClassifier(hidden_layer_sizes=(32,), solver="sgd", learning_rate_init=0.05, momentum=0.9, batch_size=32, alpha=0, random_state=seed)
    best, best_loss = None, np.inf
    for epoch in range(1, 101):
        net.partial_fit(X[tr], noisy, classes=np.arange(10))
        loss = log_loss(y[va], net.predict_proba(X[va]), labels=np.arange(10))
        if loss < best_loss:
            best, best_loss, best_epoch = copy.deepcopy(net), loss, epoch  # early stopping keeps the weights of the best epoch
    return best_epoch, best.score(X[te], y[te]), net.score(X[te], y[te])
r = np.array([run(s) for s in range(10)])
print(f"10 splits and noise draws: lowest validation loss at epoch {r[:, 0].min():.0f} to {r[:, 0].max():.0f}; test at that epoch {r[:, 1].mean():.1%}, after 100 epochs {r[:, 2].mean():.1%}; lower after 100 in {(r[:, 2] < r[:, 1]).sum()} of 10")

10 splits and noise draws: lowest validation loss at epoch 13 to 23; test at that epoch 87.2%, after 100 epochs 76.4%; lower after 100 in 10 of 10