Learning curves: more data or a better model?

In lesson 3.1 the model was too simple or too complex, while the amount of data stayed the same. Today you change the amount of data. It is a practitioner’s everyday question: the model is so-so, so do you collect more examples or change the model? The answer comes from a learning curve: the score on training and on validation, drawn for a larger and larger training set.

Try this

• With the detailed model, move the slider from 50 to 1438 images. Which curve changes more: training or validation?
• Switch to the model of blurred digits. Where do its curves meet, and how high?
• Compare the detailed model with 100 images and the blurred one with 1438. Which wins?

Once you have checked it in the lab, let us name what you see.

The idea in plain words

A learning curve is two lines: accuracy on the training data and on validation, against the number of training examples. With a few dozen images the model memorises each of them: training is 100% and validation 82.3%. The more data, the harder it is to memorise everything, so training falls (to 98.0%) and validation rises (to 95.8%). The gap between them shrinks from 17.7 to 2.2 points.

You read the decision off the shape of those lines:

• A big gap, validation still rising (high variance): the model knows its training data far better than new data. More data will help, and so will regularisation.
• A small gap, both lines low (high bias): the model is too simple. The curves have met and more data will barely help. A richer model or better features will.

The model of blurred digits sees each digit as a 4 × 4 picture, 16 pixels instead of 64. With 1438 images it scores 86.9% on training and 85.0% on validation, a gap of only 1.9 points. Validation usually does not rise above training, and that is only 86.9%. The detailed model already reaches 87.6% with just 100 images.

Jak to policzono

• Digits: Optical Recognition of Handwritten Digits, UCI, licence CC BY 4.0. 1797 images of 8 × 8 pixels, 10 digits.
• A stratified 80/20 split, the same as in the Atlas widget: 1438 images for learning, 359 for validation. For each size the mean of 5 draws of which images are used for learning.
• Model: logistic (softmax) regression with a penalty of 0.001 on large weights. The blurred model sees the means of 2 × 2 pixel blocks.

Why it works

On training the model sees the answers, so with little data it can memorise them. Every new example makes memorising harder and finding a rule that also works outside training easier. That is why the lines move towards each other: training down, validation up. They meet at the level this model can reach at all.

That level depends on the model, not on the data. A blurred picture loses the details that tell, for example, a 3 from an 8, and no number of examples brings them back. In the code below the same experiment runs on 50 different splits: with all the data the blurred model scores 86.4% on training and 85.1% on validation, while the detailed one reaches 96.6% on validation. The detailed model from 100 images beats the blurred one from 1437 images in 49 splits out of 50.

Common mistake

True or false: “more data always helps”. False. More data helps when the model is too powerful for the amount of data: then the gap is big and validation keeps rising. When the model is too simple, the curves meet early and stay flat. In the lab the blurred model has 83.6% on validation with 400 images, and 85.0% with a set almost four times larger. The detailed model has 95.8% with the same data. Before you pay for collecting data, draw a learning curve: if the training score itself is low, data will not help.

Note: draw the curves on validation or in cross-validation, not on the test set. The test is kept for the final evaluation only.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

X, y = load_digits(return_X_y=True)
X = X / 16  # 8 × 8 = 64 pixels from 0 to 1, as in the lab
X16 = X.reshape(-1, 4, 2, 4, 2).mean(axis=(2, 4)).reshape(len(X), -1)  # the same digit blurred to 4 × 4 = 16 pixels
def run(Z, n, seed):
    Xtr, Xva, ytr, yva = train_test_split(Z, y, test_size=0.2, stratify=y, random_state=seed)
    i = np.random.default_rng(seed).permutation(len(ytr))[:n]  # the first n training images of a shuffle
    m = LogisticRegression(C=1000 / n, max_iter=5000).fit(Xtr[i], ytr[i])  # C = 1000 / n is the lab's penalty 0.001
    return m.score(Xtr[i], ytr[i]), m.score(Xva, yva)
for name, Z in (("64 pixels", X), ("16 pixels", X16)):
    for n in (50, 100, 400, 800, 1437):
        tr, va = np.mean([run(Z, n, s) for s in range(50)], axis=0)  # mean of 50 different splits
        print(f"{name}, {n} images: training {tr:.1%}, validation {va:.1%}, gap {(tr - va) * 100:.1f} points")
wins = sum(run(X, 100, s)[1] > run(X16, 1437, s)[1] for s in range(50))
print(f"64 pixels from 100 images beat 16 pixels from 1437 images in {wins} of 50 splits")

64 pixels, 50 images: training 100.0%, validation 81.2%, gap 18.8 points
64 pixels, 100 images: training 100.0%, validation 89.5%, gap 10.5 points
64 pixels, 400 images: training 99.7%, validation 94.9%, gap 4.8 points
64 pixels, 800 images: training 99.0%, validation 96.0%, gap 2.9 points
64 pixels, 1437 images: training 98.2%, validation 96.6%, gap 1.6 points
16 pixels, 50 images: training 98.5%, validation 72.6%, gap 25.9 points
16 pixels, 100 images: training 94.1%, validation 79.8%, gap 14.4 points
16 pixels, 400 images: training 87.7%, validation 83.8%, gap 3.8 points
16 pixels, 800 images: training 86.9%, validation 84.6%, gap 2.3 points
16 pixels, 1437 images: training 86.4%, validation 85.1%, gap 1.3 points
64 pixels from 100 images beat 16 pixels from 1437 images in 49 of 50 splits