Data, features, labels and splits

If a teacher gives a class the exact questions that will appear in the final exam, and everyone scores full marks, what has the exam measured? Memory of those questions, nothing more. Machine learning faces the same problem every single time, and the way it solves it is surprisingly simple: keep some of the questions locked in a drawer.

Try this

• Set the test-share slider to a small value and then move the seed slider through several positions. Watch how much the test accuracy changes even though the model and the data stay the same.
• Now set the test-share slider to a large value and repeat the seed experiment. Does the test accuracy jump around more or less than before?
• Keep the seed fixed and move the test-share slider from one end to the other. Notice how the number of training rows shrinks as the test set grows.
• Find the seed that gives the best test accuracy — and then ask yourself whether you would be allowed to report that number.

Now that you've tried it, let's see what was actually going on.

The idea in plain words

Almost every dataset in this course is a table. Each row is one example: one tumour, one patient, one passenger, one wine. Each column is a feature: a measured property such as a radius, a blood pressure, an age or an alcohol content. One special column holds the label — the answer we want the model to predict, such as "malignant" or "survived". A model is a rule that maps the features of a row to its label.

Before any learning happens, the rows are shuffled and divided. The bigger part, the training set, is what the model learns from. The smaller part, the test set, is locked away and used once, at the end, to estimate how the model will do on rows it has never seen. It plays the role of the questions in the drawer.

Think of a chef developing a new soup. She tastes it constantly while cooking — that is training. But to know whether customers will like it, she serves it to a few people who have never tasted it before and have no reason to be polite. Their verdict is the test score. If she only asked herself, she would always think the soup was perfect.

Why it works

The test set is useful only because the model never saw it. As long as the training and test rows come from the same source and were split at random, the test rows behave like future cases, and the test accuracy is a fair estimate of future accuracy.

But it is only an estimate, and here is the part beginners rarely hear. The test score depends on which rows happened to land in the test set. Shuffle the data with a different random seed and you get a different test set, and a different score — with exactly the same model and the same data. With a small test set the score can jump around noticeably, because a handful of unusual rows has a big effect on the percentage. Statistics gives a rough size for this wobble. For an accuracy p measured on n test rows, the standard error is about

SE ≈ √(p · (1 − p) / n)

so four times as many test rows only halves the wobble. This is why a single test score is never the whole truth.

There is also a trade-off. Every row you put into the test set is a row the model cannot learn from. A huge test set gives a reliable grade for a model that learned from too little; a tiny test set gives a well-trained model an unreliable grade. Common choices put somewhere between a fifth and a third of the rows in the test set, and the next lessons will show cleverer ways around the dilemma.

Common mistake

The most tempting error is trying many splits and keeping the one with the nicest test score. The test set has then been used to make a decision, and the reported score is optimistic. A quieter version of the same error is called data leakage: information from the test rows sneaks into training, for example when you compute averages for scaling or filling missing values on the whole table before splitting. The rule is strict: split first, then do everything else using the training rows only.

With a clean split in hand we can finally build a real model. In the next lesson we fit the simplest one there is — a straight line through a cloud of points.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X = X.filter(like="mean ")  # the 10 "mean" features, as in the lab
accs = []
for seed in range(20):  # same model, 20 different splits
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=seed)
    model = make_pipeline(StandardScaler(), LogisticRegression()).fit(Xtr, ytr)
    accs.append(model.score(Xte, yte))
n, p = len(yte), float(np.mean(accs))
print(f"test rows: {n}")
print(f"test accuracy: min {min(accs):.1%}, max {max(accs):.1%}")
print(f"standard error: {np.sqrt(p * (1 - p) / n):.1%} (n={n}), {np.sqrt(p * (1 - p) / (4 * n)):.1%} (n={4 * n})")

test rows: 171
test accuracy: min 90.6%, max 97.1%
standard error: 1.8% (n=171), 0.9% (n=684)