Validation and cross-validation

In the last lesson we picked the best polynomial degree by peeking at the test set. It felt harmless, but think about what it means: we tried several models and kept the one the test set liked best. The test set is no longer an unseen exam — it has become a judge we consulted while preparing. How do we choose settings without spoiling our only honest measurement?

Try this

• Set the k slider to its smallest value and read the accuracy of each fold. How different are the folds from each other?
• Increase the k slider step by step. Watch the mean and the spread, and notice that each fold now contains fewer wines.
• At a large k, look for folds with unusually low or high accuracy. With only a handful of wines in a fold, a single mistake moves that fold's score a lot.
• Compare the spread with the differences between settings you might want to tune. Is a small gain in the mean bigger than the spread?

Now that you've tried it, let's see what was actually going on.

The idea in plain words

The first fix is a third pile of data. Split the rows into training, validation and test sets. Train candidate models on the training set, compare them on the validation set, pick the winner, and only then open the test set, once, to report the final score. The validation set absorbs all the peeking so the test set does not have to.

The problem: every split costs data. With a small dataset, a validation set big enough to be reliable leaves too little to train on, and lesson 2 showed that a small set gives a jumpy score. Cross-validation solves this by reusing the data cleverly.

Imagine judging a new cook. Tasting one dish tells you little: maybe it was her best, maybe her worst. So you ask for several dishes on different days and look at the average and how much they vary. Cross-validation does the same with a model.

Why it works

In k-fold cross-validation the data (excluding the final test set) are cut into k equal parts called folds. The model is trained k times. Each time, one fold is held out for validation and the remaining k − 1 folds are used for training. In the end every row has been used for validation exactly once and for training k − 1 times. Each training run sees this share of the data:

training share = (k − 1) / k

The result is not one score but k scores, one per fold. Their mean is a more reliable estimate than any single split, because it does not depend on one lucky or unlucky division of the rows. Their spread is just as valuable: it shows how much the score depends on which rows the model happened to get. Two models whose means differ by less than that spread cannot really be told apart.

The choice of k is a trade-off. A small k (say 2) trains each model on only half the data, so every model is weaker than the one you will finally build, and the estimate is pessimistic. A large k trains on almost all the data, but each fold is small, so individual fold scores become noisier, and the computation takes k times as long as a single fit. Values of 5 or 10 are a common compromise.

The example in this lesson is the Wine dataset: 178 wines from the same region of Italy, made from three different grape cultivars, each described by 13 results of a chemical analysis. The task is to tell the three cultivars apart.

Common mistake

Cross-validation protects you only if everything learned from data happens inside each fold. If you scale features or select the "best" features using all rows first and then cross-validate, information from the validation folds has already leaked into training, and the scores are too optimistic. The second mistake is to tune dozens of settings by cross-validation and report the best cross-validated score as the final result. After so many comparisons, the winner is partly lucky; that is why a separate test set, used once, still matters.

With honest evaluation in our toolkit we can explore more powerful models without fooling ourselves. In the next lesson we meet the most readable of them: the decision tree.

import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))  # scaler is refitted inside every fold
for k in (2, 10):
    runs = [cross_val_score(model, X, y, cv=StratifiedKFold(k, shuffle=True, random_state=s)) for s in range(20)]
    gap = np.mean([a.max() - a.min() for a in runs])
    print(f"k={k:2d}: mean accuracy {np.mean(runs):.3f}, average worst-to-best gap {gap:.3f}")

k= 2: mean accuracy 0.956, average worst-to-best gap 0.020
k=10: mean accuracy 0.966, average worst-to-best gap 0.113