Tuning hyperparameters without cheating

In lesson 3.2 cross-validation scored one setting of a model. In practice you try many and take the best. That is called tuning. The settings you choose before training (the depth of a tree, the strength of the penalty α from lesson 3.4) are hyperparameters: the model does not learn them from the data, you choose them. Today you will see that the winner's score in tuning is too optimistic, and why.

Try it

• With the grid, move the budget from 4 to 9. What does the winner score in cross-validation, and what on the test?
• Switch to random search and click "new draw" a few times. Is the winner ever better on the test than in validation?
• In the second part move the seed slider from 1 to 20. How does the best score grow, and how far does it move from the mean of 200 splits?

Once you have checked it in the lab, let's name what you see.

The idea in plain words

Tuning means comparing many settings on validation data and choosing the best one. Grid search tries every combination of a few values. Random search draws combinations from the whole range. On the same budget a 3×3 grid tries only 3 depths: 1, 6 and 12. Nine random tries usually see more different values of each hyperparameter.

Every validation score is the setting's true quality plus a bit of luck: which biopsies happened to land in which part. When you pick the best score, you also pick the setting that luck favoured most. On new data that luck is gone. This is the winner's curse. It grows with the number of settings you compare and shrinks with the size of the data.

The same happens when you look for the best split of the data. The same logistic regression on 200 random splits scores 93.8% on the test on average. If you try seeds 1–20 and report the best one, you get 96.5%, 2.7 points more. The code is the same; you only picked the luck.

Jak to policzono

• Biopsies: Breast Cancer Wisconsin (Diagnostic), UCI, CC BY 4.0 licence. 569 biopsies, 10 "mean" measurements each.
• Tuning: stratified split, 398 biopsies for tuning and 171 for the test. A CART tree (Gini), 10 depths times 11 minimum numbers of biopsies in a leaf, i.e. 110 settings, each in 5-fold cross-validation on the 398 biopsies.
• Winner: the best mean over the 5 parts; on a tie the simpler tree. On the test, the tree trained on all 398 biopsies is scored.
• Seeds: logistic regression (C = 1) on standardised measurements, 200 random 70/30 splits, 171 test biopsies in each.

Why it works

Take 9 settings of almost the same true quality. Their validation scores differ mostly by noise. The highest of 9 scores is almost always one where the noise was positive. In the lab the best CV gets 373 of 398 biopsies right, and several settings are only one or two biopsies behind. Differences that small are largely chance.

Selection shows in random mode: over 200 random draws of 9 settings, the winner scores higher in validation than on the test in every single one, by 3.6 points on average. The test played no part in the choice, so it carries no selection, only the ordinary noise of a finite set. All 200 draws, though, are different settings on one and the same data split, and that split landed on the pessimistic side: even with no choice at all, the 110 settings score 0.6 points higher in validation than on the test on average. In the code below, on 50 different splits, the winner loses 1.4 points on average, and in 17 of the 50 splits validation does not come out higher than the test.

That is why the test cannot be used for choosing. If you chose a setting by its test score, the best cell would get 92.4% on the test. That number would be inflated too, only harder to notice, because no data would be left to check it.

Common mistake

True or false: "the score of the tuning winner is a forecast for new data". False. The winner's validation score is inflated, because it is the maximum of many noisy numbers. In the lab the winner of 9 settings has 93.7% in validation and 88.3% on the test. In the code below, over 50 splits and 110 settings, the winner scores 93.3% in validation on average and 91.9% on the held-out test.

A fair forecast comes from one of two routes. One: a held-out test that you use for no choice at all and look at once, at the end. Two: nested cross-validation, the whole tuning repeated inside every part of an outer cross-validation. On all 569 biopsies the winner gets 94.0% in ordinary cross-validation, while the nested one gives 91.9%. The second number is closer to what the model will do tomorrow.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV, StratifiedKFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X, y = load_breast_cancer(return_X_y=True)
X = X[:, :10]  # the 10 "mean" measurements, as in the lab
grid = {"max_depth": [1, 2, 3, 4, 5, 6, 7, 8, 10, 12], "min_samples_leaf": [1, 2, 3, 4, 6, 8, 12, 16, 24, 32, 48]}  # the lab's 110 settings
tree = DecisionTreeClassifier(random_state=0)
r = []
for s in range(50):
    tr, te = train_test_split(np.arange(len(y)), test_size=0.3, stratify=y, random_state=s)
    search = GridSearchCV(tree, grid, cv=StratifiedKFold(5, shuffle=True, random_state=s)).fit(X[tr], y[tr])
    r.append((search.best_score_, search.score(X[te], y[te])))  # the winner's CV score, then the winner on the held-out test
r = np.array(r)
print(f"winner of 110 settings, 50 splits: CV {r[:, 0].mean():.1%}, held-out test {r[:, 1].mean():.1%}; CV higher in {(r[:, 0] > r[:, 1]).sum()} of 50")
outer = StratifiedKFold(5, shuffle=True, random_state=0)
flat = GridSearchCV(tree, grid, cv=outer).fit(X, y).best_score_  # the winner's score on the same folds that chose it
nested = cross_val_score(GridSearchCV(tree, grid, cv=StratifiedKFold(5, shuffle=True, random_state=1)), X, y, cv=outer).mean()  # the search runs inside each outer fold
print(f"all 569 biopsies: winner's CV {flat:.1%}, nested CV {nested:.1%}")
model = make_pipeline(StandardScaler(), LogisticRegression())
acc = []
for s in range(200):  # 200 random splits, 30% test, as in the lab
    tr, te = train_test_split(np.arange(len(y)), test_size=0.3, random_state=s)
    acc.append(model.fit(X[tr], y[tr]).score(X[te], y[te]))
acc = np.array(acc)
best = acc.reshape(10, 20).max(axis=1)  # ten people, each tries 20 seeds and reports the best one
print(f"logistic regression, 200 seeds: mean {acc.mean():.1%}; best of 20 seeds {best.mean():.1%} on average (+{(best.mean() - acc.mean()) * 100:.1f} points), higher in {(best > acc.mean()).sum()} of 10")

winner of 110 settings, 50 splits: CV 93.3%, held-out test 91.9%; CV higher in 33 of 50
all 569 biopsies: winner's CV 94.0%, nested CV 91.9%
logistic regression, 200 seeds: mean 93.7%; best of 20 seeds 96.4% on average (+2.6 points), higher in 10 of 10