Which model when?

You now know five families of models: logistic regression, k nearest neighbours (kNN), the random forest, gradient boosting and the SVM with the RBF kernel. Which one should you pick? The honest answer: the one that does best in validation on your data. This lesson checks how much the result depends on the data and how to tell a real difference from noise. We build a model zoo: each of the five models with its default settings, no tuning, on five datasets, and on every dataset on the same 20 splits. Each time 75% of the rows go to learning and 25% to validation.

Try it

• At the start: digits, kNN against the SVM. The gap between the means is smaller than kNN's spread between splits. On how many of the 20 splits is the SVM right more often?
• Go to the breast tumours and compare the SVM with gradient boosting. How big is the gap between the means, and how do the splits divide?
• Find a pair of models where one leads on the digits and the other on the penguins.
• On each dataset check how far apart the worst and the best split of one model lie.

Once you have checked it in the lab, let us name what you see.

The idea in plain words

The same model on the same dataset gets different results on different splits, because validation gets easier examples one time and harder ones another. On the breast tumours logistic regression gets from 89.4% to 96.5% on single splits. Differences between models are often much smaller. So one split is not enough to rank them.

It is better to compare models in pairs: on the same split both have it equally easy or equally hard. We count on how many splits each one is right more often. On the tumours the SVM gets 94.2% on average and gradient boosting 93.9%. The SVM is right more often on 10 splits, boosting on 7, and 3 are ties. Such a gap of 0.3 points is a draw, not a win. It can also go the other way. On the digits the SVM is only 0.7 points ahead of kNN, while kNN's result jumps between splits from 96.0% to 99.1%. Even so, the SVM is right more often on 16 splits of 20, and kNN on 2. What counts is whether the lead repeats, not how big the mean is.

When does a lead repeat? If the two models were equally good, every split without a tie would be like a coin toss. A coin gives 16 to 2 less often than once in 100 series, and 10 to 7 often. The lab calls a lead repeatable when such a result would come up less often than once in 100 series. This is the sign test. Ties do not weigh on it: they are simply not counted.

Most important: the order depends on the data. On the digits the SVM is right more often than logistic regression on 17 splits of 20, logistic regression more often than the SVM on 3. On the penguins it is the other way round: logistic regression is right more often on 9 splits, the SVM on none, and 11 are ties. Both leads pass the sign test. On the digits the SVM is right more often than each of the other models on at least 16 splits.

All this is with default settings, no tuning, on small datasets (from 150 to 1797 rows) with numeric features only. It is not a verdict on the models in general. On larger tables, with features of different types and after tuning, the order can look different (boosting is in lesson 5.3). The conclusion is a different one: you do not take a ranking from someone else's table, you check it on your own data.

The leader is also easy to overrate. Take the model with the best mean on splits 1–10 and check who leads on splits 11–20. On the irises, the wines and the penguins another model leads there. When several models lie close together, noise decides to a large extent which one comes out on top, as when choosing settings in lesson 3.5.

Jak to policzono

• Data: Iris (Fisher, 1936), Wine (Aeberhard and Forina, 1992), Breast Cancer Wisconsin (Wolberg et al., 1993; 10 „mean” measurements) and Digits (Alpaydin and Kaynak, 1998), all from UCI under the CC BY 4.0 licence; Palmer Penguins (Gorman et al., 2014; Horst et al., 2020; licence CC0 1.0), 342 penguins with 4 body measurements.
• Models: scikit-learn's default settings without tuning. Logistic regression (C = 1), kNN (k = 5), random forest (100 trees), gradient boosting (100 trees, step 0.1, depth 3), SVM with the RBF kernel (C = 1). Logistic regression, kNN and the SVM get features scaled on each split using the learning part only.
• Agreement with scikit-learn on the same 20 splits: the SVM gets the same rows right on every split, logistic regression and kNN differ by at most 3 rows on a split. The trees in the lab cut every feature into at most 32 bins, while scikit-learn grows exact trees, so here the differences are bigger: on the tumours the forest gets the same result on 3 splits of 20, boosting on 4. With exact trees boosting, not the SVM, has the best mean on the tumours. The conclusions about pairs in this lesson come out the same in both versions.
• Splits: 20 stratified splits, 75% to learn from and 25% to validate on, the same for every model. There is no test here. All 500 results were computed by the lab's code, and the test recomputes them from scratch. The sign test is two-sided, on the splits without a tie, with a threshold of 1%.

Why it works

The „no free lunch” theorem (Wolpert, 1996) says that averaged over all possible tasks no learning algorithm is better than another. A model wins only when its assumptions fit the particular data: straight boundaries for logistic regression, similar neighbours for kNN, thresholds on single features for trees. That is why choosing a model is a measurement, not knowledge from a table.

Comparing in pairs works because the difficulty of a split hits both models at once and cancels out in the difference. A caveat: 20 splits of the same dataset share rows, so they are not 20 independent coin tosses. Ordinary statistical tests on such results tend to be overconfident (Dietterich, 1998). That is why the threshold is strict, and a repeating lead is a good guide, not a proof.

A model can also have randomness of its own. The forest draws rows and features for every tree. On the digits, on the same 20 splits, the forest with random seed 1 is right more often than logistic regression on 15 splits and less often on 5, and with seed 4 more often on 11 and less often on 9. With no seed from 1 to 5 does this lead pass the sign test. That is why the examples in this lesson rest on models without randomness: logistic regression, kNN and the SVM.

The code below does the same in scikit-learn, on other splits and with exact trees. The top changes: there the SVM has the best mean on four datasets and logistic regression on the penguins. The direction of this lesson's pairs stays, but not every lead passes the sign test there. One more reason not to call differences within the noise a win.

Common mistake

True or false: „a model with a mean 0.3 points higher is better”. False. On the tumours the SVM gets 94.2% and gradient boosting 93.9%, but the SVM is right more often on only 10 splits of 20, boosting on 7. In the code below the same pair splits 10 to 7, and the gap between the means is 0.002.

A second common mistake: „always take boosting, XGBoost for example”. In this zoo, with default settings, on the wines logistic regression is right more often than gradient boosting on 15 splits of 20, and boosting more often than it on 2. That is not a verdict against boosting, only against choosing without validation.

import pandas as pd
from scipy.stats import binomtest
from sklearn.datasets import load_iris, load_wine, load_breast_cancer, load_digits
from sklearn.model_selection import StratifiedShuffleSplit, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC

body = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g"]
peng = pd.read_csv("data/raw/penguins.csv").dropna(subset=body)
cancer = load_breast_cancer()
data = {"iris": load_iris(return_X_y=True), "wine": load_wine(return_X_y=True), "cancer": (cancer.data[:, :10], cancer.target),
        "penguins": (peng[body].values, peng["species"].values), "digits": load_digits(return_X_y=True)}
# the scaler sits inside each pipeline, so it learns its means only from the learning part of every split
models = {"logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
          "kNN": make_pipeline(StandardScaler(), KNeighborsClassifier()),
          "forest": RandomForestClassifier(random_state=0),
          "boosting": GradientBoostingClassifier(random_state=0),
          "SVM": make_pipeline(StandardScaler(), SVC())}
splits = StratifiedShuffleSplit(n_splits=20, test_size=0.25, random_state=0)
acc = {}
for name, (X, y) in data.items():
    acc[name] = {m: cross_val_score(model, X, y, cv=splits) for m, model in models.items()}
    row = " | ".join(f"{m} {a.mean():.3f}" for m, a in acc[name].items())
    best = max(acc[name], key=lambda m: acc[name][m].mean())
    print(f"{name:8s} {row}  best: {best}")
# sign test: only the splits without a tie; were the models equally good, each of them would be a coin toss
for name, a, b in [("digits", "SVM", "kNN"), ("digits", "SVM", "logistic"), ("penguins", "logistic", "SVM"), ("cancer", "SVM", "boosting")]:
    d = acc[name][a] - acc[name][b]
    win, lose = (d > 0).sum(), (d < 0).sum()
    print(f"{name}: {a} beats {b} in {win} of 20 splits, loses in {lose}; sign test p = {binomtest(win, win + lose).pvalue:.3f}; mean difference {d.mean():.3f}")

iris     logistic 0.955 | kNN 0.945 | forest 0.949 | boosting 0.947 | SVM 0.959  best: SVM
wine     logistic 0.982 | kNN 0.964 | forest 0.981 | boosting 0.946 | SVM 0.986  best: SVM
cancer   logistic 0.931 | kNN 0.938 | forest 0.930 | boosting 0.939 | SVM 0.942  best: SVM
penguins logistic 0.988 | kNN 0.985 | forest 0.978 | boosting 0.973 | SVM 0.983  best: logistic
digits   logistic 0.969 | kNN 0.974 | forest 0.975 | boosting 0.964 | SVM 0.979  best: SVM
digits: SVM beats kNN in 14 of 20 splits, loses in 4; sign test p = 0.031; mean difference 0.006
digits: SVM beats logistic in 17 of 20 splits, loses in 3; sign test p = 0.003; mean difference 0.010
penguins: logistic beats SVM in 7 of 20 splits, loses in 1; sign test p = 0.070; mean difference 0.005
cancer: SVM beats boosting in 10 of 20 splits, loses in 7; sign test p = 0.629; mean difference 0.002