Baselines: what do you compare your model with?

In lesson 0.1 the tree got 95.1% of the penguins right and guessing got 43.7%. That second number is the baseline: the score of an answer that learns nothing. Without it you cannot tell whether a model has learned anything. Today you will see how to compute it in two tasks and when accuracy alone fools you. The data: 569 breast tumour biopsies and 442 patients with diabetes.

Try this

• Start with the biopsies and all 64 malignant ones. By how much does the learned rule beat “always benign”?
• Move the slider right until there are 5 malignant biopsies, then 1. What happens to the accuracy of “always benign”, and what happens to the number of malignant biopsies found?
• Switch to diabetes. Compare the error of the line on BMI and of the model with 10 measurements with the error of the mean alone.

Once you have checked it in the lab, let us name what you see.

The idea in plain words

A baseline is the simplest answer you can give without learning. In classification it is the most common class: of the 398 learning biopsies 250 are benign, so the answer is always “benign”. In regression it is the mean: every patient gets the same predicted disease progression, the average of the learning patients.

A model makes sense only when it clearly beats the baseline on the same test data. On the biopsies the rule with one measurement and one threshold gets 91.2% against 62.6%. In diabetes the model with 10 measurements has an error of 2827 against 5076 for the mean, 44.3% lower. A line on BMI alone lowers the error by only 19.3%.

There is a weaker baseline too: guessing in proportion, that is “benign” with a chance equal to the share of benign biopsies among the learning ones. On average it gets 53.2%, less than “always benign”. That is why the most common class is the baseline to use: random guessing is easier to beat than real understanding.

Jak to policzono

• Biopsies: Breast Cancer Wisconsin (Diagnostic), UCI, licence CC BY 4.0. 569 biopsies (357 benign, 212 malignant), 10 measurements of the cell nuclei.
• Diabetes: the Diabetes dataset (Efron et al., 2004) shipped with scikit-learn; the authors gave no licence. 442 patients, 10 measurements, disease progression a year later.
• Split 70/30, the same as train_test_split(test_size=0.3, random_state=1) in scikit-learn; biopsies stratified. The test has 171 biopsies and 133 patients.
• Rule: a decision tree with one question (CART). Regression model: least squares.

Why it works

The accuracy of “always the most common class” is simply that class’s share of the test. On the full test 107 of 171 biopsies are benign, hence 62.6%. Keep 1 malignant biopsy and the benign ones are 107 of 108, hence 99.1%. The baseline’s accuracy says nothing about the biopsies. It only says how rare the other class is.

That is why with a rare class accuracy no longer separates a good model from a useless one. With 5 malignant biopsies in the test “always benign” gets 95.5% and the rule 91.1%, although only the rule finds all 5. This is not one lucky split: on 100 different splits “always benign” beats the rule on accuracy in 82. What helps is a second number next to accuracy: how many cases of the rare class the model actually found.

In regression the mean is the best constant answer: no other single number gives a lower mean squared error on the learning data. Any model that has measurements should beat it, and here it always does. On 100 splits the line on BMI and the model with 10 measurements have a lower error than the mean every time, on average 33.7% and 48.5% lower.

Common mistake

True or false: “90% accuracy is a lot”. False, or rather: the number alone does not tell you. When 1 person in 100 is sick, the test “always healthy” has 99% accuracy and finds no one. A model with 90% would then be worse than nothing on accuracy, yet it could be very valuable if it finds the sick. Always report the baseline next to the score and check how many cases of the rare class the model found.

Where the analogy breaks down: a baseline is a floor, not a goal. Beating it by a hair usually means nothing, because another split of the data can turn the result around.

import numpy as np
from sklearn.datasets import load_breast_cancer, load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LinearRegression

X, y = load_breast_cancer(return_X_y=True)
X = X[:, :10]  # the 10 "mean" measurements, as in the lab; y: 0 = malignant, 1 = benign
def biopsies(seed, keep_malignant):
    tr, te = train_test_split(np.arange(len(y)), test_size=0.3, stratify=y, random_state=seed)
    te = np.sort(te)  # test rows in file order, as in the lab
    major = np.bincount(y[tr]).argmax()  # baseline: the most common class among the learning biopsies
    rule = DecisionTreeClassifier(max_depth=1, random_state=0).fit(X[tr], y[tr])  # one measurement, one threshold
    rows = np.r_[te[y[te] == 1], te[y[te] == 0][:keep_malignant]]  # every benign + the first m malignant
    pred = rule.predict(X[rows])
    return (y[rows] == major).mean(), ((major == 0) & (y[rows] == 0)).sum(), (pred == y[rows]).mean(), ((pred == 0) & (y[rows] == 0)).sum()
for m in (64, 5, 1):
    base, base_found, acc, found = biopsies(1, m)
    print(f"malignant {m}: always benign {base:.3f} (finds {base_found}), rule {acc:.3f} (finds {found})")
wins = sum(biopsies(s, 5)[0] > biopsies(s, 5)[2] for s in range(100))
print(f"100 splits, 5 malignant: always benign beats the rule in {wins}")

Xd, yd = load_diabetes(return_X_y=True)
def diabetes(seed):
    a, b, c, e = train_test_split(Xd, yd, test_size=0.3, random_state=seed)
    mse = lambda p: ((e - p) ** 2).mean()
    return mse(c.mean()), mse(LinearRegression().fit(a[:, [2]], c).predict(b[:, [2]])), mse(LinearRegression().fit(a, c).predict(b))
mean, bmi, full = diabetes(1)
print(f"diabetes: mean {mean:.0f}, BMI {bmi:.0f} ({1 - bmi / mean:.1%} lower), all 10 {full:.0f} ({1 - full / mean:.1%} lower)")
r = np.array([diabetes(s) for s in range(100)])
print(f"100 splits: BMI beats the mean in {(r[:, 1] < r[:, 0]).sum()}, all 10 in {(r[:, 2] < r[:, 0]).sum()}; error lower by {1 - r[:, 1].mean() / r[:, 0].mean():.1%} and {1 - r[:, 2].mean() / r[:, 0].mean():.1%} on average")

malignant 64: always benign 0.626 (finds 0), rule 0.912 (finds 59)
malignant 5: always benign 0.955 (finds 0), rule 0.911 (finds 5)
malignant 1: always benign 0.991 (finds 0), rule 0.907 (finds 1)
100 splits, 5 malignant: always benign beats the rule in 82
diabetes: mean 5076, BMI 4096 (19.3% lower), all 10 2827 (44.3% lower)
100 splits: BMI beats the mean in 100, all 10 in 100; error lower by 33.7% and 48.5% on average