Pipelines: from a table to a prediction

In lesson 4.3 we scaled measurements, and in lesson 4.5 we hunted for leakage. Today we put it together into one machine. A pipeline is a list of steps that turns a raw table into a prediction: scaling, encoding, the model. In scikit-learn it is one object: you fit it like a model and ask it like a model. The key question is: in what order?

Try it

• Start with the scaler from training. Switch to the scaler from all data. How many predictions change, and why?
• Choose the scaler fitted on the test and change how the penguins arrive. When does the model stop working?
• Put the pipeline together from the blocks. Which of the three traps tempts you most?

Once you have checked it in the lab, let's name what you see.

The idea in plain words

A scaler is not data preparation “before the model”. It is the first part of the model. For each column it keeps two numbers: a mean and a standard deviation. First it learns them, then it converts with them. Like every part of a model, it learns on the training part only. You convert the test with the same numbers, just as you will convert every new penguin later.

A scaler fitted on the test itself breaks this rule: the test gets its own mean and spread. When the whole test arrives at once and resembles the training data, the result looks fine: 98.0%. When batches from one island arrive, it is only 82.4%. Torgersen has only Adelie penguins in the data. Once their own mean is subtracted, the model recognises 6 of 13. When the penguins arrive one at a time, the accuracy falls to 44.1%. A single penguin has no spread, so after the conversion it is zero in every column. The model calls every one of them “Adelie”, which is exactly what “always Adelie” does.

A pipeline glues the steps into one object, here the scaler and kNN. Fitted on the training part, it learns the means, the standard deviations and the model itself in one go, and when predicting it only converts and asks. Cross-validation with a pipeline fits the scaler again in every fold, so no test fold touches the learning. What this looks like in scikit-learn is shown in the “In code” layer at the bottom of the lesson.

Jak to policzono

• Penguins: Palmer Penguins (Gorman, Williams and Fraser, 2014; licence CC0), 342 penguins with four measurements. Stratified split: 240 to learn from, 102 to test. The dots in the lab are 20 splits with other seeds.
• Model: kNN with k = 5 on standardised measurements (mean 0, standard deviation 1). The scaler is fitted on the training part, on all 342 penguins, or on the test itself: all at once, separately on each island, or separately on each penguin.

Why it works

In real use a model gets penguins that were not in the data, often one at a time. There is no “all the data” and no “whole test” then. There are new penguins and the numbers remembered during learning. An evaluation is honest when the test goes exactly the same way. A pipeline enforces it: it learns on the training part only, and everything else is only converted and scored.

A scaler fitted on all the data breaks this rule too, only quietly. On the penguins it changes next to nothing: the mean flipper length of 342 and of 240 penguins differs by 0.2 mm, and in every one of the lab's 20 splits the predictions are the same. The apparent and the honest result differ by 0 here. In the code below, scikit-learn on 50 splits gives the same predictions in 50 of 50. But a scaler never sees the labels. A step that does can leak a lot in the same place: in lesson 4.5, selecting features on all the data made a “model” out of pure noise. A pipeline protects you from both, so you do not have to check every time whether the leak is small this time.

A common mistake

True or false: “You can also fit the scaler on the test itself; after all, the test and the training data then have the same scale”. False. The test gets its own mean and spread, while the model learned on the numbers from training. On the penguins it looks harmless only when the whole test arrives at once: 98.0%. In batches from one island it is 82.4%, one penguin at a time 44.1%. Over the lab's 20 splits, 76.9% and 42.9% on average, while the pipeline gives 99.2%. In the code below: all at once 0.971, by island 0.786, one at a time 0.437, as much as “always Adelie”.

The rule is simple: the scaler learns on the training part, and the test and new data are only converted with the same numbers. A pipeline does it for you.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

df = pd.read_csv("data/raw/penguins.csv")
cols = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g"]
d = df.dropna(subset=cols)
X, y = d[cols], d["species"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
pipe = make_pipeline(StandardScaler(), KNeighborsClassifier(5))  # the scaler is part of the model
print(f"train {len(y_tr)}, test {len(y_te)}; CV on the training part: {cross_val_score(pipe, X_tr, y_tr, cv=5).mean():.3f}")
pipe.fit(X_tr, y_tr)
print(f"test, scaler fitted on the training part: {pipe.score(X_te, y_te):.3f}")
knn = pipe[-1]  # the fitted kNN, without its scaler
batch = knn.score(StandardScaler().fit_transform(X_te), y_te)  # fit_transform on the test
right = [knn.predict(StandardScaler().fit_transform(part)) == y_te[part.index] for _, part in X_te.groupby(d["island"])]
one = [knn.predict(StandardScaler().fit_transform(X_te.loc[[i]]))[0] == y_te[i] for i in X_te.index]
print(f"fit_transform on the test: all at once {batch:.3f}, by island {np.concatenate(right).mean():.3f}, one at a time {np.mean(one):.3f}")
print(f"always Adelie: {(y_te == 'Adelie').mean():.3f}")
same = equal = 0
for s in range(50):
    a, b, ya, yb = train_test_split(X, y, test_size=0.3, stratify=y, random_state=s)
    honest = make_pipeline(StandardScaler(), KNeighborsClassifier(5)).fit(a, ya).predict(b)
    scaler = StandardScaler().fit(X)  # scaler.fit(X): all penguins, before the split
    leaky = KNeighborsClassifier(5).fit(scaler.transform(a), ya).predict(scaler.transform(b))
    same += (honest == leaky).all()
    equal += (leaky == yb).sum() == (honest == yb).sum()
print(f"scaler.fit(X) before the split, 50 splits: the same predictions in {same} of 50, the same test accuracy in {equal} of 50")

train 239, test 103; CV on the training part: 0.983
test, scaler fitted on the training part: 0.961
fit_transform on the test: all at once 0.971, by island 0.786, one at a time 0.437
always Adelie: 0.437
scaler.fit(X) before the split, 50 splits: the same predictions in 50 of 50, the same test accuracy in 50 of 50