Categories and scaling

In lesson 1.4 kNN got 76.7% of the penguins right without scaling and 96.1% after standardising, because body mass in grams drowned out the other measurements. Today: two things that break a model before it starts learning. The first is features on very different scales; scaling brings them to a comparable range. The second is categories, names instead of numbers, such as the island a penguin comes from. Encoding categories turns them into numbers, and it can be done badly.

Try it

• Start with the wines without scaling. Which measurement takes up almost the whole height of the chart?
• Switch to min-max, standardisation and robust scaling. How much does kNN gain, and how much do the three differ?
• Go to the penguins. Compare the codes 0, 1, 2 in alphabetical order and in the order of the file. Should a model depend on the order in which you list the islands?
• Pick one-hot. Where do the prediction lines lie now compared with the dots?

Once you have checked it in the lab, let's name what you see.

The idea in plain words

kNN calls two wines similar when they are close in all 13 measurements at once. Proline has a standard deviation of 314, the least variable measurement only 0.124, so in the sum of differences proline covers everything else. Scaling evens out these ranges. Min-max moves every measurement into the interval from 0 to 1, standardisation to mean 0 and standard deviation 1, robust scaling subtracts the median and divides by the interquartile range, so single extreme values move it less. On the wines kNN scores 0.958, 0.963 and 0.946 instead of 0.695. The choice of method matters less than scaling at all.

Categories have a different problem. An island is Biscoe, Dream or Torgersen, and linear regression needs numbers. The simplest idea is the codes 0, 1, 2. But a number carries an order and a distance: a model with one weight will assume that the island with code 1 lies exactly halfway between the other two. In the lab a regression predicts a penguin's body mass from its island and sex. With alphabetical codes the cross-validated R² is 0.469. The same codes in the order of the file give 0.193, although the data are identical. One-hot gives every island its own 0/1 column and its own weight: R² 0.546, and the mean error falls from 476 to 411 g.

Jak to policzono

• Wines: Wine (Aeberhard and Forina, 1992; UCI, licence CC BY 4.0), 178 wines, 13 chemical measurements, 3 cultivars. kNN with k = 5, cross-validation in 5 parts, repeated 10 times with a different shuffle. The scaler is fitted only on the training part of every split.
• Penguins: Palmer Penguins (Gorman, Williams and Fraser, 2014; palmerpenguins package: Horst, Hill and Gorman, 2020), licence CC0. 333 of 344 penguins: 11 have no recorded sex. Linear regression: body mass from island and sex (one 0/1 column), cross-validation in 5 parts, 10 repeats. Lines on the chart: the model fitted on all 333.
• R²: the share of the variation in body mass explained on the validation parts (1 = all of it, 0 = as much as the mean). Mean error: the mean absolute difference in grams.

Why it works

The distance in kNN is the square root of the sum of squared differences. A measurement with a large spread gives large differences, and they rule the sum. Standardising divides every measurement by its standard deviation, so a typical difference has a similar size in every measurement. The scaler's means, standard deviations, minima and maxima are computed from data, just like a model's weights. That is why a scaler is fitted on the training part only: in the code below a scaler fitted on all wines before the cross-validation gives 0.965 instead of 0.962. The difference is small here, but that score has seen the validation part, so it is not an honest estimate.

With island codes, linear regression has one weight for the whole column. So from code 0 to 1 and from 1 to 2 the predicted mass changes by the same amount. With alphabetical codes the model predicts that penguins from Dream weigh 625 g less than those from Biscoe, and those from Torgersen another 625 g less. In the data penguins from Dream and Torgersen weigh almost the same (3719 and 3709 g), those from Biscoe 4719 g. In the order of the file Biscoe gets the middle code, and the line through the three islands almost flattens: 243 g per step. One-hot assumes no order at all: the model predicts 997 g less for Dream and for Torgersen than for Biscoe, just as in the data.

Common mistake

True or false: “code categories as 1, 2, 3, the model will cope”. False. A number carries an order and equal gaps that do not exist between islands. In the lab merely reordering the islands changes R² from 0.469 to 0.193, although the data are the same. One-hot gives 0.546 for any order. In the code below one-hot beats the alphabetical codes in 49 of 50 validation folds.

The exception is sex: with two categories one 0/1 column is enough, because there is no middle for anything to end up in. The trouble starts at three categories without a natural order.

import numpy as np
import pandas as pd
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score, RepeatedStratifiedKFold, RepeatedKFold
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression

X, y = load_wine(return_X_y=True)
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0)
acc = {}
for name, scaler in [("no scaling", None), ("min-max", MinMaxScaler()), ("standardised", StandardScaler()), ("robust", RobustScaler())]:
    model = make_pipeline(scaler, KNeighborsClassifier(5)) if scaler else KNeighborsClassifier(5)  # the scaler in a pipeline: fitted in every split
    acc[name] = cross_val_score(model, X, y, cv=cv)
print("kNN on 178 wines, 10 x 5 CV: " + ", ".join(f"{k} {v.mean():.3f}" for k, v in acc.items()))
print(f"standardised beats no scaling in {(acc['standardised'] > acc['no scaling']).sum()} of 50 folds")
leaky = cross_val_score(KNeighborsClassifier(5), StandardScaler().fit_transform(X), y, cv=cv)  # this scaler has also seen the validation parts
print(f"scaler fitted on all wines before the CV: {leaky.mean():.3f}")
df = pd.read_csv("data/raw/penguins.csv").dropna(subset=["sex", "body_mass_g"])
male = (df.sex == "MALE").to_numpy(float)[:, None]
codings = {
    "codes alphabetical": df.island.map({"Biscoe": 0, "Dream": 1, "Torgersen": 2}).to_numpy(float)[:, None],
    "codes in file order": df.island.map({"Torgersen": 0, "Biscoe": 1, "Dream": 2}).to_numpy(float)[:, None],
    "one-hot": pd.get_dummies(df.island).to_numpy(float),
}
cv2 = RepeatedKFold(n_splits=5, n_repeats=10, random_state=0)
r2 = {}
for name, isl in codings.items():
    Xe = np.hstack([isl, male])
    r2[name] = cross_val_score(LinearRegression(), Xe, df.body_mass_g, cv=cv2, scoring="r2")
    mae = -cross_val_score(LinearRegression(), Xe, df.body_mass_g, cv=cv2, scoring="neg_mean_absolute_error").mean()
    print(f"{len(df)} penguins, body mass from island + sex, {name}: R2 {r2[name].mean():.3f}, mean error {mae:.0f} g")
print(f"one-hot beats the alphabetical codes in {(r2['one-hot'] > r2['codes alphabetical']).sum()} of 50 folds")

kNN on 178 wines, 10 x 5 CV: no scaling 0.706, min-max 0.963, standardised 0.962, robust 0.952
standardised beats no scaling in 50 of 50 folds
scaler fitted on all wines before the CV: 0.965
333 penguins, body mass from island + sex, codes alphabetical: R2 0.464, mean error 477 g
333 penguins, body mass from island + sex, codes in file order: R2 0.179, mean error 618 g
333 penguins, body mass from island + sex, one-hot: R2 0.542, mean error 412 g
one-hot beats the alphabetical codes in 49 of 50 folds