Feature engineering

In lesson 4.3 we changed how features are written: the scale of numbers and the code of categories. Today we make new features out of the ones we already have. Feature engineering means adding columns that show the model more clearly what it is looking for: a square, a ratio, a difference, the number of days since an event. One good feature can turn a hard problem into an easy one. Many weak ones can spoil a model that worked.

Try it

• Start with x alone. Where does the model predict cultivar 2? What is the test accuracy, and where does it come from?
• Add the column x². What does the boundary look like in the plane (x, x²), and on the flavanoid axis alone?
• Switch to noise. Move the slider down from 200 columns. When does kNN recognise the held-out wines best?
• Watch the distance contrast. What happens to the gap between the nearest and the farthest wine as columns pile up?

Once you have checked it in the lab, let's name what you see.

The idea in plain words

A linear model on one axis can do only one thing: put down a threshold. Cultivar 2 sits in the middle of the flavanoid axis, so a threshold cannot cut it out from either side: it cuts off at most one of the neighbouring cultivars. The best such threshold cuts off cultivar 3 and gets 61.6% of the training wines, while saying “another cultivar” for every wine gets 60.0%. The gain is tiny, and on the test it vanishes. The lab's linear SVM predicts cultivar 2 nowhere and gets 60.4%, the share of the other cultivars among the 53 test wines. Let's add the column x², the square of the standardised flavanoids. Wines from both ends of the axis have a large x², wines from the middle a small one. The same linear model now lays a straight line in the plane (x, x²). On the flavanoid axis that is two thresholds, an interval from 1.11 to 2.82. Accuracy rises to 77.4%. The model did not change; what it gets did.

More features do not mean better, though. kNN (k = 5) on 13 chemical measurements recognises 98.1% of the held-out wines. We add columns of pure noise, random numbers on the same scale. With 200 such columns accuracy falls to 77.4%. Every noise column adds as much to the distance as a real measurement, so the “nearest” wines stop being similar. This is the curse of dimensionality: in many dimensions all points lie almost equally far from one another.

Jak to policzono

• Wines: Wine (Aeberhard and Forina, 1992; UCI, licence CC BY 4.0), 178 wines, 13 chemical measurements, 3 cultivars. Stratified split: 125 wines to learn from, 53 to test.
• New feature: a linear model (a linear support vector machine, C = 1) on the standardised flavanoids, cultivar 2 against the rest. Mean flavanoids computed on all 178 wines.
• Noise: kNN with k = 5 on the 13 standardised measurements and 0 to 200 columns of numbers from a normal distribution (mean 0, standard deviation 1), drawn with a fixed seed.

Why it works

A linear model splits space with a straight line (a plane in higher dimensions). On one axis a straight line is one point, a threshold. The column x² adds a second axis. Wines from both ends of the flavanoid axis land high on the parabola, wines from the middle low. Now a straight line can cut cultivar 2 off from above. On 20 other splits of the wines, without x² the linear SVM never once predicts cultivar 2, and the best single threshold from training beats “always other” on the test in only 1 of 20. With x² it always cuts out one interval: from 67.9 to 84.9% on the test.

With noise it is the other way round. The distance in kNN is a sum of squared differences over all columns. With 13 measurements the farthest training wine is on average 296.5% farther than the nearest. With 200 noise columns only 25.7%. The differences in the noise are random, and there are many of them, so they drown the 13 real differences. Over 20 draws of the noise, 200 columns give from 64.2 to 83.0%, always below 98.1% without noise.

A common mistake

True or false: “more features make a better model”. False. A feature helps when it carries information in a shape the model can use, like x² for cultivar 2. Columns without information hurt: 200 noise columns drag kNN from 98.1% down to 77.4%. In the code below the accuracy of kNN falls from 0.962 without noise to 0.689 with 200 columns.

A good new feature usually comes from understanding the problem: what should matter, and in what shape. Throwing in everything you can is an invitation to noise.

import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score, RepeatedStratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier

wine = load_wine(as_frame=True)
x, is2 = wine.data[["flavanoids"]], (wine.target == 1).astype(int)
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0)
line = make_pipeline(StandardScaler(), SVC(kernel="linear", C=1))
squared = make_pipeline(StandardScaler(), PolynomialFeatures(2, include_bias=False), StandardScaler(), SVC(kernel="linear", C=1))  # the new column x²
a, b = cross_val_score(line, x, is2, cv=cv), cross_val_score(squared, x, is2, cv=cv)
print(f"cultivar 2 vs the rest from flavanoids, 10 x 5 CV: x only {a.mean():.3f}, x and x^2 {b.mean():.3f}; always 'other' {1 - is2.mean():.3f}")
print(f"x only predicts cultivar 2 for {line.fit(x, is2).predict(x).sum()} of {len(x)} wines; x and x^2 wins in {(b > a).sum()} of 50 folds")
X, y = wine.data.to_numpy(), wine.target.to_numpy()
noise = np.random.default_rng(0).normal(size=(len(X), 200))  # columns of pure noise
for m in [0, 10, 50, 200]:
    acc = cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier(5)), np.hstack([X, noise[:, :m]]), y, cv=cv)
    print(f"kNN on 13 measurements + {m} noise columns: {acc.mean():.3f}")

cultivar 2 vs the rest from flavanoids, 10 x 5 CV: x only 0.601, x and x^2 0.755; always 'other' 0.601
x only predicts cultivar 2 for 0 of 178 wines; x and x^2 wins in 49 of 50 folds
kNN on 13 measurements + 0 noise columns: 0.962
kNN on 13 measurements + 10 noise columns: 0.940
kNN on 13 measurements + 50 noise columns: 0.842
kNN on 13 measurements + 200 noise columns: 0.689