The whole process in 15 minutes, start to finish

Before you learn the parts, see the whole game. That is how people learn football: first a match, then passing drills. In this lesson you walk through an entire machine learning project in a quarter of an hour, on one dataset: penguins of three species, measured by researchers on islands off the coast of Antarctica. Every later module of the course opens one of the six steps you see here all at once.

Try this

• Set 0 questions. That is guessing without a model: how many penguins does “always Adelie” fit?
• Move to 1 and then 2 questions. Read the tree’s questions under the chart: which measurements does it ask about?
• With 2 questions, move the split slider. Is there a split where the lead over guessing disappears?
• Go back to split 1 and set 3 questions. Does an extra question always help?

Once you have checked it in the lab, let us name the steps you just went through.

The idea in plain words

Every machine learning project, from penguins to fraud detection in a bank, goes through the same six steps:

1. Question. What exactly are you predicting, and from what? Here: a penguin’s species from four body measurements. A good question can be checked, because we know the true species.
2. Data. The file has 344 penguins, but 2 have no measurements at all, which leaves 342. Sex is missing in 11 rows. Dropping every row with any gap would leave 333, but we do not use sex here, so we lose only those 2.
3. Split. 103 penguins, that is 30%, are hidden from the model. The model learns from the other 239, and we test it only on the hidden ones. The part to learn from is the training set, the hidden part is the test set.
4. Baseline. Before you build a model, count what an answer without thinking gets. Among the learning penguins Adelie is the most common, so “always Adelie” gets 45 of the 103 test penguins right: 43.7%.
5. Model. A decision tree is a sequence of yes/no questions. With two questions it first asks about the flipper (up to 207 mm or longer), then about the bill’s length or depth.
6. Evaluation. The tree gets 98 of 103 right, that is 95.1%: 51.5 points more than guessing.

The later modules of the course open these steps one by one: how a model learns from examples and what to compare it with (module 1), how to measure a result (module 2), whether the result survives new data (module 3), what breaks data (module 4), which models to choose (module 5) and what happens to a model out in the world (module 6).

Jak to policzono

• Data: Palmer Penguins (Gorman, Williams and Fraser, 2014; the palmerpenguins package: Horst, Hill and Gorman, 2020), licence CC0. 344 penguins in the file, 342 with all four measurements.
• Split: 70/30 with the same species mix in both parts; split 1 is the same split as train_test_split(test_size=0.3, stratify=y, random_state=1) in scikit-learn.
• Model: a decision tree (CART, Gini index), at most 1, 2 or 3 questions on every path.
• Baseline: the most common species among the learning penguins.

Why it works

The score on the hidden penguins means something because the model has not seen them. Tested on the learning penguins, it could simply remember the answers. That is why the tree with three questions scores 97.5% on the learning penguins but only 93.2% on the hidden ones, less than the tree with two questions. The third question started to learn individual penguins instead of a rule. That is overfitting, the subject of module 3.

The tree does so well because the species really are built differently. Gentoo have long flippers, so the first question nearly separates them: among learning penguins with a flipper longer than 207 mm, 85 of 91 are Gentoo. Among the shorter flippers a long bill gives away a Chinstrap and a short one an Adelie: in the group with a flipper up to 207 mm and a bill up to 43.4 mm, 101 of 103 learning penguins are Adelie.

A baseline is needed because accuracy depends on the task. With three species, rolling a die would be right one time in three, and “always the most common species” is right 43.7% of the time. A model makes sense only when it clearly beats that simple answer.

One split is one draw. Over 20 splits in the lab, two questions get between 94 and 100 of the 103 penguins right (from 91.3% to 97.1%), and guessing gets 43.7% every time. The lead stays, even though the exact number changes.

Common mistake

True or false: “95% accuracy is a great result”. False, or rather: the number alone does not say. Accuracy depends on the class proportions and on how easy the task is. Here guessing gets 43.7%, so 95.1% is a huge improvement. But when 95 of 100 emails are ordinary messages, a filter that does nothing and lets every email through also has 95% accuracy and catches no spam at all. First the baseline, then the excitement.

Where the analogy breaks: a football match has an end, and a project rarely runs in a straight line. After the evaluation you often go back to the question or to the data, because the score showed what is missing.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

df = pd.read_csv("data/raw/penguins.csv")
cols = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g"]
d = df.dropna(subset=cols)  # only penguins with all measurements
print(f"penguins {len(df)}, with measurements {len(d)}, without any gap {len(df.dropna())}")
X, y = d[cols], d["species"]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
guess = ytr.value_counts().idxmax()  # baseline: the most common species among the learning penguins
print(f"train {len(Xtr)}, test {len(Xte)}; always {guess}: {(yte == guess).sum()} of {len(yte)} = {(yte == guess).mean():.3f}")
for depth in (1, 2, 3):
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(Xtr, ytr)
    print(f"depth {depth}: {(tree.predict(Xte) == yte).sum()} of {len(yte)} = {tree.score(Xte, yte):.3f} (train {tree.score(Xtr, ytr):.3f})")
scores = []
for seed in range(100):  # 100 different random splits
    a, b, c, e = train_test_split(X, y, test_size=0.3, stratify=y, random_state=seed)
    scores.append(DecisionTreeClassifier(max_depth=2, random_state=0).fit(a, c).score(b, e))
print(f"depth 2 on 100 splits: mean {sum(scores) / 100:.3f}, min {min(scores):.3f}, max {max(scores):.3f}")

penguins 344, with measurements 342, without any gap 333
train 239, test 103; always Adelie: 45 of 103 = 0.437
depth 1: 81 of 103 = 0.786 (train 0.791)
depth 2: 98 of 103 = 0.951 (train 0.967)
depth 3: 96 of 103 = 0.932 (train 0.975)
depth 2 on 100 splits: mean 0.949, min 0.883, max 0.990