Classification: drawing a boundary

A biologist in Antarctica measures a penguin's bill and flippers. Which species is it? This is a different kind of question from "how much will the disease progress?". The answer is not a number on a scale but a choice between categories. Can the straight lines and downhill steps of the previous lessons handle that? They can, with one clever twist.

Try this

• Look at the boundary first. Find the penguins that sit on the wrong side of it: these are the cases the model gets wrong even with its best line.
• Move the probability threshold slider towards one end and watch the boundary shift towards one species. Notice that the line moves but keeps its direction.
• Set the threshold near one extreme and count how the mistakes change: one species is now almost never missed, at the cost of the other.
• Find penguins close to the boundary and compare them with penguins far away. Which ones would you double-check by hand?

Now that you've tried it, let's see what was actually going on.

The idea in plain words

Predicting a category is called classification. In this lesson we use the Palmer Penguins dataset: 344 penguins of three species from islands near Palmer Station in Antarctica, each with bill, flipper and body measurements. To keep things flat, we take two species and two measurements, so every penguin is a point on a plane — one measurement across, the other up — with a known species.

A classifier has to split that plane into two regions: "species A here, species B there". The line between the regions is the decision boundary. Logistic regression, despite its name, is a classification model, and its boundary is a straight line.

The twist is that logistic regression does not answer just "A" or "B". It answers with a probability: "this penguin is species B with probability 0.9". Think of a weather forecast. The forecaster says "80% chance of rain", and you decide whether that is enough to take an umbrella. The model plays the forecaster; the threshold is your umbrella rule.

Why it works

First the model computes a score just like linear regression does: a weighted sum of the features plus an intercept, z = w₁x₁ + w₂x₂ + b. That score can be any number, so it is squeezed into the range from 0 to 1 by the sigmoid function:

p = 1 / (1 + e^(−z))

A large positive score gives a probability close to 1, a large negative score gives a probability close to 0, and a score of exactly zero gives 0.5. Penguins far from the boundary get confident probabilities; penguins near it get probabilities close to one half — the model is unsure, and it says so.

The weights are learned with gradient descent, as in lesson 4, but with a different loss. Instead of squared error the model uses log loss (also called cross-entropy). It barely punishes a confident correct answer and punishes a confident wrong answer very hard. This pushes the model to be sure only when the data justify it.

Finally, the threshold turns the probability into a decision: probability above the threshold → species B, below → species A. The usual default is 0.5, but nothing forces it. Moving the threshold moves the boundary. Because the probability depends on the weighted sum, every threshold corresponds to a line with the same direction, just shifted: the model's opinion about which way the species differ stays the same, only your demand for certainty changes.

Common mistake

Two errors are common. The first is treating 0.5 as a law. The threshold is a decision about costs, not part of the model; in medicine you may accept many false alarms to miss fewer cancers. The second is reading the probability as certainty. A probability of 0.97 means the model expects to be wrong in about three such cases out of a hundred — and only if the new data resemble the training data. A penguin from a population the model never saw can get a confident and wrong answer.

A boundary that gets most penguins right sounds good, but "most" hides a lot: which mistakes, how many, and how bad? In the next lesson we open the box of mistakes and learn to count them properly.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = pd.read_csv("data/raw/penguins.csv")
df = df[df["species"].isin(["Adelie", "Chinstrap"])].dropna(subset=["bill_length_mm", "bill_depth_mm"])
X, y = df[["bill_length_mm", "bill_depth_mm"]], df["species"] == "Chinstrap"
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
p = model.predict_proba(X)[:, 1]  # probability of Chinstrap
for t in [0.1, 0.3, 0.5, 0.9]:
    wrong = int(((p >= t) != y).sum())  # call Chinstrap when p >= t
    print(f"threshold {t}: {wrong} mistakes, accuracy {1 - wrong / len(y):.1%}")

threshold 0.1: 21 mistakes, accuracy 90.4%
threshold 0.3: 3 mistakes, accuracy 98.6%
threshold 0.5: 7 mistakes, accuracy 96.8%
threshold 0.9: 17 mistakes, accuracy 92.2%