Evaluating a classifier: beyond accuracy

A smoke alarm that never makes a sound is never wrong about a false fire. A smoke alarm that screams every time you make toast never misses a real one. Which is the better alarm? Neither — and a single accuracy number would not even tell you which one you have. This lesson is about counting mistakes in a way that shows their kind, not just their number.

Try this

• Move the threshold slider to the left. Watch the false negatives in the confusion matrix shrink and the false positives grow, and see recall and precision move in opposite directions.
• Move the threshold slider to the right and watch the reverse happen. Keep an eye on accuracy: does it tell you which kind of mistake you are making?
• Find the highest threshold at which no malignant tumour is missed. Read how many benign tumours are falsely flagged at that point.
• Pick the threshold you would choose for a screening test, and say in one sentence why.

Now that you've tried it, let's see what was actually going on.

The idea in plain words

Back to the 569 breast tumours from lesson 1. A model gives each tumour a score: the higher the score, the more likely it is malignant. A threshold turns the score into a decision. Every decision then lands in one of four boxes:

This table is the confusion matrix. The diagonal holds the correct decisions. The two other cells hold two very different kinds of error: a false positive is a false alarm (a healthy patient sent for more tests), a false negative is a miss (a cancer sent home).

The smoke alarm makes the point. The silent alarm has no false positives and plenty of false negatives. The toast alarm has the reverse. A good alarm balances the two according to what each mistake costs.

Why it works

Two numbers summarise the matrix far better than accuracy does:

precision = TP / (TP + FP) recall = TP / (TP + FN)

Recall answers: of all the malignant tumours, what share did we catch? Precision answers: of all the tumours we called malignant, what share really were? Recall is about not missing; precision is about not crying wolf.

The threshold controls the trade-off between them. Lower the threshold and more tumours are flagged as malignant: you catch more cancers (recall rises) but also flag more benign tumours (precision usually falls). Raise it and the opposite happens. No threshold can maximise both, unless the model separates the classes perfectly. Choosing a threshold is therefore not a technical detail but a decision about values: how many false alarms are you willing to accept to avoid one missed cancer?

Accuracy blends all four cells into one number and hides this choice. It also hides class imbalance. Since 357 of the 569 tumours are benign, a model that calls everything benign gets nearly two thirds right while catching zero cancers — recall of zero. This is the accuracy paradox: a model can look decent by accuracy and be useless for the purpose it was built for.

Common mistake

The most common mistake is reporting accuracy alone, especially on imbalanced data where one class is much rarer than the other. The second is keeping the default threshold of 0.5 without asking what the errors cost. A screening test, where a miss can be fatal and a false alarm means an extra examination, should usually have a lower threshold than a test that triggers surgery. A third trap is mixing up precision and recall; remembering the questions they answer — "of those I flagged" versus "of those that exist" — helps more than memorising formulas.

So far we have scored models on one fixed test set. But a model can do brilliantly on the data it learned from and poorly on everything else. In the next lesson we make that happen on purpose: overfitting.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, b = load_breast_cancer(return_X_y=True, as_frame=True)
X = X.filter(like="mean ")  # the 10 "mean" features, as in the lab
y = 1 - b  # in sklearn 0 = malignant; flip so that 1 = malignant
print(f"tumours {len(y)}, benign {(y == 0).sum()}, always-benign accuracy {(y == 0).mean():.3f}")
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
score = make_pipeline(StandardScaler(), LogisticRegression()).fit(Xtr, ytr).predict_proba(Xte)[:, 1]
for t in (0.5, 0.1):
    flag = score >= t
    tp, fp, fn = (flag & (yte == 1)).sum(), (flag & (yte == 0)).sum(), (~flag & (yte == 1)).sum()
    print(f"t={t}: TP {tp} FN {fn} FP {fp} precision {tp / (tp + fp):.3f} recall {tp / (tp + fn):.3f}")

tumours 569, benign 357, always-benign accuracy 0.627
t=0.5: TP 53 FN 11 FP 2 precision 0.964 recall 0.828
t=0.1: TP 61 FN 3 FP 13 precision 0.824 recall 0.953