Thresholds, ROC and PR: every threshold at once

In lesson 2.2 you moved one threshold and watched the confusion matrix change. Every threshold is a different decision, although the model’s scores stay the same. Today you will see every threshold at once, on two curves: ROC and precision–recall (PR). You will also check which of them tells the truth when the sick are few. The model is weak on purpose: a logistic regression that sees only one measurement, the mean texture of the tumour, so the curves do not stick to the corners.

Try it

• Start with 212 malignant and threshold 0.5. Read recall and FPR, then move the threshold to 0.3. Which way does the point go on both curves?
• Switch to 19 malignant at threshold 0.3. There are as many false alarms as a moment ago. What happened to precision?
• Compare ROC AUC and AP at 212, 40 and 19 malignant.

Once you have checked it in the lab, let’s name what you see.

The idea in plain words

The model does not say “malignant” or “benign”. It gives a score from 0 to 1, and a threshold makes the decision. Each threshold gives a different pair of numbers: how many malignant biopsies were found (recall) and how many benign ones got a false alarm. The ROC curve draws these pairs for every threshold at once: the share of false alarms among benign biopsies (FPR) across, recall up. A high threshold is the bottom left corner (few alarms), a low one is the top right (an alarm for every biopsy).

The precision–recall (PR) curve shows the same thresholds differently: recall across, precision up, that is, the share of alarms that are right. The area under the ROC curve is ROC AUC, the area under the PR curve is AP (average precision). Each of them sums up every threshold at once, so it does not depend on the threshold you pick.

On all 569 biopsies the model has ROC AUC 0.773 and AP 0.591. Random guessing would give ROC AUC 0.5 and an AP equal to the share of malignant biopsies, 0.373. The model is clearly better than guessing, but far from the ideal, which is 1.

Jak to policzono

• Biopsies: Breast Cancer Wisconsin (Diagnostic), UCI, licence CC BY 4.0. 569 biopsies (357 benign, 212 malignant). The model sees one measurement: the mean texture of the tumour.
• Scores from 5-fold cross-validation (StratifiedKFold(5) in scikit-learn): every biopsy is scored by a model that never saw it.
• Rarer class: all 357 benign stay, with 40 or 19 malignant drawn at random. The chart shows one draw; the means are over 200 draws.
• Model: standardization and logistic regression (C = 1).

Why it works

ROC counts false alarms against the benign biopsies: FPR is the false alarms divided by 357. Recall counts hits against the malignant ones. When you take malignant biopsies away at random, both fractions stay roughly the same, so the ROC curve and the area under it hardly move. At threshold 0.3 the model raises an alarm for 144 of 357 benign biopsies (FPR 40.3%), whether there are 212 malignant or 19.

Precision counts false alarms against all alarms. With 212 malignant and threshold 0.3 the model finds 177, so precision is 55.1%. With 19 malignant it finds 17, and the 144 false alarms stay: precision falls to 10.6%, that is 8.5 false alarms per correct one. The PR curve falls with precision, and AP from 0.591 to 0.137 on average. The random-guessing line on PR falls too, to the share of malignant biopsies: 0.051.

This is not luck of one draw or one way of cutting the data into parts. With 40 malignant (10%) ROC AUC is 0.773 on average and AP 0.232. In the code below the same model on 20 other cuts into parts has, with 19 malignant, an AP below half of its full-data AP every single time.

Common mistake

True or false: “a high ROC AUC means few false alarms”. False. ROC AUC measures how well the model sorts: the probability that a random malignant biopsy gets a higher score than a random benign one. It does not say how many alarms will be false, because that depends on how rare the class is. In the lab, with 19 malignant ROC AUC is 0.772 on average, nearly as much as with 212, and at threshold 0.3 there are 8.5 false alarms per correct one. When a class is rare, look at the PR curve and at the precision at the threshold you will really use.

A warning the other way: AP is not a number “about the model alone” either. It depends on the share of the class in the data, so the APs of two datasets with different proportions cannot be compared directly. Compare AP with the random-guessing line on the same data.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_val_predict
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score

X, y = load_breast_cancer(return_X_y=True)
X, y = X[:, [1]], 1 - y  # mean texture only, as in the lab; 1 = malignant
model = make_pipeline(StandardScaler(), LogisticRegression())
def scores(cv):  # validation scores: every biopsy is scored by a model that never saw it
    return cross_val_predict(model, X, y, cv=cv, method="predict_proba")[:, 1]
ben, mal = np.flatnonzero(y == 0), np.flatnonzero(y == 1)
def rare(p, keep, draws, seed):  # all 357 benign + `keep` random malignant, mean over the draws
    rng = np.random.default_rng(seed)
    rows = [np.r_[ben, rng.choice(mal, keep, replace=False)] for _ in range(draws)]
    return np.mean([(roc_auc_score(y[i], p[i]), average_precision_score(y[i], p[i])) for i in rows], axis=0)
p = scores(StratifiedKFold(5))
print(f"212 malignant ({212 / 569:.0%}): ROC AUC {roc_auc_score(y, p):.3f}, AP {average_precision_score(y, p):.3f}")
for keep in (40, 19):
    auc, ap = rare(p, keep, 1000, 0)
    print(f"{keep} malignant ({keep / (357 + keep):.0%}), mean of 1000 draws: ROC AUC {auc:.3f}, AP {ap:.3f}")
r = []
for s in range(20):  # 20 other ways to cut the data into 5 parts
    q = scores(StratifiedKFold(5, shuffle=True, random_state=s))
    r.append([roc_auc_score(y, q), average_precision_score(y, q), *rare(q, 19, 200, s)])
r = np.array(r)
print(f"20 other fold splits, 212 -> 19 malignant: ROC AUC {r[:, 0].mean():.3f} -> {r[:, 2].mean():.3f}, AP {r[:, 1].mean():.3f} -> {r[:, 3].mean():.3f}; AP below half in {(r[:, 3] < r[:, 1] / 2).sum()} of 20")

212 malignant (37%): ROC AUC 0.773, AP 0.591
40 malignant (10%), mean of 1000 draws: ROC AUC 0.773, AP 0.231
19 malignant (5%), mean of 1000 draws: ROC AUC 0.774, AP 0.140
20 other fold splits, 212 -> 19 malignant: ROC AUC 0.774 -> 0.775, AP 0.595 -> 0.140; AP below half in 20 of 20