A metric and threshold for the cost of mistakes

In lesson 2.3 the curves showed a menu of thresholds, and in lesson 2.4 you checked when the model's numbers are real percentages. Now you will pick one threshold from that menu. Not by habit (0.5) and not for a favourite metric, but by what each mistake really costs.

Try it

• Keep the prices at 10 and 1 and move the threshold down from 0.5. Where is the cost lowest? How many misses and how many false alarms are there then?
• Set both costs to 1. Where is the cheapest threshold now? Compare it with the threshold with the best F1.
• Make a false alarm dearer than a miss. On which side of 0.5 does the cheapest threshold land?
• Go back to prices 10 and 1 and compare three costs: the cheapest threshold, the honest check and the theory threshold. Which choice is cheapest on biopsies that were not used to make it?

Once you have checked it in the lab, let's name what you see.

The idea in plain words

Every mistake has a price. A missed malignant tumour (FN) is a disease found too late. A false alarm (FP) is an unnecessary biopsy and stress. Once you know the prices, the cost of a threshold is a plain sum: misses times the price of a miss plus false alarms times the price of an alarm. The best threshold is the one with the smallest sum on validation data.

At prices 10 and 1, the threshold 0.5 gives 24 misses and 14 false alarms, a cost of 254. The cheapest threshold on validation is 0.06: 2 misses and 91 alarms, a cost of 111. That number is optimistic, though, because the threshold was picked and priced on the same scores (more in “Why it works”). For comparison, a biopsy for everyone costs 357, and no biopsy at all 2120.

If the model is calibrated, the threshold can also be computed without searching. A biopsy has a probability of malignancy p. An alarm then costs (1 − p) · 1 on average, and no alarm costs p · 10. An alarm pays off when p · 10 > (1 − p) · 1, that is when p > 1/11 ≈ 0.09. In general the threshold is c_FP / (c_FP + c_FN). This comes from decision theory, not from a measurement: it assumes the model's numbers are real percentages (lesson 2.4). This model is well calibrated (ECE 0.010). In the lab the threshold 0.09 costs 125, less than half the cost of the threshold 0.5, and nobody tuned it on these data.

Jak to policzono

• Biopsies: Breast Cancer Wisconsin (Diagnostic), UCI, licence CC BY 4.0. 569 biopsies (212 malignant, 357 benign), 10 “mean” measurements each.
• Scores from 5-fold cross-validation (StratifiedKFold(5) in scikit-learn): every biopsy is scored by a model that never saw it.
• Model: standardisation and logistic regression (C = 1). Thresholds from 0.01 to 0.99 in steps of 0.01.
• Prices of mistakes: a miss 10, a false alarm 1. This is an example, not a medical valuation.

Why it works

The lab and the theory almost agree: the cheapest threshold comes out at 0.06, theory says 0.09. The gap comes from 569 biopsies being few, while one miss weighs as much as 10 alarms, so the cost curve jumps. Its bottom is flat and ragged: the threshold 0.13 costs the same as 0.06 (111), and 0.09 costs 125. On 20 other cuts into folds (code below) the cheapest threshold lands between 0.04 and 0.13, 0.08 on average. Theory hits the middle of that range, and the lab shows how wide the bottom is.

The cheapest threshold has a catch, though. The lab tries 99 thresholds and picks the one that did best on these 569 biopsies, partly thanks to lucky hits. So the cost 111 is optimistic: it is the same winner's curse as in tuning (lesson 3.5). An honest check: pick the threshold on 4/5 of the biopsies and compute its cost on the other 1/5, five times. A threshold picked this way costs 141, more than the theory threshold 0.09 (125), which nobody tuned. On 20 other cuts the honest cost of the picked threshold is 106 to 153 (129 on average), while the theory threshold costs 105 to 136 (117 on average) and is cheaper in 16 of 20. The upshot: with a calibrated model the theory threshold is a strong, cheap default, and tuning the threshold on the same data overstates the savings.

The direction is always the same: the dearer a miss, the lower the threshold, and the dearer a false alarm, the higher it goes. At equal prices the cheapest threshold is 0.65 (cost 33). When an alarm costs more than a miss, the cheapest threshold lands above 0.5.

Common mistake

True or false: “F1 is a good metric for everything”. False. F1 = 2·TP / (2·TP + FN + FP), so F1 weighs a miss and a false alarm the same, even when their prices differ. In the lab the threshold with the best F1 is 0.65 (F1 0.918). At prices 10 and 1 it costs 276: more than the usual 0.5 (254) and almost 2.5 times the cheapest threshold (111). On 20 other cuts the best-F1 threshold costs 1.6 to 2.9 times the cheapest one.

Even at equal prices F1 is not the same as cost. In the lab, at prices 1 and 1, the cheapest threshold happens to fall where F1 is best (0.65), but that is not a rule. The cost at equal prices is simply the number of mistakes, while F1 is a fraction in which correct “no” answers (TN) do not count at all. On 20 other cuts the two thresholds differ in 3.

When you know the prices of mistakes, compute the cost. When you don't, ask for them before you choose a metric: choosing a metric is also a decision about prices, just an unspoken one.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_val_predict
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score

X, y = load_breast_cancer(return_X_y=True)
X, y = X[:, :10], 1 - y  # the 10 "mean" measurements, as in the lab; 1 = malignant
model = make_pipeline(StandardScaler(), LogisticRegression())
t = np.arange(1, 100) / 100  # thresholds 0.01 ... 0.99, as in the lab
def costs(p, i=slice(None), c_fn=10, c_fp=1):  # a miss costs 10, a false alarm 1; i = which biopsies
    return np.array([c_fn * ((p[i] < s) & (y[i] == 1)).sum() + c_fp * ((p[i] >= s) & (y[i] == 0)).sum() for s in t])
def scores(cv):  # every biopsy scored by a model that never saw it
    return cross_val_predict(model, X, y, cv=cv, method="predict_proba")[:, 1]
def held_out(p, cv):  # pick the cheapest threshold on 4/5 of the biopsies, price it on the other 1/5, five times
    return sum(costs(p, b)[costs(p, a).argmin()] for a, b in cv.split(X, y))
p = scores(StratifiedKFold(5))
c, f1 = costs(p), np.array([f1_score(y, p >= s) for s in t])
print(f"cheapest threshold {t[c.argmin()]:.2f} (cost {c.min()}); theory 1/11 = {1 / 11:.2f} (cost {c[8]}); 0.5 costs {c[49]}; best F1 at {t[f1.argmax()]:.2f} costs {c[f1.argmax()]}")
print(f"picked on 4/5, priced on the other 1/5: {held_out(p, StratifiedKFold(5))}; theory 1/11: {c[8]}")
r = []
for s in range(20):  # 20 other cuts into 5 parts
    cv = StratifiedKFold(5, shuffle=True, random_state=s)
    q = scores(cv)
    cq, fq = costs(q), np.array([f1_score(y, q >= u) for u in t])
    r.append((t[cq.argmin()], cq[fq.argmax()] / cq.min(), cq[49] / cq.min(), held_out(q, cv), cq[8], costs(q, c_fn=1).argmin() != fq.argmax()))
r = np.array(r)
print(f"20 other cuts: cheapest threshold {r[:, 0].min():.2f} to {r[:, 0].max():.2f} (mean {r[:, 0].mean():.2f}); best F1 costs {r[:, 1].min():.1f} to {r[:, 1].max():.1f} times the cheapest, 0.5 costs {r[:, 2].min():.1f} to {r[:, 2].max():.1f} times")
print(f"20 other cuts, held out: the picked threshold costs {r[:, 3].min():.0f} to {r[:, 3].max():.0f} (mean {r[:, 3].mean():.0f}), theory 1/11 costs {r[:, 4].min():.0f} to {r[:, 4].max():.0f} (mean {r[:, 4].mean():.0f}); theory is cheaper in {(r[:, 4] < r[:, 3]).sum():.0f} of 20; at equal prices the cheapest and the best-F1 threshold differ in {r[:, 5].sum():.0f} of 20")

cheapest threshold 0.06 (cost 111); theory 1/11 = 0.09 (cost 125); 0.5 costs 254; best F1 at 0.65 costs 276
picked on 4/5, priced on the other 1/5: 141; theory 1/11: 125
20 other cuts: cheapest threshold 0.04 to 0.13 (mean 0.08); best F1 costs 1.6 to 2.9 times the cheapest, 0.5 costs 2.0 to 2.5 times
20 other cuts, held out: the picked threshold costs 106 to 153 (mean 129), theory 1/11 costs 105 to 136 (mean 117); theory is cheaper in 16 of 20; at equal prices the cheapest and the best-F1 threshold differ in 3 of 20