Calibration: does 0.8 mean 80%?

In lesson 2.3 the ROC and PR curves checked how well a model sorts biopsies. They never asked whether the model's numbers are true. The model says “0.8”. Does that mean 80% of such biopsies are malignant? That is the question of calibration. Today you will check it on two models and see that sometimes one number fixes it, without changing a single decision.

Try it

• Start with naive Bayes without calibration. Where are the circles compared with the diagonal? How many biopsies get a confidence above 99%, and how many of them are mistakes?
• Switch the temperature on. What happens to the circles, to the Brier score and to ROC AUC? How many decisions changed?
• Switch to logistic regression and compare it without calibration and with temperature. Did this model need a fix?

Once you have checked it in the lab, let's name what you see.

The idea in plain words

A model is calibrated when you can take its numbers literally: among the biopsies it gives 0.8, about 80% are malignant, and among those with 0.1 about 10%. A reliability diagram checks this. The biopsies are split into bands by the declared probability, and in each band we count what share really is malignant. A calibrated model lies on the diagonal. Circles below the diagonal on the right mean overconfidence.

Calibration is not the same as sorting. ROC AUC from lesson 2.3 only asks whether a malignant biopsy gets a higher score than a benign one. Naive Bayes sorts well (ROC AUC 0.966, almost like logistic regression with 0.969), but its percentages are inflated. Among the 55 biopsies it gives at least 0.9, it declares 99.6% on average, and 89.1% are malignant.

The Brier score measures both at once: the mean squared difference between the declared probability and what happened (1 for malignant, 0 for benign). Lower is better. ECE says how far, on average, the circles lie from the diagonal.

Jak to policzono

• Biopsies: Breast Cancer Wisconsin (Diagnostic), UCI, licence CC BY 4.0. 569 biopsies, 10 “mean” measurements each.
• Stratified split: 171 validation biopsies (30%); of the rest, 266 to learn from and 132 to fit the calibration on. Neither the model nor the calibration sees the validation biopsies.
• Models: naive Bayes with a normal distribution (GaussianNB), and standardisation plus logistic regression (C = 1).
• Temperature: one number T, fitted on the 132 calibration biopsies so that the log-loss is smallest. Diagram and ECE: 10 bands of 0.1.

Why it works

Naive Bayes assumes the measurements of a tumour are independent of each other. In these data they are not: the radius, perimeter and area of a tumour say almost the same thing. So the model counts the same evidence several times and keeps adding it to the score until its confidence reaches 99% and more. The order of the biopsies stays sensible, because a bigger tumour still gets a higher score, but the percentages grow too fast.

Temperature divides the model's score before the sigmoid by one number T. For naive Bayes T = 4.12, so every score shrinks more than four times. Dividing by a positive number changes neither the sign of a score nor the order of the biopsies. No decision at threshold 0.5 changes, and ROC AUC stays 0.966. Only the percentages change: the mean confidence falls from 98.3% to 91.1%, the Brier score from 0.100 to 0.071, and ECE from 0.101 to 0.055. Now 34 biopsies get a confidence above 99%, and none of them is a mistake.

Logistic regression is calibrated almost from the start: it declares 92.5% and is right 91.2% of the time. Its temperature comes out at T = 0.93, close to 1, and the Brier score stays 0.063. That is no accident: logistic regression learns by lowering the log-loss, which is exactly the penalty for badly stated percentages.

This is not a fluke of one split. In the code below the same experiment runs on 50 other splits: naive Bayes declares 98.5% on average and is right 91.2% of the time, and temperature lowers its Brier score every single time. For logistic regression temperature changes almost nothing.

Common mistake

True or false: “a score of 0.8 from a model means an 80% chance”. False, until you check it. A model gives a number from 0 to 1, but nothing guarantees it is a probability in the everyday sense. In the lab naive Bayes gives 149 of the 171 biopsies a confidence above 99%, and 5 of them are mistakes. If those percentages were true, there would be fewer than two mistakes among them. Read a model's score as a chance only after checking the reliability diagram on data the model has not seen.

A warning the other way: calibration does not improve sorting. Temperature changes neither ROC AUC nor accuracy, and it gives a well-calibrated model nothing: the Brier score of logistic regression stays 0.063. You need calibration when the model's numbers go further: into a threshold set by the cost of mistakes (lesson 2.6), into adding up risk, into a conversation with a patient.

import numpy as np
from scipy.optimize import minimize_scalar
from scipy.special import expit
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss

X, y = load_breast_cancer(return_X_y=True)
X, y = X[:, :10], 1 - y  # the 10 "mean" measurements, as in the lab; 1 = malignant
def score(m, rows):  # the model's score before the sigmoid: log P(malignant) - log P(benign)
    lp = m.predict_log_proba(X[rows])
    return lp[:, 1] - lp[:, 0]
def run(model, seed):  # 266 to learn from, 132 to calibrate on, 171 to judge (validation)
    rest, va = train_test_split(np.arange(len(y)), test_size=0.3, stratify=y, random_state=seed)
    tr, ca = train_test_split(rest, test_size=1 / 3, stratify=y[rest], random_state=seed)
    m = model.fit(X[tr], y[tr])
    zc, zv = score(m, ca), score(m, va)
    loss = lambda T: np.mean(np.logaddexp(0, -zc / T) * y[ca] + np.logaddexp(0, zc / T) * (1 - y[ca]))
    T = minimize_scalar(loss, bounds=(0.05, 20), method="bounded").x  # temperature: one number, fitted on the calibration part
    p0, p1 = expit(zv), expit(zv / T)
    conf, acc = np.maximum(p0, 1 - p0).mean(), ((p0 >= 0.5) == y[va]).mean()
    same = ((p0 >= 0.5) == (p1 >= 0.5)).all()  # dividing by T never moves a decision
    return conf, acc, brier_score_loss(y[va], p0), brier_score_loss(y[va], p1), T, same
for name, model in (("naive Bayes", GaussianNB()), ("logistic regression", make_pipeline(StandardScaler(), LogisticRegression()))):
    r = np.array([run(model, s) for s in range(50)])
    print(f"{name}, 50 splits: declares {r[:, 0].mean():.1%}, accuracy {r[:, 1].mean():.1%}; Brier {r[:, 2].mean():.3f} -> {r[:, 3].mean():.3f} with T = {r[:, 4].mean():.2f}; lower in {(r[:, 3] < r[:, 2]).sum()} of 50, decisions kept in {int(r[:, 5].sum())} of 50")

naive Bayes, 50 splits: declares 98.5%, accuracy 91.2%; Brier 0.077 -> 0.061 with T = 3.86; lower in 50 of 50, decisions kept in 50 of 50
logistic regression, 50 splits: declares 93.4%, accuracy 93.5%; Brier 0.047 -> 0.047 with T = 0.94; lower in 17 of 50, decisions kept in 50 of 50