Naive Bayes: a false assumption, good decisions

A doctor sees fever and a cough. How strongly does that point to flu? Naive Bayes answers like this: take how common flu is in general (that is the prior), multiply by how typical a fever is in people with flu, and by how typical a cough is in them. Do the same for other illnesses and pick the largest product. The model is “naive” because it multiplies the symptoms as if, within each illness, they were independent of each other. In real data they almost never are.

Try it

• Start with the sepals. How are the species' ellipses laid out, and where is the probe when the model hesitates?
• Switch to the petals. In which species do the measurements correlate most strongly, and how much does the model get right on validation?
• Go to the tumours. Compare radius alone with the trio radius, perimeter, area. What happens to the confidence, and what to the accuracy?
• With all 10 measurements, count how many calibration tumours get a confidence above 99%.

Once you have checked it in the lab, let's name what you see.

The idea in plain words

For each species naive Bayes keeps only two numbers per measurement: the mean and the spread (variance). From them it makes a bell curve: how typical a given petal length is for setosa, for versicolor, for virginica. A new flower gets a product for each species: prior × density of the length × density of the width. The largest one wins. When you divide each product by the sum of the three, you get the species' probabilities, the posterior.

Independence is an assumption about shape: the species' ellipses lie along the axes, not at a slant. On the petals that is false, in versicolor r = 0.80. And yet the model gets 97.1% right on validation and 93.3% on the test. A decision does not need exact products, only that the right species has the largest one. On the petals the species are separated well enough that even crookedly drawn ellipses usually point the right way.

The percentages are worse off. On breast tumours radius, perimeter and area are almost one measurement: radius correlates with perimeter at r = 0.998 and with area at r = 0.987. Naive Bayes multiplies them as three independent pieces of evidence, so it counts the same evidence three times. On the 132 calibration biopsies of lesson 2.4 radius alone gives an accuracy of 88.6% and a mean declared confidence of 89.2%. Radius, perimeter and area together: accuracy 86.4%, confidence 96.5%. So three copies raise the confidence, not the accuracy. All 10 measurements also bring new information, such as texture and smoothness: accuracy rises to 93.9%, but confidence (98.5%) still exceeds it.

Jak to policzono

• Irises: Iris (Fisher, 1936; UCI, CC BY 4.0 licence), 150 flowers, 3 species. Stratified split: 105 flowers to learn from, 45 for the test. Validation: 5-fold cross-validation inside the 105 flowers.
• Tumours: Breast Cancer Wisconsin (Diagnostic) (Wolberg et al., 1993; UCI, CC BY 4.0 licence), 569 biopsies, 10 “mean” measurements. Split and names as in lesson 2.4: 266 biopsies to learn from, 132 calibration biopsies (the lab compares the sets of measurements on them) and 171 validation biopsies, only to look at.
• Model: naive Bayes with the normal distribution, like scikit-learn's GaussianNB; the lab's numbers match it.

Why it works

A classifier is wrong only when a wrong species gets the largest product. Domingos and Pazzani (1997) showed that by this measure naive Bayes can be the best possible even with strongly dependent features: what counts is the order of the products, not their values. The assumption does have a price. A model that accounts for correlation (tilted ellipses, called quadratic discriminant analysis, QDA) gets 97.1% right on the petals on average over 20 other splits, against 95.8% for naive Bayes. It is better in 11 splits and worse in none. On the sepals naive Bayes does better on average: 80.0% against 78.8%. The usual explanation is that it has fewer numbers to estimate from 105 flowers.

The overconfidence is not a quirk of one split. On 20 other splits of the tumours, radius alone gives a difference between confidence and accuracy from −3.6 to 2.2 points. Radius, perimeter and area together give more confidence than accuracy every time: from 4.9 to 11.5 points. All 10 measurements: from 4.7 to 10.8 points.

A common mistake

True or false: “a false assumption means a useless model”. False. On the petals the independence assumption is clearly false, and the model gets 93.3% of the test flowers right. In the code below it gets 0.960 in cross-validation, almost as much as the model with correlations (0.973). What the false assumption spoils is the percentages: with all 10 tumour measurements the model declares 0.990 and is right 0.906 of the time.

The practical conclusion: trust naive Bayes's decisions as far as validation shows. Do not trust its percentages without calibrating on separate data (lesson 2.4).

from sklearn.datasets import load_iris, load_breast_cancer
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

iris = load_iris(as_frame=True)
for part in ["sepal", "petal"]:
    cols = [f"{part} length (cm)", f"{part} width (cm)"]
    r = iris.data[iris.target == 1][cols].corr().iloc[0, 1]  # correlation within versicolor
    nb = cross_val_score(GaussianNB(), iris.data[cols], iris.target, cv=10).mean()
    qda = cross_val_score(QuadraticDiscriminantAnalysis(), iris.data[cols], iris.target, cv=10).mean()
    print(f"{part}s: versicolor r = {r:.2f}; 10-fold CV accuracy: naive Bayes {nb:.3f}, with correlations (QDA) {qda:.3f}")
bc = load_breast_cancer(as_frame=True)
X, y = bc.data.iloc[:, :10], 1 - bc.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
for cols in [["mean radius"], ["mean radius", "mean perimeter", "mean area"], list(X.columns)]:
    p = GaussianNB().fit(X_tr[cols], y_tr).predict_proba(X_te[cols])
    acc, conf = (p.argmax(1) == y_te).mean(), p.max(1).mean()
    print(f"{len(cols):2d} of 10 measurements: accuracy {acc:.3f}, mean declared confidence {conf:.3f}, over 99% sure: {(p.max(1) > 0.99).sum()} of {len(y_te)}")

sepals: versicolor r = 0.53; 10-fold CV accuracy: naive Bayes 0.793, with correlations (QDA) 0.773
petals: versicolor r = 0.79; 10-fold CV accuracy: naive Bayes 0.960, with correlations (QDA) 0.973
 1 of 10 measurements: accuracy 0.883, mean declared confidence 0.884, over 99% sure: 26 of 171
 3 of 10 measurements: accuracy 0.895, mean declared confidence 0.969, over 99% sure: 127 of 171
10 of 10 measurements: accuracy 0.906, mean declared confidence 0.990, over 99% sure: 150 of 171