ML Atlas

Machine learning from zero · Lesson 2.5 · 12 of 33 lessons ready · 12 min

Imbalanced classes

In short

When one class is rare, accuracy hides the losses and the model learns to say “the common class”. Recall, balanced accuracy and class weights show and patch it.

Warm-up Two questions from earlier lessons. A minute, and it never blocks the lesson.
From lesson 2.4: Calibration: does 0.8 mean 80%? Naive Bayes has ROC AUC 0.966 but declares 98.3% with an accuracy of 88.3%. What follows?

Show the answer

Correct answer: It sorts the biopsies well, but its percentages are inflated

ROC AUC measures order, calibration measures percentages. They are two different things: temperature fixes the percentages of naive Bayes, and ROC AUC stays 0.966.

From lesson 2.3: Thresholds, ROC and PR: every threshold at once When does the PR curve tell you more than the ROC curve?

Show the answer

Correct answer: When the positive class is rare and what counts is how many alarms are right

With a rare class the ROC curve looks the same while precision falls. In the lab with 19 malignant, AP falls to 0.137 on average although ROC AUC hardly moves.

In lesson 1.3 “always benign” scored a high accuracy and found no one. There the rare class was only in the test. Today it is rare in the learning data, as in real screening, fraud or failures. You will see what that does to a model, how to measure it and what can be fixed without new data.

Predict, then check A logistic regression learns from biopsies. The learning data keeps all benign ones, but only 8 malignant instead of 159. On the same validation part (set aside for checking the model, not for learning), which falls more: accuracy or recall?
How sure are you?

Interactive lab

This interactive lab runs in a browser with JavaScript enabled. The text around it describes what the lab shows.

Show the answer

Correct answer: Recall, by more than 20 points, while accuracy falls by only about 6

Recall falls from 83.0% to 61.5% (mean of 5 draws of which malignant ones stay), while accuracy falls only from 91.5% to 85.6%. In validation 89 of the 142 biopsies are benign, and the model recognises those more and more confidently, so accuracy hardly sees the loss.

Try it

  • Without weights, move the slider from 159 to 8 malignant. Which line falls faster: recall or accuracy?
  • At 8 malignant, switch on class weights. How much recall comes back, and how much is still missing compared with all the data?
  • Look at the mean predicted P(malignant). How does it change as malignant biopsies disappear?

Once you have checked it in the lab, let’s name what you see.

The idea in plain words

Classes are imbalanced when one is much rarer than the other. The model then learns two things at once: what malignant biopsies look like and how rarely they happen. The second lesson works like a prejudice: when the measurements are unclear, the model chooses “benign” more and more often. The mean predicted P(malignant) falls from 0.36 to 0.24 when 8 malignant biopsies stay in learning instead of 159.

Accuracy hardly sees it. Recall falls from 83.0% to 61.5%, accuracy only from 91.5% to 85.6%. That is why with a rare class you report the recall of the rare class or the balanced accuracy, the mean of the recalls of both classes. “Always benign” has a balanced accuracy of 0.5, whatever the proportions.

The first fix is simple: class weights. A mistake on the rare class costs as many times more in learning as the class is rarer (class_weight='balanced'). With 8 malignant, weights raise recall from 61.5% to 80.0%, and accuracy even rises, to 91.4%. Compared with all the data with weights (86.8%) something is still missing: weights change the proportions, they do not add examples.

Jak to policzono
  • Biopsies: Breast Cancer Wisconsin (Diagnostic), UCI, licence CC BY 4.0. 569 biopsies, 10 “mean” measurements.
  • Validation part: 25% of the biopsies, stratified (142: 53 malignant, 89 benign), the same in every setting. Learning: 268 benign and from 159 down to 8 malignant; below 159 it is the mean of 5 draws of which malignant ones stay.
  • Model: standardization on the learning data and logistic regression (C = 1), with or without class weights; threshold 0.5.

Why it works

Logistic regression looks for the model that makes the smallest mistakes on average on the learning data. With 268 benign and 8 malignant, the malignant biopsies are only 2.9% of the total penalty. The model moves the boundary towards the malignant side: it gives up a few malignant biopsies so as to get benign ones wrong less often. Class weights even out the penalty: the 8 malignant together weigh as much as the 268 benign.

One split could turn out this way by chance. In the code below the same experiment runs on 50 different splits: recall falls on average from 89.3% to 67.3%, accuracy from 93.5% to 87.7%, and weights with 8 malignant give 84.0%. Recall falls more than accuracy in 50 splits out of 50, and weights raise it in every one of them.

Accuracy can even rise while the model does not change at all. In the Atlas widget on balanced accuracy, a weak model on two measurements scores the same biopsies while malignant ones go from 212 down to 20. Its accuracy rises from 75.9% to 83.8%, while balanced accuracy stays put: 0.728 and 0.726. “Always benign” with 20 malignant scores as much as 94.7%.

Common mistake

True or false: “SMOTE always helps”. False. SMOTE makes synthetic examples of the rare class on the segments between real examples and their nearest neighbours, until both classes are the same size. That moves the decision boundary just as class weights do: the model says “malignant” more often. In the Atlas widget on SMOTE (357 benign, 30 malignant, two measurements, 5 × 5 cross-validation) the model without a fix finds 60.0% of the malignant biopsies at a precision of 90.0%. SMOTE raises detection to 86.7%, but precision falls to 42.9%. Class weights give the same recall at a precision of 40.2%, and plain duplication at 39.6%. That is one draw of 30 malignant, the one with the largest drop: on average over 20 draws detection rises from 68.7% to 91.9% and precision falls from 91.0% to 54.7%. It falls in every one of them. SMOTE adds no knowledge, it only changes the proportions. Whether that helps depends on the cost of mistakes: more found at the price of many false alarms.

A warning: the fix goes on the learning data only. Synthetic or duplicated examples in validation or the test would give a score that means nothing.

Quiz

Three questions to check yourself. Pick an answer, or open the answer under a question. Keyboard: arrow keys choose, Enter or Space confirms.

1. 8 malignant biopsies stay in learning instead of 159. Why does accuracy hardly show it?
Show the answer

Correct answer: Because benign biopsies are the majority in validation, and the model gets them right ever more surely

In validation 89 of the 142 biopsies are benign. Recall falls from 83.0% to 61.5%, accuracy only from 91.5% to 85.6%.

2. What does balanced accuracy measure?
Show the answer

Correct answer: The mean of the recalls of both classes, so “always the common class” gets exactly 0.5

Each class counts equally, however numerous it is. The widget’s weak model has a balanced accuracy of 0.728 with 212 malignant and 0.726 with 20, although its accuracy rises from 75.9% to 83.8%.

3. With 8 malignant in learning, class weights raise recall from 61.5% to 80.0%. What do they not do?
Show the answer

Correct answer: They do not add new examples, so something is still missing compared with all the data

Weights only change how much a mistake on each class costs. All the data with weights gives a recall of 86.8%, so with 8 malignant 6.8 points are missing.

Key takeaways

  • When a class is rare, accuracy hides the losses: report the recall of the rare class or balanced accuracy.
  • A model also learns the class proportions; with a rare class in learning it picks the common class more and more often.
  • Class weights and SMOTE move the boundary: more found at the price of false alarms, with no new knowledge.
  • Apply fixes to the learning data only, and check the result on untouched validation, ideally over many splits.

Going further: formula, code, practice

The math

Accuracy = (TP + TN) / n, recall = TP / (TP + FN), specificity = TN / (TN + FP), balanced accuracy = (recall + specificity) / 2. “Always benign” has recall 0 and specificity 1, so a balanced accuracy of (0 + 1) / 2 = 0.5.

Logistic regression minimises the mean logistic loss L = Σᵢ wᵢ · ℓ(yᵢ, pᵢ) / Σᵢ wᵢ. Without weights wᵢ = 1, so the 8 malignant biopsies are 8 / 276 ≈ 2.9% of the total loss. The “balanced” weights are w = n / (2 · n_class): 276 / 16 = 17.25 for a malignant biopsy and 276 / 536 ≈ 0.51 for a benign one, so each class weighs n / 2 = 138 in total.

In code (Python)

The same experiment in scikit-learn on 50 different splits: all benign and 159 or 8 malignant in learning, with and without class weights, scored on the untouched validation part.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score, accuracy_score

X, y = load_breast_cancer(return_X_y=True)
X, y = X[:, :10], 1 - y  # the 10 "mean" measurements, as in the lab; 1 = malignant
def run(seed, m, weights):
    tr, va = train_test_split(np.arange(len(y)), test_size=0.25, stratify=y, random_state=seed)
    mal = np.random.default_rng(seed).choice(tr[y[tr] == 1], m, replace=False)
    keep = np.r_[tr[y[tr] == 0], mal]  # all benign + m malignant to learn from
    clf = make_pipeline(StandardScaler(), LogisticRegression(class_weight="balanced" if weights else None))
    pred = clf.fit(X[keep], y[keep]).predict(X[va])  # the validation part never changes
    return recall_score(y[va], pred), accuracy_score(y[va], pred)
for m, w in ((159, False), (8, False), (8, True)):
    rec, acc = np.mean([run(s, m, w) for s in range(50)], axis=0)
    print(f"{m} malignant{' + class weights' if w else ''}: recall {rec:.1%}, accuracy {acc:.1%} (mean of 50 splits)")
drop = np.array([np.subtract(run(s, 159, False), run(s, 8, False)) for s in range(50)])
gain = np.array([run(s, 8, True)[0] - run(s, 8, False)[0] for s in range(50)])
print(f"recall fell more than accuracy in {(drop[:, 0] > drop[:, 1]).sum()} of 50 splits; weights raised recall in {(gain > 0).sum()}")

Output

159 malignant: recall 89.3%, accuracy 93.5% (mean of 50 splits)
8 malignant: recall 67.3%, accuracy 87.7% (mean of 50 splits)
8 malignant + class weights: recall 84.0%, accuracy 91.0% (mean of 50 splits)
recall fell more than accuracy in 50 of 50 splits; weights raised recall in 50

The lab shows one split and 5 draws, the code 50 splits, so the numbers differ: on the lab’s split recall with all the data is 83.0%, on average 89.3%. The conclusion is the same in every split.

Download the notebook (.ipynb). Open it in Jupyter or upload it to Google Colab; it fetches any missing data itself.

For practitioners
  • In scikit-learn weights come from class_weight="balanced" (logistic regression, SVMs, trees, forests) and balanced accuracy from balanced_accuracy_score. Report the recall and precision of the rare class too.
  • Split with stratification (stratify=y, StratifiedKFold), so the rare class reaches every part.
  • SMOTE and duplication live in the imbalanced-learn library. Use them inside its Pipeline, so they act only on the learning parts in cross-validation.
  • Class weights and a lower threshold do a similar thing: they move the decision towards the rare class. You can pick the threshold on validation from the PR curve (lesson 2.3) without training the model again.

Go deeper in the Atlas

Encyclopedia entries on the ideas in this lesson: