Ensembles: why many trees beat one

Ask one person to guess how many sweets are in a large jar and the guess may be wildly off. Ask a few hundred people and average their guesses, and the result is often startlingly close. No single guesser became smarter. Their errors simply pointed in different directions and cancelled out. Can we build a model the same way, out of many mediocre ones?

Try this

• Start with the number-of-trees slider at one and compare the forest with the single tree. With one tree, the "forest" is just one randomised tree.
• Move the number-of-trees slider up gradually and watch the forest's test accuracy. Notice where it stops improving noticeably.
• Keep increasing the number of trees. Does the forest's accuracy start to fall, as the deep tree's did in lesson 9, or does it settle?
• Compare the single tree with the large forest. Which one would you trust with a case you have never seen?

Now that you've tried it, let's see what was actually going on.

The idea in plain words

An ensemble is a model made of many models whose predictions are combined, by voting for categories or averaging for numbers. The most famous ensemble is the random forest, introduced by Leo Breiman in 2001: a large number of decision trees, each grown a little differently, voting together.

Lesson 9 ended on a weakness of trees: a small change in the data can produce a completely different tree. A single deep tree is like one overconfident guesser — sharp but erratic. A forest turns that erratic behaviour into an asset. If every tree makes different mistakes, the majority vote is right more often than a typical tree.

Why it works

The key word is different. Averaging identical models changes nothing; their mistakes are the same. A random forest makes its trees differ in two ways. First, each tree is trained on a bootstrap sample: rows drawn at random from the training set with replacement, so each tree sees a slightly different dataset, with some rows repeated and others left out. This idea is called bagging. Second, at each split a tree may only choose among a random subset of the features, so trees cannot all lean on the same strongest feature.

Statistics tells us how much this helps. If each tree's error has variance σ² and the errors of any two trees are correlated with correlation ρ, the average of n trees has variance

ρ·σ² + (1 − ρ)·σ² / n

The second term shrinks as you add trees and disappears for large n. The first term does not: it is the part of the error that all trees share. This is why randomness matters. The less the trees resemble each other (the smaller ρ), the more the forest gains — and why adding trees beyond a certain point brings almost nothing.

A pleasant consequence: adding more trees does not make a forest memorise more. Each new tree just joins the vote, and the forest's predictions settle down instead of getting wilder. The cost of more trees is computing time, not overfitting.

There is a second big family of ensembles, boosting, which works the other way round: instead of independent trees that vote, it builds small trees one after another, each correcting the mistakes of the ones before. Gradient-boosted trees are a frequent winner on table-shaped data, including many Kaggle competitions.

Common mistake

An ensemble cannot create information that is not in the features. If every tree is fooled the same way — for example by a feature that leaks the answer, or by data that do not represent the people you will predict for — the vote just repeats the same mistake with more confidence. The correlated term in the formula is exactly this shared error. Another mistake is assuming more trees always help: after the gains flatten out, extra trees only cost time. Finally, a forest is much harder to read than a single tree; you trade the flowchart for accuracy.

Trees and forests split the feature space into boxes. In the next lesson we meet a model that bends its boundary smoothly instead: the neural network.

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

df = pd.read_csv("data/raw/titanic.csv")
X = pd.DataFrame({"female": (df["sex"] == "female").astype(int)}).join(df[["pclass", "age", "sibsp", "parch", "fare"]])
Xtr, Xte, ytr, yte = train_test_split(X, df["survived"], test_size=0.3, stratify=df["survived"], random_state=1)
Xtr, Xte = Xtr.fillna({"age": Xtr["age"].median()}), Xte.fillna({"age": Xtr["age"].median()})  # train median for both
tree = DecisionTreeClassifier(random_state=0).fit(Xtr, ytr)
print(f"one full tree: test {tree.score(Xte, yte):.3f}")
for n in (1, 10, 100, 500):
    forest = RandomForestClassifier(n_estimators=n, max_features=2, random_state=0).fit(Xtr, ytr)
    print(f"forest of {n:3d}: test {forest.score(Xte, yte):.3f}")
oob = RandomForestClassifier(n_estimators=500, max_features=2, oob_score=True, random_state=0).fit(Xtr, ytr)
print(f"out-of-bag estimate (500 trees, training rows only): {oob.oob_score_:.3f}")

one full tree: test 0.780
forest of   1: test 0.754
forest of  10: test 0.806
forest of 100: test 0.802
forest of 500: test 0.802
out-of-bag estimate (500 trees, training rows only): 0.809