03 · Supervised · 4 min read · Interactive · updated 8 October 2026

How does a decision tree work and why is XGBoost built from them?

What it is

A decision tree is a model that guides an example from the root to a leaf through a series of questions about one feature and a threshold, e.g. "age ≤ 6.5?"; the leaf gives a class or a value. Each question splits feature space along a single axis, so the tree carves it into rectangles (hyperrectangles) and makes a separate decision in each.

Trees are used on their own when readable rules matter, but above all as building blocks of random forests and gradient boosting (XGBoost, LightGBM, CatBoost) — the standard for tabular data.

Mechanism — why it works this way

Construction is greedy (the CART algorithm, Breiman et al. 1984). At the root, the tree tries every feature and every sensible threshold and picks the split after which the two resulting groups are most homogeneous — measured by the drop in Gini impurity or entropy (classification) or variance (regression). It then repeats this in each branch separately, until a branch is pure, too small, or the depth limit is reached.

Why is that enough? The tree does not need to know how the features combine into a pattern: successive questions carve out regions on their own, including thresholds and interactions that no straight line of a linear model can draw. Two questions about the same feature give an interval, questions about different features give a rectangle, and a question asked in only one branch is an interaction ("age matters, but only for men"). Each leaf is a constant, so a tree of depth d has at most 2ᵈ leaves and is a step function. Diagonal relationships, such as the sum of two features, need many questions instead of one.

A single tree has high variance: a small change in the data changes the root question, and with it the whole tree. A deep tree memorises the data, a shallow one underfits. A random forest averages many trees trained on different samples and feature subsets (variance reduction); boosting adds shallow trees, each correcting the errors of the previous ones (bias reduction). The same rectangles then add up to a smoother, more accurate boundary.

Caveat: trees need no standardisation (a threshold question is unchanged by any increasing transformation of the feature), cope well with different scales and — in some implementations — with missing values. They do not, however, extrapolate beyond the training range, and without aggregation they are unstable.

By example

Titanic: 891 passengers, 38.4% survived. Features: class, sex, age (missing values filled with the median), number of siblings or spouses, parents or children, ticket fare; 70/30 split, random_state=0. The "nobody survived" model scores 61.6% on the test set. A depth-1 tree asks a single question — about sex — and reaches 79.1%. A depth-2 tree on the full dataset reads like an account of the disaster: of the women in 1st and 2nd class, 161 of 170 survived; of the women in 3rd class, 72 of 144; of boys up to 6.5 years old, 16 of 24; and of older males only 93 of 553.

A depth-3 tree (8 leaves) scored 83.2% on the test set. An unrestricted tree grew to 143 leaves, reached 98.6% on training and only 78.0% on test — worse than the single question about sex. In 5-fold cross-validation, depths 3–4 did best (81.8% and 82.2%), and the unrestricted tree scored 77.8%.

Try it yourself: the interactive illustration (labels in Polish for now).

In practice

• scikit-learn: DecisionTreeClassifier(criterion="gini", max_depth=None) grows to pure leaves by default — set max_depth, min_samples_leaf or ccp_alpha (pruning); plot_tree and export_text show the rules.
• XGBoost: max_depth=6 by default; LightGBM grows leaf-wise (num_leaves=31); CatBoost builds symmetric trees.
• Depth 1–3 gives the weak trees typical of boosting; a standalone, readable tree usually has depth 3–5.
• On tabular data, boosted trees usually beat neural networks; networks win on images, text and very large datasets.
• Typical mistake: max_depth=None on small data and trusting the training score; another — standardising "for the tree", which changes nothing.

Frequently asked questions

How does a decision tree choose what to ask?

It tries every feature and every threshold between adjacent values, computes how much the split reduces impurity (Gini, entropy) or variance, and picks the largest reduction. It repeats this in each branch separately — greedily, without looking several steps ahead.

Does a decision tree require standardised data?

No. The question "x ≤ threshold" gives the same split after any increasing transformation of the feature, so scaling, logs or standardisation change nothing. This is one of the main conveniences of trees and boosting compared with neural networks and linear regression.

Why does a single tree overfit, but a forest or boosting less so?

A tree grown to pure leaves memorises every example and is unstable. A forest averages hundreds of trees on different samples, which smooths out accidental cuts; boosting uses shallow trees and a small step size, so each tree contributes only a little.

Sources

See also