Overfitting: when a model memorises

Two students prepare for a maths exam. One works through the ideas behind the problems. The other memorises the answers to every past paper, word for word. On the past papers the second student is flawless. Then the real exam arrives with new numbers. Who do you bet on? Models face exactly this choice, and left alone, a flexible model will happily become the second student.

Try this

• Start at degree 1 and note both errors. Then raise the degree slider one step at a time and watch the training error fall.
• Keep going and watch the test error. Find the degree where it is lowest — that is the sweet spot for this sample.
• Push the degree slider to the maximum. Look at the curve between the points: does it still look like a sensible description of how BMI relates to progression?
• Compare the gap between training and test error at the lowest and at the highest degree. A growing gap is the fingerprint of overfitting.

Now that you've tried it, let's see what was actually going on.

The idea in plain words

In lesson 3 we fitted a straight line. A straight line is stiff: it cannot bend to follow a curve, even when the data clearly curve. A polynomial is a more flexible family of curves:

ŷ = w₀ + w₁x + w₂x² + … + w_d·x^d

The degree d controls the flexibility. Degree 1 is the straight line; degree 2 can make one bend; higher degrees can wiggle more and more. With enough degree a polynomial can pass through every single training point.

That sounds like success, but real data contain noise — random variation that no feature explains. Two patients with the same BMI do not progress identically. A model flexible enough to hit every point is not only learning the trend; it is also learning the noise, the accidents of this particular sample. That is overfitting: the model fits the training data better and better while its predictions for new data get worse. The opposite failure, a model too stiff to capture the real pattern, is underfitting.

Why it works

Take a small random sample of the diabetes patients from lesson 3 and split it into training and test points. A small sample makes the effect easy to see, because a few unusual points can drag a flexible curve far off course.

Follow what happens to the two errors as the degree goes up. The training error keeps going down: a more flexible curve can always do at least as well on the points it is fitted to, because it can imitate any simpler curve. The test error behaves differently. At first it falls too, because the extra flexibility captures real structure. Then, past some degree, it starts to rise, because the curve now bends to chase individual training points, and those bends are wrong for everybody else. The result is the famous U shape of test error.

The reason has a name: the bias–variance trade-off. A stiff model has high bias — it is wrong in a systematic way, whatever sample it sees. A very flexible model has high variance — draw a slightly different sample and you get a wildly different curve. The best model sits in between, flexible enough to follow the signal and stiff enough to ignore the noise. Where that point lies depends on how much data you have: with more data, the noise averages out and more flexibility becomes safe.

Common mistake

The fundamental error is choosing model complexity by training error. Training error always rewards more flexibility, so it will always vote for the most complex model — the memorising student. The second error is assuming overfitting is rare or exotic. It is the default behaviour of any flexible model on limited data, which covers decision trees, neural networks and most of what you will build. The cure is not to avoid flexible models but to control them: simpler models, more data, or regularisation, which penalises extreme weights.

There is a catch hidden in this reasoning: we picked the best degree by looking at the test error, which is exactly what lesson 2 warned against. In the next lesson we learn how to choose settings honestly, using validation and cross-validation.

import numpy as np
from sklearn.datasets import load_diabetes

X, y = load_diabetes(return_X_y=True, as_frame=True, scaled=False)
x, y = X["bmi"].to_numpy(), y.to_numpy()
perm = np.random.default_rng(0).permutation(len(x))
tr, te = perm[:30], perm[30:]
te = te[(x[te] >= x[tr].min()) & (x[te] <= x[tr].max())]  # stay inside the training range
rmse = lambda i, p: np.sqrt(np.mean((y[i] - p(x[i])) ** 2))
for deg in (1, 3, 10):
    p = np.polynomial.Polynomial.fit(x[tr], y[tr], deg)  # fits on a rescaled x, so degree 10 is stable
    print(f"degree {deg:2d}: train RMSE {rmse(tr, p):6.1f}, test RMSE {rmse(te, p):6.1f}")

degree  1: train RMSE   61.6, test RMSE   63.5
degree  3: train RMSE   60.9, test RMSE   63.8
degree 10: train RMSE   54.7, test RMSE  199.5