Why does linear regression square the residuals instead of just adding them up?

So that misses in both directions cannot cancel, and big ones cost more

Because squared residuals are computed faster than plain ones on a computer

Because residuals are always negative, so squaring makes them positive

With the slope fixed at zero, which intercept gives the lowest MSE?

The mean of the target variable in the training data

Zero, because a flat line passes through the origin

The largest value of the target variable in the data

The median of the feature values, i.e. the middle of the horizontal axis

The best line has a positive slope. What can you conclude?

That in these data higher BMI goes with more disease progression on average

That artificially raising a patient's BMI will speed up the disease

That BMI explains all the differences in progression between patients

Machine learning from zero · Lesson 1.5 · 6 of 33 lessons ready · 10 min

Linear regression: fitting a line

In short

Linear regression draws the straight line that misses the data least. On diabetes data, mean squared error decides which line is best.

Warm-up Two questions from earlier lessons. A minute, and it never blocks the lesson.

From lesson 1.4: Nearest neighbours: the simplest model How does kNN answer a question about a new case? It applies a rule with thresholds it learned from the learning data It takes the k nearest learning examples and picks their majority It computes the mean of all the learning examples of the same class

Show the answer

Correct answer: It takes the k nearest learning examples and picks their majority

kNN has no rule: the k nearest stored examples vote. The background in the lab shows the result of that vote at every point.

From lesson 1.3: Baselines: what do you compare your model with? The test has 108 biopsies, 1 of them malignant. “Always benign” gets 99.1%, the rule 90.7%. Which is better for screening? “Always benign”, because it has higher accuracy on the same test Neither, because with one malignant biopsy nothing can be said The rule, because it finds the malignant one and “always benign” never will

Show the answer

Correct answer: The rule, because it finds the malignant one and “always benign” never will

With a rare class accuracy rewards ignoring it. “Always benign” will never find a malignant biopsy, however many there are.

Does body weight say anything about how fast a disease gets worse? Doctors have plenty of hunches like this. Machine learning lets you turn a hunch into a number: draw the best straight line through the data and read off how steep it is. The only question is what "best" means — and the answer to that question is the seed of everything that follows in this course.

Predict, then check You fit the best straight line to the BMI and disease progression chart. What share of the variation in progression, that is of the differences between patients, does BMI alone explain? This share is called R². About 80% About 5% About 34%

How sure are you? I’m sure I’m not sure

Interactive lab

This interactive lab runs in a browser with JavaScript enabled. The text around it describes what the lab shows.

Show the answer

Correct answer: About 34%

The best line has R² = 0.34 (mean squared error 3890, against 5930 for a flat line at the mean): BMI explains roughly a third of the variation, and the other 66% depends on something else.

Try this

Move the slope slider and watch the line tilt and the MSE change. Then move the intercept slider. Try to bring the MSE as low as you can by hand.
Press Best fit and compare the least squares line with yours. How close did you get?
Set the slope slider to zero, so the line is flat, and move only the intercept until the MSE is as low as possible. You have just found the average progression: with no feature, the best guess for everyone is the mean.
Compare that flat line's MSE with the best fit's MSE. The difference is what knowing BMI is worth.

Now that you've tried it, let's see what was actually going on.

The idea in plain words

In lesson 1 the model answered "yes" or "no". Now it answers with a number, which is called regression. The simplest regression model is a straight line:

prediction = slope × feature + intercept

The slope says how much the prediction changes when the feature grows by one unit. The intercept shifts the whole line up or down. Learning means finding the two numbers that make the line pass as close as possible to the data points.

Picture a used-car dealer pricing cars by age. His rule of thumb is a straight line: start from a price for a new car (the intercept) and take off a fixed amount for every year (the slope). No real car sells for exactly that price, but the line captures the trend, and it is easy to explain.

In this lesson we use data from a classic medical study of 442 diabetes patients. For each one we know the body mass index (BMI) and a measure of how much the disease progressed one year after the first visit. So each patient is a pair of numbers, which we can think of as one point: BMI across, progression up.

Why it works

For every patient the line makes a prediction, and the gap between the real value and the prediction is called the residual — the vertical distance from the point to the line. To score a line we need to turn hundreds of residuals into one number. The standard choice is the mean squared error:

MSE = (1/n) · Σ (yᵢ − ŷᵢ)²

where yᵢ is the real value for patient i, ŷᵢ is the line's prediction and n is the number of patients. Squaring does two jobs. It makes every miss positive, so misses above and below the line cannot cancel out. And it punishes big misses much more than small ones: a point twice as far from the line costs four times as much. A line that is slightly off for everyone usually beats a line that is perfect for most and terribly wrong for a few.

The line with the lowest MSE is called the least squares fit. For a straight line it does not even have to be searched for: a short formula, known since the early nineteenth century, gives the exact best slope and intercept. Notice what this means. Learning is the same recipe as before — a family of rules (all possible lines), a score (MSE), and the rule with the best score — only now the family is infinite and the score is continuous.

One more thing to notice: even the best line leaves big residuals on this data. BMI is related to progression, but it is far from the whole story. A model can only be as good as the information in its features.

Common mistake

A fitted line is not a law of nature. Two classic errors follow from forgetting that. The first is reading the slope as cause: patients with higher BMI progress faster on average in these data, but the line alone cannot say whether weight causes progression or whether both follow from something else. The second is extrapolation — using the line far outside the range of the data. The patients in this study have BMIs within a limited range; the line says nothing trustworthy about values it has never seen.

Finding the best line by formula is a luxury that most models do not have. In the next lesson we find the same line a different way — by taking small downhill steps — which is how neural networks learn.

Catch the AI’s mistake

This is how a chatbot answered. One sentence is factually wrong. Which one?

Question: I fitted a straight line of disease progression on BMI for 442 diabetes patients and got R² = 0.34. What does that tell me?

Chatbot’s answer The least squares line is the one with the lowest mean squared error, so no other straight line fits these patients better. An R² of 0.34 compares that error with the error of a flat line drawn at the mean progression. In practice it means the line predicts the progression of about 34% of the patients correctly and misses the rest. The positive slope shows that higher BMI goes with faster progression on average, although on its own it does not prove that weight causes it. To explain more of the variation, you would need features that carry more information than BMI alone.

Show the wrong sentence

The wrong sentence: In practice it means the line predicts the progression of about 34% of the patients correctly and misses the rest.

R² is not a share of patients: R² = 0.34 means BMI explains about a third of the variation in progression (mean squared error 3890 against 5930 for the flat line), and the other 66% depends on something else. Regression predicts a number, and almost every patient keeps some residual.

Key takeaways

Regression predicts a number; a straight line is the simplest regression model.
Mean squared error scores a line by averaging squared residuals, so large misses weigh the most.
The least squares line is the one with the lowest MSE and has an exact formula.
With no useful feature, the best constant prediction is the mean.
A slope is an association within the data's range, not a cause and not a licence to extrapolate.

Going further: formula, code, practice

The math

For one feature x the line is ŷ = w·x + b. Least squares picks the w and b that minimise MSE(w, b) = (1/n) · Σ (yᵢ − w·xᵢ − b)². Setting both partial derivatives to zero gives the closed form:

w = Σ (xᵢ − x̄)(yᵢ − ȳ) / Σ (xᵢ − x̄)², the covariance of x and y divided by the variance of x;
b = ȳ − w·x̄, so the line always passes through the point of means (x̄, ȳ).

With the slope fixed at 0 the MSE is smallest at b = ȳ, and that minimum equals the variance of y. That is why R² = 1 − MSE(line) / MSE(mean) measures how much of the variance the feature explains. For a line with one feature, R² is the square of the Pearson correlation between x and y.

In code (Python)

The same 442 patients as the lab, with BMI in its original units (scaled=False):

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

X, y = load_diabetes(return_X_y=True, as_frame=True, scaled=False)
bmi = X[["bmi"]]  # one feature, kept as a table (sklearn expects 2-D input)
model = LinearRegression().fit(bmi, y)
pred = model.predict(bmi)

print(f"slope {model.coef_[0]:.2f}, intercept {model.intercept_:.1f}")
print(f"MSE of the line: {mean_squared_error(y, pred):.0f}")
print(f"MSE of the mean: {mean_squared_error(y, [y.mean()] * len(y)):.0f}")
print(f"R2: {r2_score(y, pred):.2f}")

Output

slope 10.23, intercept -117.8
MSE of the line: 3890
MSE of the mean: 5930
R2: 0.34

Every point of BMI adds about 10 to the predicted progression. The intercept of −118 is only an extrapolation to BMI 0, far outside the data (BMI 18–42), so it means nothing on its own.

Download the notebook (.ipynb). Open it in Jupyter or upload it to Google Colab; it fetches any missing data itself.

For practitioners

Scores computed on the training data are optimistic. Here that is harmless (two parameters, 442 points), but in general, report MSE and R² on a test set (lesson 2).
Check the residuals, not just R². Plot them against the prediction: a curve or a funnel shape means the straight line, or constant noise, is the wrong assumption.
MSE is in squared units of the target (here, progression²). Its square root (RMSE, about 62 here) is easier to explain to anyone else.
One outlier with a large residual can pull a least-squares line a long way. When a few points are unusual, look at them first, and only then consider a robust loss such as Huber.
A slope is comparable between features only when they share a scale. Standardise the features before reading coefficients as "importance", and even then it is association, not cause.

Go deeper in the Atlas

Encyclopedia entries on the ideas in this lesson:

Linear regression: fitting a line

Try this

The idea in plain words

Why it works

Common mistake

Catch the AI’s mistake

Quiz

Key takeaways

Going further: formula, code, practice

Go deeper in the Atlas