Exploration: look before you model

Module 4 starts before the model. Before you train anything, look at the data. Exploratory data analysis (EDA) means drawing the data and computing simple summaries to see what you are dealing with: the shape, the groups, the points far from the rest. A mean or a correlation is a summary, and a summary can hide exactly what matters. Two examples today: Anscombe's four small data sets and the penguins of lesson 0.1.

Try it

• Click through sets I–IV. Which numbers under the charts change, and which stay the same?
• In which set does a single point set the line? What would happen to the line without it?
• Switch to the penguins and pick bill length × bill depth. What sign does the correlation have? Then split the penguins by species.
• Check the same pair split by island. Why is the picture different from the split by species?

Once you have checked it in the lab, let's name what you see.

The idea in plain words

Summary statistics, that is the mean, the variance, the correlation and the regression line, condense data into a few numbers. A summary always loses something. In 1973 Francis Anscombe built four sets of 11 points in which these numbers are almost identical: the mean of x is 9, the mean of y 7.50, the variance of x 11, the variance of y 4.12–4.13, the correlation 0.816, the line y = 3.00 + 0.500·x. On a chart set I is a cloud, II an arc, III a line with one outlier, IV a vertical column and one distant point. Linear regression fitted to each of them gives the same line, although it sensibly describes only the first set.

The penguins show the same on real data. Over all 342 penguins the correlation of bill length and bill depth is −0.235: the longer the bill, the shallower it is. Split by species, the sign flips: Adelie 0.391, Chinstrap 0.654, Gentoo 0.643. Inside every species a longer bill is also deeper. One number for the whole set describes a relationship that exists in no group.

Jak to policzono

• Quartet: Anscombe (1973), The American Statistician. 4 sets of 11 points, copy from the seaborn-data repository. Variance divided by n − 1.
• Penguins: Palmer Penguins (Gorman, Williams and Fraser, 2014; palmerpenguins package: Horst, Hill and Gorman, 2020), licence CC0. 342 penguins with bill measurements, 3 species: Adelie 151, Chinstrap 68, Gentoo 123.
• Correlation r: Pearson's coefficient, from −1 to 1; 0 means no linear relationship.
• Lines: least squares, separately for every set and every group.

Why it works

Where does the flipped sign in the penguins come from? Gentoo have long but shallow bills: on average 47.5 mm long and 15.0 mm deep. Adelie have short, deep bills: 38.8 and 18.3 mm. One correlation for everyone mostly compares the species with each other: Gentoo sit in the bottom right of the chart, Adelie in the top left, so the line through all points falls. Inside every species the relationship is positive, because a bigger penguin has a bigger bill in both directions. That is Simpson's paradox: when groups differ in their means, the relationship in the whole can have the opposite sign to the one in every group.

In the quartet the mechanism is even simpler. The mean, the variance and the correlation are sums over the points, and different arrangements of points give the same sums, just as different sets of numbers have the same mean. Points far from the rest pull hardest. In set IV the point at x = 19 alone sets the slope, because the other 10 points share the same x = 8 and say nothing about direction.

Is the penguin result a fluke of this one sample? In the code below we resample the penguins with replacement 1000 times: the same number of penguins, some twice, some not at all. In 1000 of 1000 resamples the correlation over all is negative and positive in every species.

Common mistake

True or false: “summary statistics are enough; if two data sets have the same mean, variance and correlation, they look alike”. False. In the lab Anscombe's four sets share these numbers to two decimal places and look completely different. Numbers summarise: they do not show the shape, the groups or the single points that pull the result. In the penguins one correlation of −0.235 says “longer bill, shallower bill”, and inside every species it is the other way round.

For a model this has concrete consequences. Linear regression fitted to set II predicts a straight line where the data lie on an arc. A penguin model that does not know the species learns a negative relationship that exists in no species. That is why you draw the data first, and only then choose the model and the features.

import numpy as np
import pandas as pd

q = pd.read_csv("data/raw/anscombe.csv")
for name, d in q.groupby("dataset"):
    slope, intercept = np.polyfit(d.x, d.y, 1)
    print(f"set {name}: mean x {d.x.mean():.2f}, mean y {d.y.mean():.2f}, variance x {d.x.var():.2f}, variance y {d.y.var():.2f}, r {d.x.corr(d.y):.3f}, line y = {intercept:.2f} + {slope:.3f}x")
df = pd.read_csv("data/raw/penguins.csv").dropna(subset=["bill_length_mm", "bill_depth_mm"])
r = lambda d: d.bill_length_mm.corr(d.bill_depth_mm)
print(f"{len(df)} penguins, bill length vs bill depth: all r = {r(df):.3f}; " + ", ".join(f"{s} {r(g):.3f}" for s, g in df.groupby("species")))
rng = np.random.default_rng(0)
same = 0
for b in range(1000):
    s = df.sample(len(df), replace=True, random_state=rng)  # a resample with replacement: as many penguins, some twice
    same += r(s) < 0 and all(r(g) > 0 for _, g in s.groupby("species"))
print(f"1000 bootstrap resamples: negative over all and positive in every species in {same}")

set I: mean x 9.00, mean y 7.50, variance x 11.00, variance y 4.13, r 0.816, line y = 3.00 + 0.500x
set II: mean x 9.00, mean y 7.50, variance x 11.00, variance y 4.13, r 0.816, line y = 3.00 + 0.500x
set III: mean x 9.00, mean y 7.50, variance x 11.00, variance y 4.12, r 0.816, line y = 3.00 + 0.500x
set IV: mean x 9.00, mean y 7.50, variance x 11.00, variance y 4.12, r 0.817, line y = 3.00 + 0.500x
342 penguins, bill length vs bill depth: all r = -0.235; Adelie 0.391, Chinstrap 0.654, Gentoo 0.643
1000 bootstrap resamples: negative over all and positive in every species in 1000