Project B: a clean table

The task

You get the table as it was published: 344 penguins, 2 of them without any measurement and 11 without a recorded sex. Island and sex are words, not numbers. The model has to recognise the species.

1. Build a pipeline from blocks: how to fill the gaps, what to do with a missing sex, how to encode island and sex, whether to scale, and which model. For every step that can learn from rows, choose where it is fitted: in the pipeline or before the split.
2. Write one sentence of justification and submit the pipeline to the hidden test.
3. In three pipelines written by someone else, point to the step that leaks.

You check the pipeline with cross-validation. The hidden test grades it once, at the end.

Data and split

Palmer Penguins (Gorman, Williams and Fraser 2014; the palmerpenguins package, Horst, Hill and Gorman 2020), licence CC0 1.0: 344 penguins of three species (Adelie, Chinstrap, Gentoo) from three islands of the Palmer Archipelago in Antarctica. Measurements: bill length and depth, flipper length, body mass. Plus island and sex. palmerpenguins

• Learning: 258 penguins (Adelie 114, Chinstrap 51, Gentoo 93). Cross-validation in 5 parts: each part is the validation once, and the pipeline learns on the others.
• Hidden test: 86 penguins (Adelie 38, Chinstrap 17, Gentoo 31), set aside once, at random, keeping the species proportions (seed 1). The pipeline learns on all 258 and only then scores the test.
• “Before the split” means: a step fitted on the whole table, 344 penguins including the hidden test. “In the pipeline” means: only on the penguins the model learns from (in every fold, and at the end on all 258).

“Hidden” here means “not shown on the page”, not locked. The page code is public and anyone can read it. Your browser counts the submissions, so clearing the site data resets the limit. This is an exercise in discipline, not an exam.

How to pass

Passing is automatic, by three rules:

• a leak-free pipeline: every step that learns from rows (the mean or median for the gaps, the most frequent sex, the scaler’s mean and spread) sits in the pipeline. A step that looks at one row at a time (the constant “unknown”, dropping a column) may stand anywhere;
• accuracy on the hidden test higher than both simple rules: “always the most common species” (44.2% on the test) and “one question”, a depth-1 tree that asks whether the flipper is at most 207.5 mm (77.9%);
• only the first submission counts, and only the first pick in each of the three pipelines written by someone else: you need three leaks out of three. You have 3 submissions, but the later ones are “after the fact”: they show the test result and do not change the pass.

Why the leak is checked in the structure, not in the number. On penguins such a leak barely moves the score: in every one of the 2304 pipelines you can build here, moving steps from the pipeline to before the split changes the CV accuracy by at most 0.8 percentage points. The number will not warn you, so the pipeline is judged by how it is built, as in a code review.

The tool computes CV itself, so validation cannot be gamed. What can be gamed is the test: submit, see the result, adjust. That is why only the first submission counts. Before you submit, the tool shows the usual range: from CV − 4 to CV + 4 percentage points. That was the spread of the test around CV for the example pipelines on 200 other random splits (95% of cases). One penguin on the test is 1.2 points.

The leak-free example pipelines pass on 100% of those 200 splits. kNN without scaling passes on only 35%: body mass in grams drowns out the other measurements, so in CV such a pipeline (75.7%) loses even to “one question” (79.0%), and the luck of the split would decide its pass. There is no ranking: only you see your result.

Your pipeline

Three pipelines by someone else

Each of these pipelines runs cross-validation on all 344 penguins, and in each one step leaks: three of the eight leakage types from the lesson on data leakage (Kapoor and Narayanan 2023). After your pick you see the type, the repair and how much the leak changes on the penguins.

Completion card

Once you pass, you can download a card (PNG). You type your name on this page, the card is made in your browser, and the name never goes into the page address or to a server. “Share on LinkedIn” passes only the address of this page.

For coders: the notebook

The notebook does the same in scikit-learn: the same dataset (the CSV file from the seaborn-data repository, pinned to one commit), the same 86 hidden test rows (given as row numbers), the same choices as the blocks on the page, 5-part cross-validation, the leak check from the structure and the pass rule. Run the test cell once, after you have chosen: the first run is the one that counts. In the checked examples kNN gives the same numbers as the page; a tree may differ by single penguins, because scikit-learn breaks ties differently.

Download the notebook (.ipynb)

README template for GitHub

A notebook copied from a tutorial says little about you. A README with your own decisions and limits says more. Copy the template and fill in every section in your own words.

# A clean table: a pipeline without leaks

Project B of the ML Atlas course: https://mlatlas.org/en/course/ml/project-b/

## Problem
Recognise a penguin’s species from its measurements, island and sex when the table has gaps and categories. <In your own words: who it is for and what for.>

## Data and licence
Palmer Penguins, Gorman, Williams and Fraser (2014), the palmerpenguins package (Horst, Hill and Gorman 2020), CC0 1.0, https://allisonhorst.github.io/palmerpenguins/. 344 penguins, 2 without measurements, 11 without sex.

## Validation
<How you split the data, how many cross-validation parts, where every step is fitted (in the pipeline or before the split) and how many times you checked the hidden test.>

## Result ± spread
<The pipeline, CV accuracy and its spread across the parts; accuracy on the test.>

## Baseline
<The accuracy of “always the most common species” and “one question” on the same data.>

## What the model cannot do
<Limits: three islands, the years 2007–2009, what about a penguin without measurements, what about another species.>

## Next steps
<What you would check next, and why.>

Download README.md