Project A: screening

The task

You are building a first sieve: a model scores a biopsy and decides who goes on to further tests. A missed malignant tumour (FN) costs as much as 10 false alarms (FP).

1. Pick a model (logistic regression, a decision tree or kNN) and its one setting.
2. Pick a threshold: from which score the model says “to biopsy”.
3. Write one sentence of justification: why this model and this threshold.

You choose everything on validation. The hidden test checks your choice at the end, the way new patients check a model after deployment.

Data and split

Breast Cancer Wisconsin (Diagnostic), UCI Machine Learning Repository, licence CC BY 4.0: 569 biopsies, 212 of them malignant. The model sees the 10 “mean” measurements of cell nuclei from the biopsy image (radius, texture, concavity and so on). UCI

• Learning: 427 biopsies (159 malignant). Cross-validation in 5 parts: each part is the validation once, and the model learns on the others. That gives the CV cost.
• Hidden test: 142 biopsies (53 malignant), set aside once, at random, keeping the class proportions (seed 1). The model learns on all 427 and only then scores the test.

“Hidden” here means “not shown on the page”, not locked. The page code is public and anyone can read it. Your browser counts the submissions, so clearing the site data resets the limit. This is an exercise in discipline, not an exam.

How to pass

Cost is counted per 100 people: (10 × misses + false alarms) × 100 / number of people. Passing is automatic, by two rules:

• the cost on the hidden test is lower than that of both simple rules: “everyone to biopsy” (62.7 per 100 people on the test) and “no one to biopsy” (373.2);
• only the first submission counts. You have 3 submissions, but the later ones are “after the fact”: they show the test cost and do not change the result.

Why this rule. The tool computes the CV cost itself, so validation cannot be gamed. What can be gamed is the test: submit, see the result, adjust. That is why only the first submission counts.

On the hidden test (142 biopsies) a single miss costs 7.0 per 100 people, so the test result jumps around. Before you submit, the tool shows the usual range: from CV − 25 to CV + 25. That was the spread of the test around CV for the example solutions from the lessons on 200 other random splits (95% of cases). Example: logistic regression at threshold 0.30 has CV 18.7 and test 48.6, outside the range. That is the bad luck of the split, not a mistake, and the choice still passes because it is cheaper than both rules.

The example solutions pass on at least 98% of those 200 splits. There is no ranking: only you see your result.

Your choice

Completion card

Once you pass, you can download a card (PNG). You type your name on this page, the card is made in your browser, and the name never goes into the page address or to a server. “Share on LinkedIn” passes only the address of this page.

For coders: the notebook

The notebook does the same in scikit-learn: the same dataset, the same 142 hidden test rows (given as row numbers), 5-part cross-validation, the cost per 100 people, the usual range around CV and the pass rule. Run the test cell once, after you have chosen: the first run is the one that counts. A tree and kNN may differ from the page by single cases there, because scikit-learn breaks ties differently.

Download the notebook (.ipynb)

README template for GitHub

A notebook copied from a tutorial says little about you. A README with your own decisions and limits says more. Copy the template and fill in every section in your own words.

# Screening: a model and a threshold for the cost of mistakes

Project A of the ML Atlas course: https://mlatlas.org/en/course/ml/project-a/

## Problem
Who should go to biopsy? Missing a malignant tumour costs as much as 10 false alarms. <In your own words: who uses the model and what happens after its decision.>

## Data and licence
Breast Cancer Wisconsin (Diagnostic), UCI Machine Learning Repository, CC BY 4.0, https://doi.org/10.24432/C5DW2B. 569 biopsies, 212 malignant, 10 “mean” measurements.

## Validation
<How you split the data, how many cross-validation parts, how many times you checked the hidden test.>

## Result ± spread
<Model, setting, threshold. CV cost per 100 people and its spread across the parts; the test cost.>

## Baseline
<The cost of “everyone to biopsy” and “no one” on the same data.>

## What the model cannot do
<Limits: how much data there is, where it comes from, what changes with other prices of mistakes.>

## Next steps
<What you would check next, and why.>

Download README.md