Evaluation & Metrics

Precision, recall, ROC/AUC, calibration, and honest model evaluation and validation.

intermediate#metrics#roc#auc#cross-validation

Start here — a model that is 99% accurate and completely useless

A hospital wants to screen for a rare disease. About 1 person in 100 actually has it. They ask two teams to build a model, and they test both on the same 10,000 people.

Team A ships a real model that reads scans and flags patients at risk.

Team B ships one line of code:

def predict(patient):
    return "healthy"   # always

Team B's model never looks at anything. It just says "healthy" every single time. And it is 99% accurate — because 9,900 of the 10,000 people really are healthy, and it got every one of them right.

Team A's real model scores 95.6%.

Read that again

The model that detects nothing scored higher than the model that actually works. If the hospital picks the winner by accuracy, it ships the model that misses every single sick patient — and it will look like a great decision on the dashboard.

This is not a trick question or an edge case. It is the single most common way real machine-learning projects fail, and it is why this whole lesson exists.

is not wrong, exactly. It answers a question — how often is the model right? — perfectly well. The problem is that it is almost never the question you actually care about.

One number cannot describe two different mistakes

A classifier can fail in two completely different ways: it can raise a false alarm on a healthy person, or it can miss a sick one. Those mistakes cost wildly different things. A false alarm costs a follow-up test and a bad afternoon; a miss can cost a life. Accuracy blends both into one number and then throws away which was which. Everything in this lesson is about not doing that.

Think of it like a smoke alarm that never goes off:

An alarm that is silent 365 days a year is correct on 364 of them. Rate it on "days it was right" and it scores 99.7% — better than a working alarm that chirps once when you burn toast. But you do not buy a smoke alarm to be right on quiet days. You buy it for the one day that matters, and that day never appears in the average.

Try to recall

A fraud detector is 99.9% accurate on a dataset where 0.1% of transactions are fraud. What is the cheapest way to get that score, and does it catch any fraud?

Hint: What does the model that always predicts the majority class score?

How to read this page

It teaches from first principles: every metric arrives as a plain-language question first, then a worked example with real numbers, and only then the formula. Flip the Depth switch at the top for the formal definitions and the derivations — nothing is hidden for good, and the deeper panels open automatically once you have the prerequisites.

The four boxes — the confusion matrix

Since one number hides which mistake happened, let us stop compressing and just count all four outcomes separately.

Every prediction on a yes/no problem lands in exactly one of four boxes: the model said yes or no, and the truth was yes or no. That two-by-two table is the , and every metric in the rest of this lesson is built out of it.

— flagged, and genuinely sick. A correct catch.
— flagged, but actually healthy. A false alarm.
— cleared, but actually sick. A miss.
— cleared, and genuinely healthy. A correct all-clear.

Here is what Team A's real model actually produced on those 10,000 people. Look at the picture before the numbers:

The confusion matrix for Team A, on 10,000 patients— interactive, drag & zoom

Loading chart…

Colour tracks the raw count — and one box swallows the whole picture. The 9,489 correct all-clears are so numerous that they alone decide the accuracy score, which is exactly why accuracy could not tell Team A from a model that predicts nothing. The three boxes that describe whether this model is any good are the three faint ones.

Reading the four boxes, one row at a time

Read the table the way a doctor would, one true group at a time.

The 100 people who really were sick. The model flagged 75 of them (TP) and cleared 25 (FN). So it caught three out of four, and 25 sick people were sent home believing they were fine.

The 9,900 people who really were healthy. The model cleared 9,489 (TN) and flagged 411 (FP). So 411 healthy people got a frightening phone call and an unnecessary follow-up test.

Now the accuracy. Correct predictions are the two boxes on the diagonal:

\text{accuracy} = \frac{75 + 9{,}489}{10{,}000} = \frac{9{,}564}{10{,}000} = 95.6\%

And there is the whole problem in one line. The number 9,489 dominates that fraction so completely that the 75 catches and the 25 misses barely move it. Accuracy is not lying — it is answering a question about the 9,900 healthy people, when the entire point of the model is the 100 sick ones.

Run the counts yourself. This cell hard-codes Team A's four boxes and Team B's one-line model, and puts their scores side by side:

Python · runs in your browser

What this does: Takes the four confusion-matrix counts for the real model and for the always-healthy model, and prints accuracy for both. It shows the punchline of this whole section: the useless model wins on accuracy, 99.0% to 95.6%, while catching zero sick patients.

Try to recall

Of the four boxes, which one is a miss and which one is a false alarm?

Hint: Positive means the model said yes.

Precision and recall — two different questions

Rather than blending the four boxes into one score, ask two sharp questions and answer each separately.

Two questions a manager would actually ask

Question 1 — When the model raises the alarm, should I believe it? Of everyone it flagged, how many were really sick? That is precision. It is a question about the model's outputs.

Question 2 — Of the people who need finding, how many did it find? Of everyone who really was sick, how many did it flag? That is recall. It is a question about the truth.

They are different questions with different denominators, and a model can ace one while failing the other.

asks: how trustworthy is an alarm?

asks: how complete is the search?

Think of it like fishing with a net:

Cast a net into a lake and haul it in.

Precision is: of everything in your net, what fraction is actually fish? Drag up a net full of boots and weeds and your precision is terrible, however many fish you also got.

Recall is: of all the fish in the lake, what fraction ended up in your net? Leave most of the lake unfished and your recall is terrible, however clean your catch.

And now the tension that runs through this entire lesson: you can trivially max out either one. Want perfect recall? Drag a net across the whole lake — you catch every fish, plus every boot (precision collapses). Want perfect precision? Reach in and grab one fish you can already see — one for one, flawless, and you left the lake full (recall collapses). Neither number means anything on its own. They only mean something together.

Precision and recall for Team A, from the four boxes

Team A's counts again: $TP = 75$ , $FP = 411$ , $FN = 25$ , $TN = 9{,}489$ .

Precision — of everyone flagged, how many were sick? The model flagged $75 + 411 = 486$ people in total, and 75 of them were genuinely sick:

\text{precision} = \frac{75}{75 + 411} = \frac{75}{486} = 0.154

15.4%. When this model raises the alarm, it is wrong more than five times out of six. Roughly 85 of every 100 people it calls back are perfectly healthy.

Recall — of everyone sick, how many were flagged? There were $75 + 25 = 100$ sick people, and it found 75:

\text{recall} = \frac{75}{75 + 25} = \frac{75}{100} = 0.75

75%. It catches three sick patients in four, and misses the fourth.

Notice how much more this tells you than "95.6% accurate". Two numbers, two clear sentences: it finds three quarters of the disease, and five sixths of its alarms are false. Whether that is a good model now depends on something no formula knows — how much a missed diagnosis costs compared to an unnecessary follow-up test. For screening, where a miss can be fatal and a false alarm just means another blood draw, this trade might be exactly right. For an automated system that starts chemotherapy, it would be indefensible. The metric cannot make that call for you. It can only make the trade-off visible.

A spam filter flags 200 emails. 180 of them really are spam. There were 400 spam emails in the inbox in total. What are its precision and recall?

F1 — when you really do need one number

Sometimes you have to rank fifty models on a leaderboard, and two numbers per model will not do. The usual compromise is the .

A single number that refuses to be gamed

The obvious move — average precision and recall — fails immediately. Flag every single person and you get recall 1.0 with precision near 0.01; the plain average is about 0.50, which sounds respectable for a model that is useless. F1 uses a different kind of average, one that is dragged down hard by the smaller of the two. On that same useless model F1 is about 0.02. The rule of thumb: F1 is only high when both are high.

The threshold is a dial, not a fact

Here is the thing almost every beginner misses. Team A's model does not output "sick" or "healthy". Like nearly every classifier, it outputs a score between 0 and 1 — a confidence, say 0.83. Something has to turn that score into a decision, and that something is a cut-off you choose:

flag = score >= 0.5     # 0.5 is a DEFAULT, not a law of nature

That 0.5 is the , and it is where the precision-recall trade-off actually lives.

One model, infinitely many confusion matrices

Everything you computed above — the 75, the 411, the precision of 0.154 — describes the threshold, not the model. Lower the threshold to 0.3 and the model flags far more people: recall climbs, precision falls. Raise it to 0.8 and it flags only the most obvious cases: precision climbs, recall falls. The trained model never changed. Only the dial did.

This is genuinely liberating once it clicks: you do not need to retrain a model to change its precision and recall. You need to move one number, which takes about a second.

Think of it like the sensitivity dial on a metal detector:

Turn the sensitivity up and the detector beeps at every bottle cap, buried nail, and coin — you will not walk past the buried treasure (high recall), but you will be digging holes all afternoon (low precision). Turn it down and it only beeps for something big and metallic — every hole you dig is worth it (high precision), but you will stroll straight past smaller finds (low recall). Same detector, same beach. One dial.

Watch it happen. This is Team A's exact model, swept across every threshold from 0.05 to 0.95:

Precision, recall, and F1 as the threshold moves— interactive, drag & zoom

Loading chart…

One trained model, every threshold. Recall falls as the threshold rises and precision climbs to meet it — they cross near 0.65. The dashed grey line marks the default 0.5, which lands in a low-precision region here purely by accident of where someone once set a default. F1 peaks around 0.65, but the hospital would deliberately sit far to the left of that, accepting terrible precision to keep recall high.

The most useful thing on this page

If you take one practical habit away, take this: after training, sweep the threshold and look at this curve before you ship anything. It costs one loop over your validation set, it never requires retraining, and it routinely delivers a bigger improvement in the metric you actually care about than a week of architecture tuning.

Do the sweep yourself on a set small enough to check by eye — twelve patients, six sick and six healthy, sorted by the model's score:

Python · runs in your browser

What this does: Sweeps the decision threshold across twelve patients and prints the confusion matrix, precision, recall and F1 at each cut-off. Watch recall fall and precision rise as the threshold goes up — one model, one sorted list of scores, and every row is a different confusion matrix.

scores = [0.95, 0.88, 0.81, 0.74, 0.66, 0.59, 0.52, 0.44, 0.37, 0.29, 0.18, 0.07]
truth  = [   1,    1,    0,    1,    1,    0,    1,    0,    0,    1,    0,    0]   # 1 = sick

print(f"{'thresh':>7} {'TP':>3} {'FP':>3} {'FN':>3} {'TN':>3} {'precision':>10} {'recall':>8} {'F1':>7}")
for t in [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2]:
    flags = [s >= t for s in scores]
    TP = sum(1 for f, y in zip(flags, truth) if f and y == 1)
    FP = sum(1 for f, y in zip(flags, truth) if f and y == 0)
    FN = sum(1 for f, y in zip(flags, truth) if not f and y == 1)
    TN = sum(1 for f, y in zip(flags, truth) if not f and y == 0)

prec = TP / (TP + FP) if TP + FP else 1.0
    rec = TP / (TP + FN)
    f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0
    print(f"{t:>7.1f} {TP:>3} {FP:>3} {FN:>3} {TN:>3} {prec:>10.3f} {rec:>8.3f} {f1:>7.3f}")

print()
print("Recall only ever goes UP as the threshold falls — lowering the bar can never lose a catch.")
print("Precision wanders, because each extra flag might be a hit or a false alarm.")

Try to recall

Your model must never miss a fraudulent transaction, even at the cost of many false alarms. Which way do you move the threshold, and what happens to precision?

Hint: Never miss means recall must be as close to 1 as possible.

ROC and AUC — grading the model, not the threshold

The threshold sweep raises an awkward question. If precision and recall depend entirely on a dial you chose, how do you compare two models without both teams tuning their own dial and declaring victory?

The answer: judge the model across every threshold at once. That is what an does. (The strange name is wartime radar jargon — it is about "receiver operators" reading radar screens, and it tells you nothing useful about the method.)

It plots two of the rates you already know against each other:

The (TPR) on the y-axis. This is just recall, wearing a different hat: of the sick, how many did we catch?
The (FPR) on the x-axis: of the healthy, how many did we wrongly alarm?

Each point on the curve is one threshold. Sweep the threshold from 1 down to 0 and you trace the curve from the bottom-left corner (flag nobody: no catches, no false alarms) to the top-right (flag everybody: all catches, all false alarms).

How to read the shape in one glance

Up and to the left is good. The top-left corner is perfection: you caught every sick person (TPR = 1) and alarmed no healthy person (FPR = 0). The diagonal line is the coin flip — a model with no information at all, where catching 30% more of the sick costs you exactly 30% more false alarms. The further your curve bulges toward that top-left corner, the better the model separates the two groups, at every threshold simultaneously.

ROC curves — Team A versus a weaker model— interactive, drag & zoom

Loading chart…

Each curve is one model traced across all thresholds. Team A hugs the top-left corner: at a 4% false-alarm rate it already catches 75% of cases — the exact operating point from the confusion matrix above. The weaker model needs a 40% false-alarm rate to reach the same recall. The area beneath each curve compresses this into one number.

That area has a name: .

AUC = 1.0 — perfect. Every sick person scores above every healthy person.
AUC = 0.5 — the diagonal. No information whatsoever.
AUC below 0.5 — worse than chance, which almost always means your labels are flipped somewhere. (Invert the predictions and you have a good model.)

What AUC actually measures, in one sentence

Pick one sick person and one healthy person at random. AUC is the probability that the model gives the sick one a higher score. That is the entire meaning — and notice what it is not about: it never mentions a threshold, or how many people are sick, or what any score literally means. AUC grades ranking quality only. A model that scores every sick person at 0.9001 and every healthy one at 0.9000 has a perfect AUC of 1.0 while being wildly overconfident nonsense.

Compute an AUC from scratch — no library, just counting which pairs the model ranked correctly:

Python · runs in your browser

What this does: Computes AUC directly from its definition by checking every sick-healthy pair and counting how often the sick patient got the higher score. It prints the raw count of correctly-ranked pairs and the resulting AUC, so you can verify the number by hand.

The trap: AUC looks fantastic on imbalanced data

Team A's AUC is 0.965. That is a number you would put on a slide. And you already know that at its actual operating point, 85% of its alarms are false.

Both facts are true. The ROC curve is not lying — it is answering a question about ranking, and the model genuinely ranks well. The problem is what the x-axis does with a large negative class.

Why the ROC curve hides the damage

FPR divides false alarms by 9,900 healthy people. Going from 0 to 411 false alarms moves the x-axis by only 0.04 — a sliver you can barely see. But precision divides those same 411 false alarms by the 486 flags actually raised, where they are overwhelming. Same errors, same model. One denominator makes them look negligible; the other shows them for what they are.

The rule: when positives are rare, plot precision against recall instead. The precision-recall curve has no term that grows with the size of the negative class, so it cannot flatter you the same way.

The same model, judged by a precision-recall curve— interactive, drag & zoom

Loading chart…

The identical model and the identical predictions that produced a 0.965 AUC. Here the collapse is impossible to miss: precision holds above 0.9 while recall stays under 0.2, then falls off a cliff. The purple X marks the threshold actually shipped — 75% recall bought at 15% precision. Note also where the no-skill baseline sits: at 1% prevalence a random model scores 0.01 precision, so unlike ROC, a PR curve's floor moves with your class balance.

Three habits that prevent most metric disasters

Always report the no-skill baseline alongside the score. AUC's baseline is always 0.5; average precision's baseline is the prevalence — 0.01 here. A PR-AUC of 0.5 is spectacular at 1% prevalence and mediocre at 45%.
Never compare PR curves across datasets with different class balance. Their y-axes are on different scales. ROC is the one that survives resampling.
Report the operating point, not just the curve. A curve is a menu of options; a deployed model is one dish. State the threshold and the precision and recall it delivers.

A model reports AUC 0.98 on a dataset where 0.5% of examples are positive. In production, 96% of its alerts turn out to be false alarms. Which statement is correct?

Calibration — do the probabilities mean anything?

Everything so far has graded the model on ordering — did it put the right examples above the others? But a score of 0.7 looks like it is claiming something stronger: a 70% chance this person is sick. Is it?

Usually, no.

The weather forecaster test

Collect every day a forecaster said 70% chance of rain. If it rained on about 70% of those days, the forecaster is calibrated — the number means what it says. If it rained on only 20% of them, the forecaster is overconfident: still possibly great at ranking rainy days above dry ones (fine AUC!), but the number itself is not a probability. It is just a score that happens to live between 0 and 1.

is that property, and you check it with a : bucket predictions by confidence, then plot what the model claimed against what actually happened.

Here is Team A's model under that test:

Reliability diagram — what the model claimed vs what happened— interactive, drag & zoom

Loading chart…

Predictions bucketed into ten confidence bins; bins with fewer than 20 examples are dropped as too noisy to plot. The curve sags far below the diagonal, which is the signature of overconfidence: of the patients this model called 54% likely to be sick, only 5.2% were. Of those it called 65% likely, 19.6% were. This is the same model with the 0.965 AUC — excellent at ranking, and its numbers still do not mean what they appear to say.

Good ranking and good probabilities are separate skills

Team A's model has an AUC of 0.965 and an expected calibration error of 0.22. Nothing is contradictory about that. AUC only ever compares scores to other scores; it is completely unchanged if you squash every prediction through any function that preserves order. Calibration asks the different question of whether a score matches a frequency in the world. A model can be superb at one and hopeless at the other, and most are.

Think of it like a doctor who is always right about who is sicker:

Imagine a doctor who can rank any waiting room perfectly from most to least ill — but who tells every mildly unwell patient they have a 60% chance of something serious. Her rankings are flawless; her numbers are alarmist. Send her patients for tests in her order and you would do very well. Let a hospital use her percentages to allocate ICU beds and you would waste a great many beds. That is the difference AUC cannot see.

Try to recall

Why does calibration matter if the model already ranks correctly?

Hint: Think about what happens after the prediction, when a decision has to be made.

Honest evaluation — the part that decides whether any of this is real

Every metric above assumes one thing: that the numbers were measured on data the model had never seen. Break that assumption and every score on this page becomes fiction — and the failure is silent, because a leaked model looks better, not worse.

Why you cannot grade a model on its own homework

A large model can simply memorise its training set. Score it on that same data and you measure memory, not understanding — like grading students on the exact practice problems they had the answer key to. Everyone scores 100%, and you learn nothing about who can handle a new question.

The standard defence is three splits, with three distinct jobs:

Training set — the model fits its parameters on this. It sees it thousands of times.
Validation set — you use this to choose hyperparameters, architectures, and the decision threshold. The model never trains on it, but you effectively do.
Test set — touched once, at the very end, to produce the number you report. Every look costs you a little of its honesty.

The validation set wears out

Here is the subtlety that catches experienced people. Try 200 architectures and pick the best on validation, and that winning score is optimistically biased — with 200 tries, some model got lucky on that particular split. You have not overfitted the model to the validation data; you have overfitted, through your choices. This is why the untouched test set exists, and why re-tuning after peeking at test results quietly turns your test set into a second validation set.

Cross-validation — when one split is not enough

A single train/test split has its own problem: you only measured once. On a small dataset, which examples happened to land in the test set can swing the score enormously.

fixes this by measuring $k$ times and averaging.

Think of it like marking an exam with five different graders:

One grader might be harsh, or might happen to get the easy papers. Five independent graders give you both a better central estimate and — just as importantly — a sense of how much graders disagree. If the five marks are 61, 62, 60, 63, 61 you can trust the average. If they are 40, 85, 55, 70, 50 you have learned that a single mark means very little, which is itself the most valuable thing the exercise told you.

Five folds of the same model on the same small dataset— interactive, drag & zoom

Loading chart…

One model, one dataset of 200 examples, five folds. The scores run from 0.550 to 0.700 — a 15-point spread produced by nothing but which rows landed in which fold. Report fold 5 alone and the model looks decent; report fold 1 and it looks near-useless. The honest summary is 0.625 plus or minus 0.063, and that spread is the reason to run cross-validation at all.

You normalise your features using the mean and standard deviation of the entire dataset, then split into train and test. What is wrong?

Metrics for other kinds of problems

The four boxes only exist for yes/no predictions. Two other families come up constantly.

Regression — when the model predicts a number, not a class:

MAE ( $\frac{1}{n}\sum|y_i - \hat{y}_i|$ ) — the average size of the error, in the units of whatever you are predicting. $4,200 off on average is a sentence anyone can act on. Treats all errors proportionally, so it shrugs off outliers.
RMSE ( $\sqrt{\frac{1}{n}\sum(y_i - \hat{y}_i)^2}$ ) — squares each error before averaging, then square-roots back into the original units. The squaring means one enormous miss counts for much more than several small ones. Use it when big errors are disproportionately costly; use MAE when they are not.
R² — the fraction of the variance in the target that the model explains. $1$ is perfect, $0$ means you did no better than always predicting the mean, and negative values are possible and mean you did worse than that.

Here $y_i$ is the truth for example $i$ , $\hat{y}_i$ (read "y-hat") is the model's prediction, and the hat is the standard mark for an estimated quantity.

Object detection — where a prediction is a box, so "correct" is a matter of degree. measures how well two boxes overlap, and mAP averages precision across recall levels and classes. Both are built directly on the precision and recall you learned above — see Object Detection for the full treatment.

Choosing the metric — the actual decision

Your situation	Report this	Because
Balanced classes, equal error costs	Accuracy	It genuinely means what it looks like here
Rare positive class	Precision, recall, PR-AUC	Accuracy and ROC both flatter you
Missing a positive is expensive	Recall, or $F_2$	Explicitly weights misses above false alarms
A false alarm is expensive	Precision, or $F_{0.5}$	Explicitly weights false alarms above misses
Comparing models before a threshold exists	ROC-AUC	Threshold-free, and stable across class balance
The probability feeds a cost calculation	ECE, Brier score	Ranking is not enough when the number gets multiplied
Small dataset	Cross-validated mean and std	A single split has too much variance to trust
Predicting a number	MAE or RMSE	MAE for robustness, RMSE when big errors hurt more

The question that comes before the metric

Notice that every row is decided by what a mistake costs, not by anything about the model. So write down the two costs first — what does a false alarm cost us, and what does a miss cost us? — and the right metric follows almost mechanically. Teams that skip this step end up optimising accuracy by default, which is how the hospital ships Team B.

Explain it yourself

Explain to a friend why a model can be 99% accurate and still useless, then explain precision and recall using the fishing-net picture — no formulas. Finish by saying what happens to each one when you lower the decision threshold. If you stall on any part, that is the exact section to reread.

Recap — the key ideas

Accuracy hides which mistake happened, and on imbalanced data the always-predict-the-common-class model sets a very high floor. Compute that no-skill baseline before believing any score.
The confusion matrix (TP, FP, FN, TN) is the raw material; every classification metric is a ratio of some of those four counts to some others.
Precision = of what you flagged, how much was right. Recall = of what was there, how much you found. Same numerator, different denominators — which is why they pull against each other. F1 is their harmonic mean, high only when both are.
The decision threshold is a dial you choose after training, not a property of the model. Sweeping it moves precision and recall a long way for free.
ROC-AUC grades ranking across all thresholds and equals the probability a random positive outscores a random negative — but it flatters models on rare positives, where a precision-recall curve tells the truth.
Calibration asks whether a 0.7 really means 70%. It is independent of ranking quality: measure it with ECE or the Brier score, and fix it cheaply with temperature scaling.
Numbers are only real if measured on untouched data. Use train/validation/test, use cross-validation with its standard deviation on small data, and hunt for leakage whenever a result looks too good.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: answer the questions below from memory before scrolling back up — pulling an answer out beats re-reading the page that contains it.
• Spaced repetition: mark this topic complete and it joins your Review queue, resurfacing right before you would have forgotten it.
• Interleaving: mix these with problems from Machine Learning and Probability & Statistics rather than grinding metrics in one block — messier practice, sturdier memory.

By hand: a model flags 50 items; 30 are correct. There were 120 true positives in the data. Compute precision, recall, and F1. Now say in one sentence what this model is good and bad at.
Find the baseline: for a dataset that is 3% positive, compute the accuracy of the always-negative model, and the average precision of a random-scoring model. Any result you report must beat both.
Sweep it: take any classifier you have trained, compute precision and recall at 20 thresholds, and pick the one that satisfies a constraint you invent — say, precision at least 0.9. Notice you never retrained anything.
Break it deliberately: normalise features using the whole dataset before splitting, and see how much the score inflates versus doing it correctly. Seeing leakage inflate a number yourself is the fastest way to learn to smell it.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: The starter builds a small imbalanced dataset and reports accuracy for a real model and for a do-nothing model. Run it and confirm the useless model wins. Then do the TODOs: (1) write the precision and recall functions, (2) sweep the threshold from 0.1 to 0.9 and find the lowest threshold whose precision is at least 0.5 — that is the operating point you would actually ship.

editor

import numpy as np

# 200 patients, 10% of them sick. The model scores each one from 0 to 1.
rng = np.random.default_rng(0)
truth = np.r_[np.ones(20, int), np.zeros(180, int)]
scores = np.r_[rng.beta(6, 3, 20), rng.beta(2, 6, 180)]   # sick tend to score higher

def confusion(scores, truth, t):
    """Return TP, FP, FN, TN at threshold t."""
    flag = scores >= t
    TP = int(np.sum(flag & (truth == 1)))
    FP = int(np.sum(flag & (truth == 0)))
    FN = int(np.sum(~flag & (truth == 1)))
    TN = int(np.sum(~flag & (truth == 0)))
    return TP, FP, FN, TN

TP, FP, FN, TN = confusion(scores, truth, 0.5)
print("confusion matrix at t=0.5 ->", f"TP={TP} FP={FP} FN={FN} TN={TN}")
print(f"real model accuracy      : {(TP + TN) / len(truth):.3f}")
print(f"always-healthy accuracy  : {np.mean(truth == 0):.3f}   (caught 0 of {truth.sum()} sick)")

# TODO 1: fill these in using the four counts.
def precision(TP, FP):
    return 0.0   # <- of everything flagged, what fraction was right?

def recall(TP, FN):
    return 0.0   # <- of everything truly sick, what fraction did we find?

# TODO 2: sweep the threshold and print precision and recall at each step.
#         Then find the LOWEST threshold whose precision is at least 0.5,
#         and report the recall you get there. That is your shipping decision.
for t in np.arange(0.1, 0.95, 0.05):
    TP, FP, FN, TN = confusion(scores, truth, t)
    # print(f"t={t:.2f}  precision={precision(TP, FP):.3f}  recall={recall(TP, FN):.3f}")

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Next, put these metrics to work choosing between real models in Ensembles & Boosting, or go back and strengthen the foundations in Probability & Statistics.