Machine Learning
Supervised learning, bias–variance, and the core ML workflow — taught from zero. Start with what it means for a machine to learn from examples, then build up to loss functions, generalization, the bias–variance trade-off, and cross-validation.
Start here — what a machine that "learns" actually does
Suppose you have to write a program that decides whether an email is spam.
The obvious approach is to write rules: if it contains "FREE MONEY", flag it. If the sender is unknown and there are five exclamation marks, flag it. You'd write a hundred rules, spammers would work around all of them, and you'd write a hundred more. This is exhausting and it never ends.
Machine learning is the other approach: don't write the rules — show examples and let the program work the rules out. Hand it 50,000 emails that humans already labelled spam or not spam, and let it discover for itself what spam tends to look like.
Machine learning is fitting a rule to examples, in a way that also works on examples you have not seen yet. That second half is the entire difficulty. Any program can memorize the answers it was shown; the job is to get the next one right.
Nobody ever gave you a rulebook for recognizing your friend's handwriting — no "the letter g has a 30-degree loop." You just saw a lot of it, and now you can spot a note they wrote at a glance, including words you have never seen them write before. You learned a rule from examples that generalizes to new cases. That is exactly what we are about to make a computer do.
It teaches from first principles and assumes nothing. Flip the Depth switch at the top for the formal notation and derivations — they also open automatically once you have finished the prerequisites (Linear Algebra, Probability & Statistics, and Optimization). Nothing is hidden for good.
The setup — examples in, a rule out
Every supervised ML problem is described with the same four words. Learn these four and you can read any ML paper's problem statement.
- A is one input number describing an example.
- A is the correct answer for that example.
- A is the pile of (features, label) pairs you learn from.
- A is the rule you end up with: features go in, a prediction comes out.
Because every example comes with its correct answer attached — a supervisor telling you the truth — this whole setting is called .
| Problem | Features (inputs) | Label (the answer) |
|---|---|---|
| House pricing | size, bedrooms, postcode, age | the price it sold for |
| Spam filter | words in the email, sender, link count | spam / not spam |
| Medical triage | age, blood pressure, test results | condition present / absent |
| Photo tagging | the pixel values | which of 1000 objects it is |
Look at the label column and you will notice it comes in exactly two flavours, and this split organizes the whole field:
- The label is a number on a scale (a price, a temperature, a probability of rain). Predicting it is .
- The label is one of a fixed set of categories (spam or not, cat or dog or bird). Predicting it is .
You want to predict how many minutes late a train will be. Regression or classification?
Hint: Look at what kind of thing the answer is.
Fitting a rule — five houses and a straight line
Time to actually do it. Here are five houses. The feature is size (in hundreds of square feet); the label is the price it sold for (in thousands).
| Size | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Price | 150 | 200 | 260 | 300 | 340 |
Our model will be the simplest thing that could possibly work — a straight line: price = intercept + slope × size. The two numbers intercept and slope are this model's : the knobs we get to turn.
So which line? Eyeballing the table, prices go up by about 50 per unit of size and start near 100, so a reasonable guess is price = 100 + 50 × size. But "reasonable guess" is not an algorithm. We need a number that scores how bad a line is, so we can hunt for the line that scores best.
Pick a line. For each house, ask: how far off was the prediction? Square each of those misses (so that being 10 too low is just as bad as 10 too high, and so that big misses hurt disproportionately), then average them. That single number is the . Lower is better, and the best line is defined to be the one with the lowest loss. That is the whole move that turns a vague wish into a math problem.
Candidate A — the eyeball line, price = 100 + 50 × size:
| Size | Actual | Predicted | Miss | Miss² |
|---|---|---|---|---|
| 1 | 150 | 150 | 0 | 0 |
| 2 | 200 | 200 | 0 | 0 |
| 3 | 260 | 250 | +10 | 100 |
| 4 | 300 | 300 | 0 | 0 |
| 5 | 340 | 350 | −10 | 100 |
Average of the squared misses: .
Candidate B — the line price = 106 + 48 × size:
| Size | Actual | Predicted | Miss | Miss² |
|---|---|---|---|---|
| 1 | 150 | 154 | −4 | 16 |
| 2 | 200 | 202 | −2 | 4 |
| 3 | 260 | 250 | +10 | 100 |
| 4 | 300 | 298 | +2 | 4 |
| 5 | 340 | 346 | −6 | 36 |
Average: .
Candidate B wins, 32 to 40 — even though it misses every single house, while candidate A nailed three of them exactly. Spreading the error thinly across all five beats being perfect on three and badly wrong on two. That is the squaring at work, and it is the first genuinely counter-intuitive thing about machine learning.
Candidate B is not a lucky guess: it is the provably best straight line for this data, and the recipe for finding it is called .
Rather than take that on faith, grab a line and try to beat it yourself. In the applet below, drag the red line over the data. The panel reports the sum of squared errors as you move — try to drive it down to the stated minimum, and notice how the squared penalty punishes one big miss far more than several small ones.
Why do we square the misses instead of just adding them up?
Hint: What happens to a +10 miss and a −10 miss if you simply add them?
Run it and see the same two numbers fall out:
Generalization — the only thing that actually matters
Here is where machine learning stops being curve fitting and becomes its own subject.
We chose our line by making the error small on the five houses we already knew the answers to. But we do not care about those five houses. We already know what they sold for. We care about the next house — the one nobody has priced yet.
is performance on unseen data, and it is the only score that counts.
Two students revise from the same ten past papers. One memorizes the ten answer sheets and can recite them perfectly. The other works out why each answer is right. Both score 100% on the past papers — so the past papers cannot tell them apart. Then the real exam arrives with different questions, and the memorizer is destroyed. Training error is the past paper. You need a real exam.
The real exam is a : before you train anything, take a random slice of your data — typically 20% — and lock it in a drawer. Train on the rest. Only at the very end, open the drawer and measure. That number is your honest estimate of how the model will do in the world.
Watch a model go from too simple to too clever
Below are 12 noisy measurements of some real process (grey dots), the true underlying relationship they came from (dashed line — in real life you never get to see this), and three models fitted to those same 12 dots. The only difference between the models is how many bends the curve is allowed to have.
Those two failures have names, and they are the two things that can go wrong with any model ever built:
- — the straight line. It is wrong everywhere, including on the data it was trained on. It has not even learned the training set.
- — the degree-9 curve. It passes far closer to the training dots, but it achieved that by chasing random noise, and the wiggles it invented are not in the real world.
The dial between them is the model's .
A large gap between training error and test error is overfitting, and it is the single most common failure in applied ML. A model with 99% training accuracy and 71% test accuracy has not learned the task — it has memorized your training set. If both numbers are bad, that is underfitting instead, and it needs the opposite cure.
The curve everyone in ML has burned into their memory
Now fit every polynomial degree from 0 to 9 to those same 12 points, and measure two errors each time: on the 12 training points, and on 2000 fresh points from the same process that no model ever saw.
Training error keeps dropping as you add capacity, but test error starts rising. What is the model doing with the extra capacity after the turning point?
Hint: What is in the training data besides the real pattern?
Your model gets 0.02 training error and 0.31 test error. What is the problem, and which fix is most likely to help?
Bias and variance — naming the two ways to be wrong
Underfitting and overfitting are not two unrelated bugs. They are the two ends of a single trade-off, and giving them precise names is the most useful piece of vocabulary in classical ML.
Archer One has a bent sight: every arrow lands in a tight cluster, but the cluster sits well off to the left of the bullseye. Consistent, and consistently wrong — that is high bias.
Archer Two's aim is true on average, but they twitch: arrows scatter all over the target, and the centre of the scatter is the bullseye. No systematic error, but no reliability either — that is high variance.
Both archers miss. They need completely opposite coaching, which is why telling them apart matters.
Translated back to models:
- is the error you get from your model being too simple to represent the truth. A straight line cannot bend, so it will miss a curved pattern the same way every time, whatever data you train it on.
- is how much your fitted model swings around when you swap in a different training sample. A very flexible model chases whatever noise it happens to see, so a different sample produces a visibly different model.
Now look at what that means in practice. Below, the same two model types are each fitted three times, on three different random 12-point samples drawn from the same underlying process.
Reach for a more flexible model to cut bias, and you hand it the freedom to chase noise — variance goes up. Simplify the model to cut variance, and it can no longer bend to the real pattern — bias goes up. Turning the capacity dial trades one for the other, and the best model is not the one that eliminates either but the one that minimizes the sum. That is why it is called a trade-off, and why "just use the biggest model" is not an answer.
Here is that sum, measured. The same experiment as before was repeated across 200 independently drawn training sets per degree, and the average squared error split into its two parts.
You train three models on the same data and measure:
| Model | Training error | Test error |
|---|---|---|
| A | 0.42 | 0.44 |
| B | 0.19 | 0.21 |
| C | 0.03 | 0.38 |
- Model A is bad on data it has already seen. It has not even learned the training set, so extra data will not help. High bias — underfitting. Add capacity or better features.
- Model C is nearly perfect on training data and far worse on the test set — a gap of 0.35. It memorized. High variance — overfitting. Simplify it, add data, or regularize.
- Model B has a small gap and a low level. Ship model B, then ask whether 0.21 is near the noise floor — if so, you are done; if not, there is still bias left to remove.
Notice you diagnosed all three from two columns of numbers, without looking at the model at all. This table is the first thing to build for any ML project.
You double the size of your training set and the test error barely moves. What does that suggest?
Choosing the dial — validation and cross-validation
The test-error curve told us degree 5 was best. But there is a problem: we used the test set to make that choice.
The moment you pick a model because it scored well on the test set, that score stops being an honest estimate of future performance. You have quietly fitted your choices to the test data — try twenty models and the winner is partly just the one that got lucky on those particular points. The test set is a one-shot resource: you open the drawer once, at the very end, after all decisions are locked.
The fix is a third split. Cut the data into three: train (fit the parameters), (compare models and tune the dials), and test (open once, at the end).
Things like the polynomial degree are , and the validation set exists precisely so you can tune them without burning your test set.
But if data is scarce — and it usually is — carving out a validation set hurts twice: less data to train on, and a small, noisy validation score. solves both.
Five friends revise together. Instead of one person permanently playing quizmaster and never getting quizzed, they take turns: each round, one asks the questions and the other four answer. After five rounds, everyone has been quizzed exactly once and everyone has studied four times. Nobody's revision was wasted on being quizmaster, and you get five independent scores instead of one.
Split the 12 training points into 4 folds of 3. Then train 4 times, each time holding out one fold to score on. For degree 3, the four held-out RMSEs come out as:
| Held-out fold | 1 | 2 | 3 | 4 | average |
|---|---|---|---|---|---|
| RMSE | 0.434 | 0.399 | 0.255 | 0.200 | 0.322 |
Repeat that whole procedure for each candidate degree:
| Degree | 0 | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|---|
| CV score | 0.685 | 0.571 | 0.709 | 0.322 | 0.368 | 0.789 | 0.843 |
Degree 3 wins, and note two things. First, the individual fold scores for degree 3 range from 0.200 to 0.434 — more than a factor of two apart. A single validation split could easily have handed you either extreme; averaging four is what makes the comparison trustworthy. Second, the scores explode past degree 4 (0.789, 0.843) far more dramatically than the test-error curve did, because each fold trains on only 9 points, and a flexible model given 9 points goes wild. Cross-validation on small data is slightly pessimistic about complex models for exactly this reason.
- Test-set peeking. Every time you look at the test score and change something, a little of its honesty leaks away. Tune on validation or cross-validation; touch test once.
- Data leakage. If you scale, impute, or select features using statistics computed over the whole dataset before splitting, information from the test set has crept into training. Your reported score will be optimistic and the model will disappoint in production. Fit every preprocessing step on the training fold only, then apply it to the held-out fold. See Feature Engineering.
Why is a 5-fold cross-validation score usually more trustworthy than a single train/validation split of the same data?
Hint: Think about how much of the data each score is based on, and how many scores you get.
Classification — when the answer is a category
Everything so far predicted a number. Now the label is a category, and two things change: how the model produces its answer, and how you score it.
A classifier does not output "spam." It outputs a probability — 0.93 spam — and you convert that to a decision with a threshold. The line in feature space separating the two verdicts is the .
To turn an unbounded score into a probability, we squash it through the . Feed in any number from −∞ to +∞, get out something between 0 and 1.
Scoring a classifier — why accuracy lies
Build a spam filter, run it on 1000 emails of which 100 are truly spam, and lay out the results in a .
From the matrix: 80 caught spam (true positives), 20 missed (false negatives), 30 wrongly flagged (false positives), 870 correctly delivered (true negatives).
- Accuracy — what fraction of all verdicts were right? . Sounds excellent.
- Now the sanity check. Consider the do-nothing model that labels every single email "not spam." It catches no spam whatsoever, and scores accuracy. Our real filter beats a model that does literally nothing by five percentage points.
- Precision — of the emails we flagged, what fraction really were spam? . So more than a quarter of everything sent to the spam folder is a legitimate email.
- Recall — of the spam that existed, what fraction did we catch? . One spam in five still lands in the inbox.
Accuracy said 95%. Precision and recall said "one in four flagged emails is an innocent bystander, and one in five spams gets through." Accuracy is a weighted average dominated by the majority class, which is why it hides the failure completely whenever the classes are imbalanced — fraud, disease, defects, and most problems worth solving.
Moving the threshold trades them off directly: flag more aggressively and recall rises while precision falls. Which you want depends entirely on the cost of each mistake. For a cancer screen, a missed case is catastrophic and a false alarm means one more test — maximize recall. For a spam filter, a lost job offer is far worse than one spam in the inbox — favour precision. This is a product decision wearing a statistics costume. Evaluation & Metrics goes through the full toolkit including ROC curves and calibration.
A fraud detector reaches 99.2% accuracy on a dataset where 0.8% of transactions are fraudulent. What should you check first?
The workflow — how this actually goes, start to finish
Every applied ML project, from a weekend notebook to a production system, runs this loop. The modelling is step 5, and it is rarely where the time goes.
- Frame the problem. What is the label? What decision does the prediction feed? What does a mistake cost? Getting this wrong makes everything downstream irrelevant.
- Get and inspect the data. Look at it. Plot it. Count the missing values. Find the duplicates. Notice that a third of the timestamps are in a different timezone.
- Split first — before anything else. Carve out the test set now, while you still know nothing, so nothing you learn can leak into it.
- Build features. Encode categories, scale numbers, handle missing values — fitting every transform on the training split only. See Feature Engineering.
- Establish a baseline, then model. Always start with something trivially simple — predict the mean, or the majority class, or a linear model. If your neural network cannot beat it, you have learned something important. Then try real models from Classical Models and Ensembles.
- Tune on validation / cross-validation. Compare capacities and hyperparameters. Diagnose bias vs variance from the train/validation gap and act on the diagnosis.
- Evaluate once on the test set, with metrics that match the actual cost of errors. Report honestly, including where the model fails.
- Ship, monitor, retrain. The world drifts; a model trained on last year's data slowly stops being right. See MLOps.
- Leaking test information into training through preprocessing, or through a feature that quietly encodes the answer (a "date_case_closed" column that only exists for resolved cases).
- Optimizing a metric nobody asked for — 99% accuracy on a problem where recall was what mattered.
- No baseline, so nobody can tell whether the model is adding anything at all.
- Reporting the best of twenty runs as if it were a single honest result.
- Assuming more capacity is always better. The test-error curve above is the counterexample; so is every over-engineered model that lost to logistic regression.
Explain to a friend why a model that gets every training example perfectly right might be worse than one that gets some wrong. Use the exam-revision picture, and then name the two ways a model can be wrong and what you would do about each. If you cannot say what distinguishes bias from variance without looking, that is exactly the section to reread.
- Supervised learning fits a rule to labelled examples: features in, label out. If the label is a number it is regression; if it is a category it is classification.
- Training means minimizing a loss — a single number scoring badness. For regression that is usually MSE; for classification, cross-entropy.
- The goal is never training error. It is generalization: performance on data the model has never seen. Hold out a test set before you start, and open it once.
- Underfitting is a model too simple to capture the pattern (high bias); overfitting is a model flexible enough to memorize noise (high variance). The tell is the gap between training and test error.
- Squared error splits exactly into bias² + variance + irreducible noise. Diagnosing which one dominates tells you whether to add capacity, add data, or stop.
- Choose hyperparameters on a validation set or by k-fold cross-validation — never on the test set.
- Accuracy hides failure on imbalanced problems. Read the confusion matrix, then precision and recall, chosen by what each mistake actually costs.
Practice — and how to make it stick
Three research-backed habits, built into this platform:
• Retrieval practice: attempt the problems below before scrolling back up — pulling an answer out of memory builds far more durable knowledge than recognizing it on the page.
• Spaced repetition: mark this topic complete and it joins your Review queue, resurfacing right before you would have forgotten it.
• Interleaving: mix these with problems from Optimization and Probability & Statistics rather than grinding one type — messier practice, sturdier memory.
- By hand: add a sixth house — size 6, price 500 — to the five in the worked example. Without computing anything, predict what happens to the least-squares slope, then recompute it and check. What does this tell you about outliers and squared loss?
- By hand: a classifier on 200 examples (40 positive) produces 28 true positives, 12 false negatives, 24 false positives. Compute accuracy, precision, and recall. Which single number would you report to a stakeholder, and why?
- From scratch: implement k-fold cross-validation in NumPy without a library, and use it to choose the polynomial degree in the lab below.
- Diagnose: train a model until training error is near zero, then plot training and validation error against training-set size (a learning curve). A converged pair that is high means bias; a persistent gap means variance. Confirm you can tell them apart from the picture alone.
- Read like a scientist: skim Domingos, A Few Useful Things to Know about Machine Learning, and find the section arguing that more data beats a cleverer algorithm. Decide whether you agree.
Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.
Next, meet the models themselves in Classical Models, learn to feed them properly in Feature Engineering, and score them rigorously in Evaluation & Metrics.