Probability & Statistics

Distributions, expectation, and reasoning under uncertainty — taught from zero, then built up to the Gaussian, Bayes' rule, sampling error, and the likelihood that every ML loss function is secretly made of.

beginner#math#probability#statistics

Start here — what this is really about

Forget the formulas for a moment. This whole subject is two questions, pointing in opposite directions:

"I know how the world works — what will I see?" — that's probability. You know the coin is fair; you predict roughly half heads.
"I've seen some data — what is the world actually like?" — that's statistics. You saw 61 heads in 100 flips; you ask whether the coin is really fair.

Probability runs forward from cause to data. Statistics runs backward from data to cause. Machine learning is the second question answered by machine: here is a pile of data, what rule produced it?

How to read this page

The page adapts to you. By default it teaches from first principles — no symbols until you already understand the idea they stand for. When you're ready, flip the Depth switch at the top to reveal the formal notation, derivations, and edge cases. Nothing is hidden for good; the deeper material sits behind "Go deeper" panels so you can open it the moment you're curious.

Probability — one number for "how often"

A is just a number between 0 and 1 that answers "how often?"

0 → never happens.
1 → always happens.
0.5 → happens half the time.

That's the entire meaning. When someone says "there's a 30% chance of rain," they're saying: on days that look like this one, it rains about 3 times in 10.

Think of it like a giant bag of tickets:

Imagine every possible outcome written on a ticket and dumped into a huge bag — one ticket per equally-likely way the world could go. The probability of something is the fraction of tickets in the bag that say it. Roll a die: six tickets, one says "4", so P(4) = 1/6. Every probability question is secretly a counting-tickets question.

Two pieces of vocabulary, and they're both simpler than they sound. The is the list of everything that could happen. An is the part of that list you care about ("rolled an even number" = the outcomes 2, 4, 6).

A probability, counted by hand

Roll one fair six-sided die. What's the probability of rolling higher than 4?

List the sample space — every outcome: 1, 2, 3, 4, 5, 6. That's 6 tickets in the bag.
List the event — outcomes higher than 4: 5, 6. That's 2 tickets.
Divide — favorable tickets over total tickets: 2 / 6 = 1/3 ≈ 0.333.

So P(higher than 4) ≈ 33%. Notice we never needed a formula — we counted.

Try to recall

A bag holds 3 red marbles and 5 blue. What is the probability of drawing red, and why?

Hint: Count the tickets.

But wait — where does "half the time" actually show up?

Flip a fair coin four times and you might get 3 heads. Does that break P(heads) = 0.5? No — and seeing why is the most important habit in this subject. A probability is a promise about the long run, not about the next few tries.

Watch it happen. Below, two separate runs of coin flipping, each plotting the fraction of flips so far that came up heads. Early on the lines swing wildly. Then they get pulled in toward 0.5 and stay there.

The law of large numbers, in action— interactive, drag & zoom

Loading chart…

Two independent runs of 500 fair coin flips. After 5 flips the running fraction is anywhere from 0.33 to 0.6; after 500 both runs sit within a couple of percent of 0.5. A probability is a statement about where the line ends up, not about where it starts.

Small samples lie; large samples tell the truth slowly

This drifting-toward-the-truth is called the law of large numbers, and it is the bridge between probability and statistics. It's also the single most useful thing to remember when reading an ML result: an accuracy measured on 20 test examples is the wild left-hand side of that chart, and an accuracy measured on 20,000 is the calm right-hand side. Same math, wildly different trustworthiness.

Distributions — the whole list of what could happen

A single probability answers one question. A answers all of them at once: it's the full table of every possible value and how likely each is.

Think of it like a weather forecast for the whole week:

"30% chance of rain tomorrow" is one probability. The whole week's forecast — Monday 30%, Tuesday 10%, Wednesday 80% — is a distribution. It doesn't tell you what will happen; it tells you the complete shape of what might.

The quantity being described gets a name: a . If $X$ is the number rolled on a die, the distribution of $X$ is the six-row table 1 → 1/6, 2 → 1/6, …, 6 → 1/6.

Building a distribution by counting — 10 coin flips

Flip a fair coin 10 times and let $X$ be the number of heads. $X$ can be anything from 0 to 10, but not with equal chance. Why not?

There is exactly one way to get 0 heads: TTTTTTTTTT.
There are ten ways to get exactly 1 head: the single head can be in any of the 10 slots.
There are 252 ways to get exactly 5 heads — vastly more arrangements land in the middle.
Every one of the $2^{10} = 1024$ possible flip-sequences is equally likely, so the probability of $k$ heads is (number of arrangements with k heads) / 1024.

For 5 heads: $252/1024 \approx 0.246$ . For 0 heads: $1/1024 \approx 0.001$ . The middle is likely because there are more ways to be in the middle — that one sentence explains most of the bell shapes in this entire lesson.

Distribution of heads in 10 fair coin flips— interactive, drag & zoom

Loading chart…

The exact distribution — these eleven bars are the complete answer to what happens when you flip a fair coin ten times, and they add up to exactly 1. Getting 5 heads is about 250 times more likely than getting 0, purely because there are 252 arrangements that give 5 heads and only 1 that gives none.

This shape has a name — the — but the name matters far less than the picture: counting arrangements piles probability up in the middle.

Try to recall

Why is getting 5 heads out of 10 much more likely than getting 10 heads out of 10?

Hint: Count the ways each could happen.

A continuous PDF has value p(x) = 2.5 at some point x. Is something wrong?

Run this to build the coin-flip distribution yourself — by simulating, and then by exact counting, so you can watch the two agree:

Python · runs in your browser

What this does: Flips 10 coins 20,000 times and tallies how often each number of heads shows up, then compares those measured fractions to the exact arithmetic answer from counting arrangements. The two columns agree to about two decimals — that agreement IS the law of large numbers, and it is why simulation is a legitimate way to answer probability questions you cannot solve on paper.

Expectation — the long-run average

If you rolled a die a million times and averaged all the numbers, what would you get? That number is the (or mean, or expected value).

It is a weighted average, not a prediction

Expectation is every possible value, weighted by how often it happens. It is not a forecast of any single outcome — the expected value of a die roll is 3.5, a number the die can never show. Think of it as the balance point of the distribution, not as a guess.

Think of it like the balance point of a see-saw:

Picture the distribution's bars as weights placed along a plank — heavier bars where the probability is higher. The expectation is the single point where the plank balances. Push probability out to the right and the balance point slides right. That's all a mean is: a center of mass.

The expected value of a die roll

Each face 1–6 has probability 1/6. Multiply each value by its probability and add:

$1 \times \tfrac16 + 2 \times \tfrac16 + 3 \times \tfrac16 + 4 \times \tfrac16 + 5 \times \tfrac16 + 6 \times \tfrac16$
Factor out the shared $\tfrac16$ : $\tfrac16 (1+2+3+4+5+6) = \tfrac16 \times 21$
$= 3.5$

So the average roll is 3.5 — the balance point of six equal weights, sitting exactly between 3 and 4. The die never shows 3.5, and that's fine: expectation describes the pile, not the ticket.

Try to recall

A game pays you $10 with probability 0.1 and $0 otherwise. What is your expected payout per play, and would you pay $2 to play?

Hint: Weight each payout by its probability.

Variance — how spread out is it?

Two classes both average 70% on a test. In one, everyone scored between 68 and 72. In the other, half scored 40 and half scored 100. Same mean, completely different stories. The mean alone is never enough; you also need the spread.

Measure distance from the mean, then average it

To capture spread, ask: how far is a typical value from the mean? Take each value's distance to the mean, square it (so that being 5 below counts the same as 5 above, instead of cancelling out), and average those squares. That average is the .

Squaring makes the units awkward — squared points, squared dollars — so we usually take the square root and get back to normal units. That's the , the number people actually quote.

Think of it like a shotgun blast versus a rifle shot:

Both are aimed at the bullseye, so both have the same mean. The rifle puts every pellet within a centimetre — tiny variance. The shotgun sprays across the whole target — huge variance. Same aim, very different reliability. When someone reports a model's accuracy without an error bar, they've told you the aim and hidden the spread.

Variance of a die roll, step by step

The mean is 3.5 (from the last section). Now measure each face's squared distance from 3.5:

Distances: 1-3.5 = -2.5, 2-3.5 = -1.5, 3-3.5 = -0.5, 4-3.5 = 0.5, 5-3.5 = 1.5, 6-3.5 = 2.5
Square them (negatives become positive): 6.25, 2.25, 0.25, 0.25, 2.25, 6.25
Average the squares, each with probability 1/6: (6.25 + 2.25 + 0.25 + 0.25 + 2.25 + 6.25) / 6 = 17.5 / 6 ≈ 2.917
Standard deviation = √2.917 ≈ 1.71

So a die roll sits about 1.71 away from 3.5 on a typical roll — which matches intuition: usually a point or two off centre, occasionally 2.5 off. Notice step 2: without squaring, the distances would have summed to exactly zero and told us nothing.

Try to recall

Why do we square the distances from the mean instead of just averaging them?

Hint: What happens to the positives and negatives?

Python · runs in your browser

What this does: Checks the two variance identities numerically on a fair die. First it computes the variance the definition's way (average the squared distances from the mean), then the shortcut way (mean of squares minus square of the mean) — they match exactly. Then it shows the n vs n-1 difference on a small sample, which is why NumPy and pandas can disagree about the same data.

The Gaussian — the bell curve, and why it is everywhere

Heights, measurement errors, exam scores, the noise in a sensor, the initial weights of a neural network: all bell-shaped. Most values cluster near the middle, and the further out you go the rarer things get, symmetrically on both sides.

is the name of that shape, and its most useful property is how little you need to know to pin it down: just two numbers. The mean $\mu$ says where the peak sits; the standard deviation $\sigma$ says how wide it is. That's the entire distribution.

Why nature keeps producing this shape

A person's height isn't caused by one thing — it's hundreds of genes and childhood factors, each nudging up or down a little. Whenever a quantity is a sum of many small independent influences, the pile-up-in-the-middle effect from the coin-flip section takes over and the result is a bell. That's not a coincidence about heights; it's a theorem, the central limit theorem, and we'll watch it happen below.

Think of it like a crowd leaving a stadium:

Everyone spills out of one exit and drifts. A few end up far left, a few far right, but most stay near the middle where all the small random jostles roughly cancelled. Take a photo from above and you've drawn a bell curve — the middle is crowded because there are more ways to end up near the middle than to end up at either extreme.

Get a feel for the two knobs before you meet the formula. Drag the sliders: the mean slides the curve sideways; the standard deviation makes it wide-and-flat or narrow-and-tall. Notice that widening it also lowers the peak — the total area is always exactly 1, so spreading out means flattening down.

Normal distribution — change the mean and the standard deviation— GeoGebra, drag & exploreOpen on GeoGebra →

Loading interactive visualization…

Move the mean slider and watch the whole bell slide left and right without changing shape. Move the standard-deviation slider and watch it stretch wide or pinch narrow — always keeping the area underneath equal to 1. These two numbers are the entire distribution.

Reading a Gaussian without any formula — the 68-95-99.7 rule

Adult male heights in a population are roughly Gaussian with mean 175 cm and standard deviation 7 cm. For any Gaussian, these three facts hold:

About 68% of values fall within 1 standard deviation of the mean → within 175 ± 7, i.e. 168–182 cm.
About 95% fall within 2 standard deviations → 175 ± 14, i.e. 161–189 cm.
About 99.7% fall within 3 standard deviations → 175 ± 21, i.e. 154–196 cm.

So a 196 cm man is at the 3-sigma edge: roughly 1 in 740 people on the high side. You just did quantitative reasoning about a whole population using two numbers and one memorized rule — no integrals.

Try to recall

A test's scores are Gaussian with mean 60 and standard deviation 10. Roughly what fraction of students scored above 80?

Hint: 80 is two standard deviations above the mean, and the curve is symmetric.

Weights in a neural network are often initialized from N(0, 0.02²). What does the second number control?

Conditional probability — updating when you learn something

Here's where probability stops being about dice and starts being about reasoning. New information changes the odds. The probability that someone is a professional basketball player is tiny. Given that they're 2.05 m tall, it's much larger. Same person, different information.

is that idea, and it is the engine of every diagnosis, every spam filter, and every language model deciding what word comes next.

Think of it like switching to a smaller room:

You're looking for someone in a building of 1000 people, 10 of whom you'd call a match — so 1%. Now you learn they're in the conference room, which holds 20 people, 5 of them matches. You haven't changed anyone; you've shrunk the room you're searching. Your odds jumped from 10/1000 to 5/20 = 25%. Conditioning is exactly this: throw away everyone the new information rules out, then recount within what's left.

Conditioning is recounting inside a smaller bag

Go back to the tickets. Learning that $B$ happened means: throw out every ticket where $B$ is false, and recount the fraction of the remaining tickets where $A$ is true. That's the whole operation — and it's why the formula below is a fraction with $P(B)$ on the bottom. You're renormalizing to the new, smaller bag.

The worked example that changes how you read every medical statistic

A 99% accurate test, and why a positive result still probably means nothing

A disease affects 1 in 100 people. A test catches 99% of people who have it, and correctly clears 95% of people who don't. You test positive. What's the chance you have the disease?

Most people say "about 99%." Watch what happens when we just count people. Imagine 10,000 of them:

Who's actually sick? 1% of 10,000 = 100 sick, and 9,900 healthy.
Of the 100 sick people, the test catches 99% → 99 test positive, 1 is missed.
Of the 9,900 healthy people, the test correctly clears 95% → but that means it wrongly flags 5% → 0.05 × 9,900 = 495 test positive.
So who gets a positive result? 99 + 495 = 594 people in total.
Of those 594 positives, how many are actually sick? Just the 99. So 99 / 594 = 16.7%.

A positive result on a "99% accurate" test means you have about a 1 in 6 chance of being sick — not 99%. Nothing was wrong with the test. The problem is that healthy people vastly outnumber sick people, so even a small false-positive rate on a huge healthy group produces more false alarms than the test produces true catches.

Who actually tests positive, out of 10,000 people— interactive, drag & zoom

Loading chart…

The 594 positive results, split by who they really are. Only 99 of them — the orange bar, 16.7% — are actually sick. The blue bar is 5% of a very large healthy group, and that is what swamps the result. This failure of intuition is called base-rate neglect, and it is the single most consequential mistake people make with probability.

This exact mistake, in ML

Swap "disease" for "fraud", "defect", or "the rare class" and you have the everyday reality of imbalanced classification. A model with 99% accuracy on a problem where 1% of cases are positive might be achieving that by predicting "negative" every single time. It is why precision — of everything I flagged, what fraction was real? — is reported alongside recall, and why accuracy alone is close to meaningless on rare-event problems. See Evaluation Metrics.

Try to recall

In the example above, which number would you change to make a positive test much more informative — and why?

Hint: Look at which group produced the 495.

Independence — when knowing one thing tells you nothing

Sometimes the new information is useless. Learning the coin came up heads tells you nothing about tomorrow's weather. Two things are when conditioning on one leaves the other's probability unchanged.

Python · runs in your browser

What this does: Reruns the medical-test example as a simulation instead of arithmetic — it generates 200,000 people, gives each a disease status and a test result with the stated error rates, then looks only at those who tested positive and asks what fraction of them are actually sick. The simulated answer lands on the same ~16.7% the counting argument gave, which is a good habit: when a probability claim surprises you, simulate it.

A spam filter finds the word 'invoice'. P(invoice | spam) = 0.4, P(invoice | not spam) = 0.05, and 10% of mail is spam. Is a message containing 'invoice' more likely to be spam or not?

From probability to statistics — what your data can and cannot tell you

Everything so far assumed you knew the distribution. Real life is the other direction: you have 500 measurements and want to know the truth behind them. That flip is statistics, and it comes with one unavoidable fact — your estimate is itself random.

A is what you want to know about; a is what you got to measure. Any number you compute from a sample — a mean, an accuracy, a win rate — is an , and estimators wobble.

Run the experiment twice and you get two different answers

Measure the average height of 30 random people and you get, say, 174.2 cm. Do it again with 30 different people: 176.8 cm. Neither is wrong — the sample mean is itself a random quantity with its own distribution. The entire content of statistics is characterizing that wobble so you can say how much of a measured difference is real and how much is luck.

Think of it like weighing yourself on a cheap bathroom scale:

Step on it five times and you get five slightly different numbers. You don't conclude your weight is changing — you conclude the scale has noise, and you average the readings. Averaging more readings gives a steadier answer. A sample mean is exactly this: a noisy scale pointed at the population, and taking more samples is stepping on it more times.

The most useful fact in all of statistics

Averages of many things are Gaussian and narrow, even when the individual things aren't Gaussian at all. Here is the proof by picture. A single die roll is perfectly flat — all six faces equally likely, nothing bell-shaped about it. But average a handful of rolls:

Averaging flat things makes a bell — the central limit theorem— interactive, drag & zoom

Loading chart…

Exact distributions (no simulation) of the average of 1, 2, and 5 fair dice. One die is perfectly flat. Two dice already form a triangle. Five dice trace a bell that the dashed Gaussian prediction — mean 3.5, standard deviation 0.76 — matches almost exactly. Notice the curve also gets narrower: the spread of an average shrinks as you average more.

Two things happened in that chart, and both matter enormously:

The shape became a bell, even though what we averaged was flat. That's the .
The bell got narrower as we averaged more. The spread of one die is 1.71; of the average of 5, it's 0.76 — smaller by a factor of $\sqrt{5}$ .

That second fact is the practical one: more data means a narrower answer, but only as fast as the square root. To halve your uncertainty you need four times the data. This is why the jump from 100 to 1,000 test examples feels transformative and the jump from 10,000 to 11,000 doesn't.

Try to recall

You measure a model's accuracy on 100 test examples and get 87%. A colleague measures 84% on a different 100 examples. Is the first model better?

Hint: Think about the width of the sampling distribution at n = 100.

Python · runs in your browser

What this does: Watches the standard error shrink as the sample size grows. For each n it draws 4,000 independent samples of size n from a deliberately non-Gaussian distribution (a coin-flip-like Bernoulli), records how much the sample means actually scatter, and compares that to the sigma-over-root-n prediction. The measured and predicted columns track each other, and both fall by half every time n goes up 4x — the square-root law you cannot escape.

Why machine learning is built on all of this

The payoff. Where does a loss function come from? Not from thin air — from probability.

Training a model is picking the distribution that makes your data look unsurprising

Treat the model as proposing a probability for whatever you observed. Good parameters make the data you actually saw look likely; bad parameters make it look like a fluke. So: search for the parameters that assign the highest probability to your training data. That principle is called maximum likelihood, and essentially every loss function you have ever used is it in disguise.

Where this shows up in ML

Every one of these is an idea from above wearing a different hat:

Softmax outputs are a probability distribution — non-negative, summing to 1, exactly the two axioms.
Cross-entropy loss is average negative log-likelihood: the surprise of the true answers.
MSE loss is maximum likelihood under Gaussian noise — the Gaussian formula with the $\exp$ logged away.
Weight initialization draws from $\mathcal{N}(0, \sigma^2)$ with $\sigma$ chosen so signals neither vanish nor explode.
Batch norm and LayerNorm compute a running mean and variance and standardize by them — the z-score formula, applied inside the network.
Dropout is a Bernoulli random variable per unit; the network learns to be right in expectation.
Train/test splits and error bars are sampling theory: your test accuracy is an estimate with a standard error of about $\sqrt{p(1-p)/n}$ .
Diffusion models, VAEs, and every generative model are explicit statements about a probability distribution, trained by maximizing likelihood.

Explain it yourself

Explain to a friend why a 99% accurate test for a rare disease can still leave you probably healthy after a positive result. Use the 10,000-people counting picture, not Bayes' formula. Then explain why measuring accuracy on 100 test examples cannot distinguish an 87% model from an 84% one. If either explanation stalls, that is exactly the section to reread.

Recap — the key ideas

A probability is a number in $[0,1]$ meaning "how often in the long run" — and probabilities over all outcomes must sum to 1.
Small samples swing wildly; the law of large numbers pulls long-run frequencies toward the true probability.
A distribution is the whole table of what could happen and how likely each is; discrete ones have a PMF, continuous ones a PDF where probability is area.
Expectation is the probability-weighted average (the balance point); variance is the average squared distance from it, and its square root, the standard deviation, is the spread in original units.
The Gaussian is fixed by just a mean and a standard deviation, follows 68-95-99.7, and appears whenever many small independent effects add up. Its negative log is a squared error — which is why MSE is a Gaussian assumption.
Conditional probability is recounting inside a smaller bag; Bayes' rule flips a conditional by combining a prior with a likelihood. Ignoring the prior (base-rate neglect) is why a 99%-accurate test for a 1-in-100 disease means only ~17%.
Independence is what licenses multiplying probabilities — and it's usually assumed rather than checked.
Statistics runs backward from data: any sample estimate wobbles, its standard error is $\sigma/\sqrt{n}$ , and the CLT makes averages Gaussian regardless of the original shape. Four times the data for half the error bar.
Every loss is a likelihood. Minimizing average $-\log p_\theta(x)$ is cross-entropy, and with a Gaussian it is MSE.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: attempt the problems below before rereading — pulling an answer from memory beats recognizing it on the page.
• Spaced repetition: mark this topic complete and it joins your Review queue, resurfacing right before you'd forget it.
• Interleaving: mix these with problems from Calculus and Linear Algebra rather than grinding one type — messier practice, sturdier memory.

By hand: a bag holds 2 red and 3 blue marbles. You draw two without replacement. What's P(both red)? Then compute P(second is red given the first was red) and check that multiplying the two conditionals gives your first answer.
By hand: redo the medical-test count with a disease affecting 1 in 10 people instead of 1 in 100. How does P(sick | positive) change, and which single number in the count did the work?
By hand: a die is rolled twice. Compute $\mathbb{E}[\text{sum}]$ and $\mathrm{Var}(\text{sum})$ , then verify against the "variances add for independent variables" rule.
From scratch: write a function that estimates $\pi$ by throwing random points into a square and counting how many land in the inscribed circle. Plot your estimate against the number of points and watch the law of large numbers converge — slowly.
Prove it numerically: show that np.var(x) and np.var(x, ddof=1) differ, and that only the second one averages out to the true variance across many small samples.
Read like a scientist: find a recent ML paper reporting a 1-point improvement on a benchmark. Look for the test-set size and whether an error bar or seed count is given. Compute the standard error yourself and decide whether the improvement is distinguishable from noise.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Find out how much test data you actually need. The starter compares two models by testing each on n examples and asking how often the worse model looks better by chance. Run it as-is, then do the TODOs: (1) change n to 1000 and rerun — how often does the comparison flip now? (2) shrink the true gap from 0.03 to 0.01 and see how much data it takes to tell them apart reliably.

editor

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Next: turn $-\log p$ into a subject of its own in Information Theory, or go see these distributions get fitted in Optimization.