Information Theory

Entropy, KL divergence, and cross-entropy — the currency behind every learning objective. Taught from zero, starting with what it means to be surprised, and built up to the exact loss function that trains every classifier and every language model.

intermediate#entropy#kl-divergence#cross-entropy

Start here — what this is really about

Information theory has an intimidating name and exactly one idea underneath it:

Learning something is being surprised. The more surprised you are, the more you just learned.

That's it. Everything on this page — entropy, cross-entropy, KL divergence, perplexity — is that sentence turned into arithmetic. And the payoff is enormous, because the number a classifier minimizes, the number a language model reports, and the number that keeps a fine-tuned model from drifting are all the same quantity wearing three different hats.

Think of it like a friend who only texts you news:

If your friend texts "the sun rose this morning," you learned nothing — you already knew. If they text "it snowed in Dubai," you learned a lot. Same number of words, wildly different amounts of information. Information theory's whole contribution is a formula that turns that gut feeling into a number you can compute, add up, and minimize with gradient descent.

How to read this page

It teaches from first principles — no symbols until you already understand the idea they stand for. Flip the Depth switch at the top for the formal definitions, derivations, and edge cases; they open automatically once you've finished the prerequisite (Probability & Statistics). Nothing is hidden for good.

Surprise — how startled should you be?

Start with a single event. Something happened, and you knew in advance how likely it was. How surprising was it?

Two demands pin the answer down, and both are common sense:

Something you were sure would happen (probability 1) should carry zero surprise.
Something unlikely should carry a lot of surprise — and the less likely it was, the more.

So surprise must be a decreasing function of probability that hits zero at probability 1. The function that does this — and does it for one very good reason we'll get to — is the negative logarithm of the probability.

is the atom of this whole subject. Everything else is an average of it.

Surprise is measured in yes-or-no questions

Measuring surprise in bits means: how many yes-or-no questions would I have needed to pin this outcome down? A fair coin landing heads is 1 bit — one question ("was it heads?"). One roll of a fair 8-sided die is 3 bits, because three halvings narrow 8 possibilities to 1. A one-in-a-million event is about 20 bits, because you'd need twenty halvings. The unit is not decoration — it is literally a count of questions.

Surprise grows without limit as an outcome gets rarer— interactive, drag & zoom

Loading chart…

Read it right to left. An outcome you were certain of costs 0 bits of surprise. At probability 0.5 it costs exactly 1 bit — one coin flip of information. At 0.02 it costs 5.6 bits, and the curve keeps climbing toward infinity as the probability approaches zero. That runaway left edge is why a model that confidently predicts the wrong answer is punished so ferociously.

How surprised should you be, in bits?

Compute the surprise of three outcomes. The recipe: flip the probability upside down, then count how many halvings it takes to get there.

A fair coin lands heads. Probability $1/2$ . One halving gets you from 2 possibilities to 1, so the surprise is 1 bit.
A fair 8-sided die shows a 5. Probability $1/8$ . Halve 8 to 4, 4 to 2, 2 to 1 — three halvings, so 3 bits.
A biased coin that comes up heads 90% of the time lands heads. Probability $0.9$ . This is barely news at all, and the arithmetic agrees: it takes only about 0.15 bits. Note it is not zero — you did learn something, just very little.

Now the flip side of case 3: when that same 90/10 coin lands tails (probability $0.1$ ), the surprise is about 3.3 bits — twenty times more than the heads case. Rare outcomes carry the information.

Try to recall

Two events have probabilities 0.5 and 0.25. Which is more surprising, and by how many bits?

Hint: Count halvings.

Python · runs in your browser

What this does: Turns probabilities into surprise, in bits, so you can see the shape of the -log curve as a table instead of a plot. Notice the last two rows: halving the probability always adds exactly 1 bit, and an event you were certain of teaches you nothing at all.

Entropy — the average surprise of a whole distribution

Surprise describes one outcome. But usually you don't care about a single outcome; you care about a source that keeps producing them. How surprising is it on average?

That average is the , and it is the single most important quantity in this lesson.

Entropy is how unpredictable something is

A coin that always lands heads has entropy 0 — you never learn anything, because you already knew. A fair coin has entropy 1 bit — the maximum possible for two outcomes. A fair 8-sided die has entropy 3 bits. Entropy answers: how many yes-or-no questions, on average, does it take to pin down what happened? More unpredictable source, more questions, higher entropy.

Think of it like packing a suitcase:

Entropy is the honest minimum size of the suitcase. A source that is wildly unpredictable (fair dice, random noise) has high entropy and genuinely needs a big suitcase — you cannot compress it. A source that is predictable (English text, a photo of a blue sky) has low entropy, and everything you thought was bulk turns out to be air. Compression algorithms are professional suitcase packers, and entropy is the wall they cannot pack past.

Entropy of three sources, by hand

The recipe: for each outcome, multiply its probability by its surprise, then add up. It's a weighted average where the weights are the probabilities themselves.

1. A fair coin. Two outcomes, each with probability $0.5$ , each carrying 1 bit of surprise.

H = 0.5 \times 1 + 0.5 \times 1 = 1 \text{ bit}

2. A coin that lands heads 90% of the time. Heads carries 0.152 bits, tails carries 3.322 bits, but tails almost never happens:

H = 0.9 \times 0.152 + 0.1 \times 3.322 = 0.137 + 0.332 = 0.469 \text{ bits}

Less than half the fair coin's entropy. The rare outcome is very informative, but it's rare, so it barely contributes to the average.

3. A coin that always lands heads. One outcome with probability 1, carrying 0 bits. $H = 0$ . A source that never surprises you carries no information at all.

The pattern: entropy is highest when you have no idea what's coming, and drops to zero as certainty grows.

Entropy of a biased coin, as the bias changes— interactive, drag & zoom

Loading chart…

The two ends are pinned to zero: a coin that always lands the same way is perfectly predictable and carries no information. The peak sits exactly at P(heads) = 0.5, where you are maximally in the dark. Notice the curve is flat near the top — a 60/40 coin still has 0.971 bits, almost as much as a fair one, so mild bias costs you almost no uncertainty. It is only near the edges that entropy collapses fast.

Try to recall

A weather forecaster in a desert says sunny with probability 0.99 every day. Is their forecast high-entropy or low-entropy, and what does that mean practically?

Hint: Look at the far right of the curve above.

Python · runs in your browser

What this does: Computes the entropy of five different distributions over the same 8 outcomes, from perfectly uniform to nearly certain. Watch the entropy fall from the ceiling of 3 bits toward 0 as probability piles onto one outcome — and notice the clamp on log2, which is how real implementations avoid the infinity at p=0.

A model's training loss stops improving at 1.9 nats. Which explanation is consistent with information theory?

Cross-entropy — paying for the wrong codebook

Here is where information theory stops being a curiosity and becomes the thing you type into your training script.

So far, both the surprise and the averaging came from the same distribution $p$ — the true one. But a model doesn't know $p$ . It has its own guess, call it $q$ . So ask the question that matters:

Reality generates outcomes according to $p$ . My model believes $q$ . How surprised is my model, on average, by reality?

That number is the , and it is the loss function you already use.

Think of it like a phrasebook for the wrong city:

You've packed a phrasebook optimised for Paris — short, handy entries for the phrases Parisians actually use. Then you land in Tokyo. Every sentence you need is buried at the back under some six-page appendix, while the front of the book is full of French you'll never say. You can still communicate, but every single exchange costs you extra flipping. Cross-entropy is the average number of pages you flip: it depends on the book you brought ( $q$ ) and the city you're actually in ( $p$ ). Bring the right book and the cost drops to the city's own entropy — the irreducible minimum. It can never drop below it.

Two distributions, two different jobs

Cross-entropy has a $p$ and a $q$ in it, and they are not interchangeable. $p$ decides which outcomes come up. $q$ decides how much each one costs you. Reality picks the questions; your model pays the price. Getting these two straight is 90% of understanding every formula on the rest of this page.

Three models predicting the same example — the truth is dog— interactive, drag & zoom

Loading chart…

Only the dog bar affects the loss — the other three are ignored entirely for this example. Model A gave it 0.70 and pays 0.51 bits. Model B gave it 0.25 and pays exactly 2 bits. Model C gave it 0.05 and pays 4.32 bits. Model C is not merely twice as bad as B: being confidently wrong is punished far out of proportion, and that steep asymmetry is the whole reason cross-entropy trains classifiers better than squared error does.

Computing a classification loss by hand

A 4-class classifier sees one image. The true label is dog, which as a probability distribution is the one-hot vector $p = [0, 1, 0, 0]$ — 100% dog, 0% everything else.

Model B predicts $q = [0.30, 0.25, 0.25, 0.20]$ . Compute the cross-entropy: for each class, multiply the true probability by the model's surprise, and add.

cat: $0 \times (\text{surprise of } 0.30) = 0$
dog: $1 \times (\text{surprise of } 0.25) = 1 \times 2 = 2$ bits
fox: $0 \times (\dots) = 0$
owl: $0 \times (\dots) = 0$

Total: 2 bits. Notice that three of the four terms vanished. Because the truth is one-hot, every term except the correct class is multiplied by zero.

So for classification with a single correct answer, the entire cross-entropy collapses to one thing:

the surprise the model assigned to the right answer, and nothing else.

That is why CrossEntropyLoss in PyTorch takes an integer label instead of a full distribution — the other entries would all be multiplied by zero anyway. Running the same arithmetic on Model A (0.70 → 0.51 bits) and Model C (0.05 → 4.32 bits) reproduces the three numbers in the chart above.

Try to recall

Your classifier assigns 0.99 probability to the correct class on example 1, and 0.01 to the correct class on example 2. Which example dominates the batch loss?

Hint: Look at the far left of the very first plot on this page.

Python · runs in your browser

What this does: Scores the three models from the chart above against the true label 'dog', in both bits and nats, and confirms that the full four-term cross-entropy sum gives exactly the same answer as the one-term shortcut -log q(true class). That equality is why a classification loss only ever needs the integer label.

Why does PyTorch's CrossEntropyLoss accept an integer class index rather than a full probability distribution over classes?

KL divergence — the extra you pay for being wrong

Cross-entropy mixes together two very different things: how uncertain the world genuinely is (which you can't fix) and how wrong your model is (which you can). Separating them gives the last big idea.

Cross-entropy is the total bill, KL is the overcharge

Even a perfect model has to pay the entropy $H(p)$ — that's the world's own randomness, and no model can beat it. Anything you pay above that is your fault. That excess is the , and it is the honest measure of how far your model is from the truth.

Think of it like a detour on your commute:

The true entropy is the length of the shortest possible route to work — a fact about the map, not about you. Cross-entropy is how far you actually drove. KL divergence is the detour: the extra distance you covered by following a bad set of directions. It's zero only if your directions were the optimal route, and it can never be negative, because you cannot drive less far than the shortest path.

Splitting a loss into floor plus fault

A weather source produces sun, cloud, and rain with true probabilities $p = [0.7,\ 0.2,\ 0.1]$ . Your model predicts $q = [0.5,\ 0.3,\ 0.2]$ .

The world's own uncertainty (entropy). Weight each outcome's own surprise by its probability: $H(p) = 1.157$ bits. No forecaster on earth beats this.
What your model actually pays (cross-entropy). Weight the model's surprise by the true probabilities: $H(p, q) = 1.280$ bits.
The overcharge (KL). Subtract: $1.280 - 1.157 = 0.123$ bits.

So of the 1.280 bits your model is billed per forecast, 1.157 was unavoidable and only 0.123 is your fault. That 0.123 is the only part training can remove — and it is exactly what you would compute directly from the KL formula.

The lesson generalises: cross-entropy = entropy + KL divergence. Since $H(p)$ doesn't depend on your parameters at all, minimising cross-entropy and minimising KL are literally the same optimisation problem.

Python · runs in your browser

What this does: Verifies the central identity of this lesson — cross-entropy equals entropy plus KL divergence — on the weather example, and then shows that swapping the two distributions changes the KL, proving it is not a symmetric distance. Note the tiny epsilon guard: without it, any zero in q would produce inf.

import numpy as np

def entropy(p, base=2):
    p = np.asarray(p, float); nz = p[p > 0]
    return float(-(nz * np.log(nz) / np.log(base)).sum())

def cross_entropy(p, q, base=2, eps=1e-12):
    p, q = np.asarray(p, float), np.clip(np.asarray(q, float), eps, 1)
    return float(-(p * np.log(q) / np.log(base)).sum())

def kl(p, q, base=2, eps=1e-12):
    p, q = np.asarray(p, float), np.clip(np.asarray(q, float), eps, 1)
    nz = p > 0
    return float((p[nz] * np.log(p[nz] / q[nz]) / np.log(base)).sum())

p = [0.7, 0.2, 0.1]      # true weather: sun, cloud, rain
q = [0.5, 0.3, 0.2]      # the model's forecast

print(f"H(p)        = {entropy(p):.4f} bits   <- the world's own randomness, unavoidable")
print(f"KL(p || q)  = {kl(p, q):.4f} bits   <- the model's fault, the only removable part")
print(f"H(p, q)     = {cross_entropy(p, q):.4f} bits   <- the total bill")
print(f"H(p) + KL   = {entropy(p) + kl(p, q):.4f} bits   <- identical, as promised")

print(f"\nKL(p || q) = {kl(p, q):.4f}   but   KL(q || p) = {kl(q, p):.4f}")
print("Not symmetric -> KL is a divergence, not a distance.")
print(f"KL of a perfect model: {kl(p, p):.4f}")

Try to recall

Minimising cross-entropy and minimising KL divergence give identical gradients. Why?

Hint: Write cross-entropy as entropy plus KL and ask which term depends on the model.

A colleague reports a KL divergence of -0.03 between two distributions. What should you conclude?

Perplexity — entropy in the units people actually quote

Open any language-model paper and you'll find perplexity, not entropy. It's the same number wearing friendlier clothes.

Perplexity is the effective number of choices

A model with a perplexity of 20 is, on average, as confused as if it were picking uniformly at random from 20 equally-likely options at every step. Perplexity 2 means it's essentially down to a coin flip. Perplexity 50,000 on a 50,000-token vocabulary means it has learned nothing at all. Lower is better, and the scale is intuitive in a way that "1.9 nats" simply isn't.

From loss to perplexity, and back

Your language model reports a cross-entropy loss of 2 bits per token.

Undo the logarithm: perplexity $= 2^{2} = 4$ .
Read it: the model is as uncertain as someone guessing among 4 equally-likely tokens at every position.

Now suppose you improve the model and the loss drops to 1.5 bits. Perplexity becomes $2^{1.5} \approx 2.83$ . Note what happened: half a bit of loss cut the effective choices from 4 to under 3. Because perplexity is exponential, small improvements in loss look large in perplexity — which is exactly why the metric is popular, and exactly why you should read it carefully.

Framework caution: PyTorch reports loss in nats, so perplexity is math.exp(loss), not 2 ** loss. Mixing the two is one of the most common reproducibility bugs in language-model reporting.

Mutual information — how much does knowing one thing tell you about another?

One last quantity, because it appears everywhere from feature selection to modern self-supervised learning.

Uncertainty before, minus uncertainty after

You're uncertain about tomorrow's weather. Then someone tells you today's barometer reading, and you're less uncertain. The amount your uncertainty dropped is the mutual information between barometer and weather. If the reading told you nothing, the drop is zero — and zero mutual information is exactly what independence means.

Think of it like a hint in a guessing game:

Twenty questions: you need 20 bits to identify the answer. A friend whispers a hint that cuts your remaining questions to 12. That hint carried 8 bits of mutual information with the answer. A useless hint ("it exists") carries 0 bits; a hint that gives it away outright carries all 20.

Where this shows up in ML

Every one of these is something from above wearing a different hat:

CrossEntropyLoss / NLLLoss — cross-entropy against a one-hot label. Reported in nats.
Every language-model training run — cross-entropy against the token that actually came next, averaged over positions; quoted as perplexity in the paper.
Label smoothing — replaces the one-hot target with a slightly softened distribution, so the loss stops pushing the model toward infinite confidence. Read it as: deliberately raise the target's entropy.
Knowledge distillation — trains a small student to match a large teacher's whole output distribution by minimising the KL between them, not just to match the hard label. The teacher's soft probabilities carry more information than a one-hot target does.
VAEs — the loss has a KL term pulling the learned latent distribution toward a standard Gaussian prior, in exactly the units on this page.
RLHF and DPO — the policy is penalised by its KL divergence from the original SFT model, which is what stops it drifting into gibberish while chasing reward.
Decision trees — split on the feature with the highest information gain, which is the mutual information between the feature and the label.
Contrastive self-supervised learning — InfoNCE maximises a lower bound on mutual information between two augmented views.
Compression — a good language model is a good compressor, at $H(p,q)$ bits per token, by the source coding theorem.

Explain it yourself

Explain to a friend, without formulas, the difference between entropy, cross-entropy, and KL divergence — using the commute or phrasebook picture. Then explain why a training loss that stops at 1.9 does not mean the model is bad. If either explanation stalls, that is exactly the section to reread.

Recap — the key ideas

Surprise is $-\log p$ : how startled you should be by an outcome. Measured in bits (base-2, counts yes-or-no questions) or nats (natural log, what every framework uses). It has to be a logarithm, because information from independent events must add while their probabilities multiply.
Entropy $H(p)$ is the average surprise of a distribution — how unpredictable a source is. It is maximised by the uniform distribution ( $\log n$ ), zero for a certain outcome, and it is the hard floor on lossless compression.
Cross-entropy $H(p,q)$ is how surprised your model $q$ is by reality $p$ . With a one-hot label it collapses to $-\log q(\text{true class})$ , which is the negative log-likelihood — so maximum likelihood and cross-entropy minimisation are the same thing.
KL divergence $D_{\mathrm{KL}}(p\|q)$ is the extra surprise caused by your model being wrong. It is never negative, zero only when $p = q$ , and not symmetric — forward KL is mass-covering, reverse KL is mode-seeking.
The identity that ties it together: $H(p,q) = H(p) + D_{\mathrm{KL}}(p\|q)$ . Since $H(p)$ is a constant of the data, minimising cross-entropy is minimising KL — and a loss that plateaus may just be sitting on the data's own entropy.
Perplexity is $\exp$ of the cross-entropy: the effective number of equally-likely choices. Same information as the loss, only comparable across models sharing a tokenizer.
Mutual information is how much your uncertainty about one variable drops once you know another — zero exactly when they're independent.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: attempt the problems below before rereading — pulling an answer from memory beats recognising it on the page.
• Spaced repetition: mark this topic complete and it joins your Review queue, resurfacing right before you'd forget it.
• Interleaving: mix these with problems from Probability & Statistics and Optimization rather than grinding one type — messier practice, sturdier memory.

By hand: compute the entropy of a fair 4-sided die, then of a loaded one with probabilities $[0.7, 0.1, 0.1, 0.1]$ . How many bits of uncertainty did the loading remove, and how does that compare to the 2-bit ceiling?
By hand: a binary classifier predicts 0.6 for the positive class. Compute the loss in bits if the true label is positive, then if it is negative. Which mistake is more expensive, and why does that asymmetry disappear when the prediction is 0.5?
By hand: verify $H(p,q) = H(p) + D_{\mathrm{KL}}(p\|q)$ for $p = [0.5, 0.5]$ and $q = [0.9, 0.1]$ , then swap $p$ and $q$ and confirm the KL changed but the inequality $D_{\mathrm{KL}} \ge 0$ still holds.
From scratch: implement Huffman coding, compress a paragraph of English, and compare the achieved bits-per-character against the entropy you measure from the character frequencies. You should land within 1 bit of the entropy — the source coding theorem's promise, verified by your own code.
Prove it numerically: sample two independent variables and estimate their mutual information; confirm it hovers near zero. Then make one a noisy copy of the other and watch it climb.
Read like a scientist: find a language-model paper reporting perplexity. Check which tokenizer and which dataset it used, and decide whether the number is comparable to another paper's. Most of the time it isn't — and the paper won't say so.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Discover for yourself why cross-entropy punishes confident errors. The starter sweeps the probability a model assigns to the TRUE class and prints the loss. Run it, then do the TODOs: (1) add a column for squared error, (1 - q)**2, and compare how the two losses grow as q approaches 0 — which one still produces a useful gradient for a confidently wrong model? (2) implement label smoothing by replacing the one-hot target with [0.05/3, 0.95, 0.05/3, 0.05/3] and find the q that now minimises the loss. It is no longer 1.0 — explain why in a comment.

editor

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Next: see this loss doing its job at scale in Language Modeling, or go back and strengthen the foundation in Probability & Statistics.