Next-Token Prediction

The one deceptively simple objective — predict the next token — that teaches a model grammar, facts, and reasoning. Built from zero: probabilities over a vocabulary, softmax, cross-entropy, perplexity, and how a predictor becomes a writer.

beginner#language-modeling#next-token#objective

Start here — the only job an LLM is ever given

Everything a large language model can do — answer questions, write code, translate, summarize, argue — comes out of training it to do one thing, over and over, a few trillion times:

Given the text so far, guess what comes next.

That's the whole training objective. Not "understand language." Not "be helpful." Just guess the next piece of text. This page is about why that one boring instruction is enough, and exactly how it turns into numbers a computer can improve.

The one-sentence version

A reads the text you have so far and produces a probability for every possible next token. Training means: show it real text, look at what actually came next, and nudge the model to have given that token a higher probability. Repeat until the guesses get good.

Remember from Tokenization that models don't read words — they read , chunks of text like " Paris" or "ing". The complete list of tokens a model knows is its . So the precise job is: given the tokens so far, put a probability on each token in the vocabulary.

Think of it like a weather forecaster, not a fortune teller:

A forecaster never says "it will rain tomorrow." They say "70% rain, 25% cloud, 5% sun" — a whole spread of possibilities with confidence attached. A language model works exactly the same way. It never picks a single next word. It hands you a forecast over every token it knows, and only later does something else decide which one to actually use.

How to read this page

It starts from zero and adds one idea at a time: what the model outputs → how raw scores become probabilities → how we measure being wrong → the number researchers actually report → how one sentence becomes hundreds of training examples → how a predictor becomes a writer. Flip the Depth switch at the top for the formal notation and the derivations; nothing is hidden for good, and the deeper panels open automatically once you have the prerequisites.

What the model actually outputs — a forecast, not a word

Say the text so far is The capital of France is. A trained model does not output " Paris". It outputs a number for every single token in its vocabulary — tens of thousands of numbers — and those numbers add up to 1.

is the name for that spread. Here's what the top of it looks like:

The model's forecast for the next token— interactive, drag & zoom

Loading chart…

An illustrative next-token distribution after the text The capital of France is. The model spreads its confidence over the whole vocabulary — most of it on one token here, with the rest scattered thinly across tens of thousands of others. Hover a bar to read the exact probability.

Two things to notice, because both matter later:

The model is confident but not certain. It gives " Paris" most of its belief, but keeps a little aside for everything else. A model that was 100% certain of everything would be a model that could never be surprised — and never learn.
Every bar is positive and they sum to exactly 1. That's not a coincidence; it's forced by the softmax you'll meet in a moment.

Try to recall

A language model is asked to continue the text The capital of France is. What does it produce?

Hint: Think about the weather forecaster.

Logits → probabilities: the softmax

Inside, the network doesn't naturally produce probabilities. Its last layer spits out one raw score per vocabulary token — any number at all, positive or negative, and they don't add up to anything in particular. Those raw scores are called .

Turning scores into shares

You have raw scores like 2.0, 1.0, and 0.1. You need positive numbers that add to 1. Two moves do it: first make every score positive by exponentiating it (raising $e$ to that power), then divide each one by the total so the shares add to 1. That two-step recipe is the .

Think of it like turning votes into percentages:

An election gives you raw vote counts — 74 for one candidate, 27 for another. Nobody quotes raw counts; they quote 66% and 24%. Softmax is that same "counts into shares" conversion, with one extra step first: exponentiating turns the model's scores (which can be negative) into something that behaves like a vote count, and it does so aggressively — a score that's 1 point higher gets about 2.7 times the share.

Softmax by hand, on three tokens

Suppose a tiny model with a three-token vocabulary produces logits [2.0, 1.0, 0.1].

Exponentiate each one. $e^{2.0} = 7.389$ , $e^{1.0} = 2.718$ , $e^{0.1} = 1.105$ . All positive now.
Add them up. $7.389 + 2.718 + 1.105 = 11.213$ . This total is the "electorate."
Divide each by the total. $7.389 / 11.213 = 0.659$ , $\;2.718 / 11.213 = 0.242$ , $\;1.105 / 11.213 = 0.099$ .

Result: [0.659, 0.242, 0.099]. They're all positive, and they sum to $1.000$ . The token with the highest logit got the biggest share — and notice how a gap of just 1.0 in the logits became a roughly 2.7× gap in probability.

Python · runs in your browser

What this does: Turns three raw scores (logits) into probabilities with softmax, exactly reproducing the hand calculation above, then proves the two properties that matter: every number is positive and they sum to 1. Change the logits and rerun to feel how a small change in a score moves the probabilities a lot.

What temperature does to the same three logits— interactive, drag & zoom

Loading chart…

The identical logits [2.0, 1.0, 0.1] pushed through softmax at three temperatures. Cooling to T = 0.5 concentrates belief on the leader; heating to T = 2 spreads it out. Nothing about the model changed — only how sharply its scores are read.

You add 100 to every single logit before applying softmax. What happens to the resulting probabilities?

Measuring wrongness — the cross-entropy loss

Now we have a forecast. Real text tells us what actually came next. To train, we need one number saying how bad was that forecast — the — because gradient descent can only shrink a number.

Loss is surprise

The right measure is how surprised the model was by the truth. If it gave the actual next token a probability of 0.9, it basically saw it coming — barely surprised, tiny loss. If it gave the true token 0.01, it was blindsided — huge surprise, huge loss. So: take the probability the model assigned to the token that really came next, and turn "high probability" into "low penalty."

The function that does that flip is the negative logarithm, $-\log p$ . And that's the entire loss: .

Think of it like a bet you have to pay off:

Imagine you must bet on the next token, spreading 100 chips across the candidates, and you're charged based on how few chips you put on the winner. Put 90 chips on the right token and you pay almost nothing. Put 1 chip on it and you pay dearly. Put zero chips on it and the penalty is infinite — which is precisely why softmax never lets any probability reach exactly zero. Training is you, over trillions of rounds, learning to bet better.

The loss for four different forecasts

Each row is one position in real text. We only ever look at the probability the model gave the token that actually appeared — the other 50000 probabilities are irrelevant to the penalty except through the fact that they stole from it.

Probability on the true token	Loss $= -\log p$	Reading
0.90	0.105	Saw it coming. Almost no penalty.
0.50	0.693	A coin flip's worth of doubt.
0.10	2.303	Genuinely surprised.
0.01	4.605	Blindsided. Big correction incoming.

Work one out by hand: $p = 0.10$ , so the loss is $-\log(0.10) = -(-2.303) = 2.303$ . The logarithm of a number below 1 is negative, and the leading minus sign flips it to a positive penalty. That sign flip is the whole reason the minus is there.

The surprise curve — loss against the probability given to the truth— interactive, drag & zoom

Loading chart…

Loss equals minus the logarithm of the probability the model assigned to the token that actually came next. Confident and right costs nearly nothing on the far right; confident and wrong shoots up toward infinity on the far left. The steepness on the left is what makes the model learn fast from its worst mistakes.

Python · runs in your browser

What this does: Takes a model's forecast over a five-token vocabulary, says which token actually came next, and computes the cross-entropy loss for it — the exact number training tries to shrink. Change the true token to a low-probability one and watch the loss jump.

Try to recall

The model gives the correct next token a probability of 0.99, but it also gave a wrong token 0.005. Does that wrong token contribute to the loss?

Hint: Look at which single number the loss formula reads.

Perplexity — the number papers actually report

Cross-entropy in nats is hard to feel. Is a loss of 3.0 good? Researchers therefore quote instead.

How many options is it effectively juggling?

A perplexity of 10 means the model is about as uncertain as someone picking uniformly at random from 10 equally likely options at every step. Perplexity 2 means it's basically down to a coin flip. Perplexity 50000 means it's guessing blindly from the entire vocabulary. Lower is better, and it's on a scale you can picture.

From loss to perplexity, and back

Perplexity is just $e$ raised to the loss.

A model reaches an average loss of $2.303$ nats. Perplexity $= e^{2.303} = 10.0$ — as uncertain as choosing among 10 options.
A weaker model sits at a loss of $3.0$ . Perplexity $= e^{3.0} = 20.1$ — twice as many effective options, so twice as lost. Loss and perplexity always move together; perplexity just puts the number on a scale you can picture.
An untrained model with a 50257-token vocabulary spreads its guess uniformly, so it gives the true token $1/50257$ and its loss is $\log 50257 = 10.82$ . Perplexity $= e^{10.82} \approx 50257$ — exactly the vocabulary size, which is the sanity check every practitioner runs on step 0 of training.

That last point is the most useful thing here: if your loss at initialization isn't close to $\log V$ , something is wrong before you've trained at all.

Perplexity is the exponential of the loss— interactive, drag & zoom

Loading chart…

The exact relationship between cross-entropy loss and perplexity, on a logarithmic vertical axis. Note the marked point at loss 10.82 — that is log of 50257, the loss of an untrained model guessing uniformly over a GPT-2-sized vocabulary, giving a perplexity equal to the vocabulary size itself.

Python · runs in your browser

What this does: Computes the cross-entropy loss and perplexity for a short sequence, one position at a time, so you can see how each token's surprise contributes to the total — and confirms the perplexity of an untrained model equals its vocabulary size.

Two language models are evaluated on the same test set. Model A reports a perplexity of 18; Model B reports 42. What can you conclude?

One sentence, hundreds of lessons — teacher forcing

Here's the trick that makes pretraining so absurdly data-efficient: a single sentence isn't one training example. It's one training example per token.

Every prefix is a question, every next token is its answer

Take the sentence the cat sat on the mat. Chop it after the first token: context the, answer cat. Chop after the second: context the cat, answer sat. And so on. A 6-token sentence yields 5 question-answer pairs — and crucially, the answers were already there. Nobody labels anything. The text labels itself. That's why this is called .

Think of it like a deck of flashcards you get for free:

Hand someone a novel and ask them to make flashcards from it, and you'd get maybe a few hundred. Feed the same novel to a language model and every single token position becomes a card: "here's everything up to here — what's next?" A 100000-token book is 100000 flashcards. Multiply by the internet and you see where trillions of training signals come from.

Turning one sentence into five training examples

The sentence the cat sat on the mat, with each position's context and target:

Position	Context the model sees	Target it must predict
1	`the`	`cat`
2	`the cat`	`sat`
3	`the cat sat`	`on`
4	`the cat sat on`	`the`
5	`the cat sat on the`	`mat`

Now the important part: the model does not process these five rows one at a time. It processes the whole sentence in a single forward pass and produces a prediction at every position simultaneously — five distributions, five losses, one average. That parallelism is exactly what the Transformer buys you, and it's why training is feasible at all.

Note also row 4: the target is the, a token the model has already seen at position 1. The correct prediction genuinely depends on context, not on the token alone.

But there's a catch. If the model sees the whole sentence at once, what stops it from peeking at position 3 while predicting position 3? Nothing — unless we forbid it.

The fix is a : before the model combines information across positions, we blank out every connection that points backward in time. Position 3 may attend to positions 1, 2, and 3; it may not attend to 4, 5, or 6.

Training this way — feeding the true prefix at every position rather than the model's own earlier guesses — is called . The name is literal: at every step the teacher hands the model the correct history, so one bad prediction can't poison the rest of the sentence.

The heatmap on the right is that mask. Each row is a position doing the predicting; each column is a position it might look at. Lit squares are allowed; dark squares are forbidden. Everything above the diagonal — the future — is dark.

Watch out

The mask is also where the classic off-by-one bug lives. Inputs and labels are the same sequence shifted by one: the label for position $i$ is the input at position $i+1$ . Shift the wrong way and the model trivially learns to copy its own input, the loss crashes toward zero, and generation produces gibberish.

A loss that drops implausibly fast is nearly always a leak.

The causal mask — who is allowed to see whom— interactive, drag & zoom

Loading chart…

Rows are the position making a prediction; columns are the positions it may read. The lit lower triangle is the past and present, which is allowed; the dark upper triangle is the future, which is blocked. Position 1 sees only itself, while the final position sees everything before it — this is what makes the model autoregressive.

Python · runs in your browser

What this does: Builds the context-and-target pairs from a sentence exactly as a training batch does, then constructs the causal mask and prints it, so you can see that inputs and labels are just the same sequence shifted by one position.

Try to recall

Why can a language model be trained on raw internet text with no human labelling at all?

Hint: Where does the correct answer for each position come from?

From predictor to writer — autoregressive generation

A model that only ever answers "what's next?" seems limited. It isn't, because you can feed its answer back in.

Predict one token, glue it on, repeat

Ask for the next token. Pick one. Append it to the text. Now ask again — with the new, longer text as context. Loop. Each pass produces exactly one token, but the loop produces essays, proofs, and programs. This feed-the-output-back-in loop is what means.

Think of it like a chess player who only ever thinks one move ahead, but always gets to move again:

Each individual decision is myopic — one token. But because the result of that decision becomes part of the next decision's input, long-range structure emerges anyway. The model writing the tenth paragraph can see the nine it already wrote.

How do you turn the forecast into an actual token? That choice is , and it is not part of training at all — you can change it freely on an already-trained model:

Greedy — always take the highest-probability token. Deterministic, and reliably dull; it also gets stuck repeating itself.
Sampling — draw randomly according to the probabilities. Varied, but occasionally picks something from the long tail that derails the text.
Top-k — keep only the $k$ most likely tokens, renormalize, then sample. Cuts off the tail.
Top-p (nucleus) — keep the smallest set of tokens whose probabilities sum to $p$ (say 0.9), then sample. Adapts: a narrow set when the model is confident, a wide one when it isn't.

Same model, different personality

Temperature and decoding strategy are why the same model can feel rigid in one product and freewheeling in another. Nothing about the weights changed — only how the forecast is read.

Here is the whole idea at microscopic scale. This model has no learned parameters at all — it just counts which token follows which, which is exactly what a does — and then generates by sampling from those counts.

Python · runs in your browser

What this does: Builds a bigram language model by counting which word follows which in a tiny corpus, prints its forecast after the word 'the', then generates a new sentence by sampling one token at a time and feeding each choice back in. This is the full autoregressive loop that an LLM runs — with counting standing in for a billion learned parameters.

During training the model always receives the real previous tokens as context, but during generation it receives its own previously generated tokens. What is this mismatch called, and why does it matter?

Why such a simple objective teaches so much

It's genuinely surprising that "guess the next token" produces something that can translate, summarize, and write code. The reason is that guessing well requires knowing things.

To finish The capital of France is ___ you must have stored a fact.
To finish She picked up the red ___ you need grammar and plausibility.
To finish def factorial(n): if n <= 1: return ___ you need to have internalized how the code works.
To finish Translate to French: The book is on the table. ___ you need translation.
To finish 17 * 24 = ___ you need arithmetic.

None of those abilities were asked for. Each one showed up because it lowered the loss. That's the whole reason the field bet so heavily on scale: the objective never saturates — there is always more structure to learn that would shave a little more surprise off the next token.

Compression is understanding

Predicting text well and compressing text well are the same problem — a good forecast lets you encode the truth in fewer bits, which is precisely what the cross-entropy loss measures. So a model driving its loss down is being squeezed into finding the shortest possible description of how language and the world behave. The regularities it extracts to save bits are what we experience as knowledge.

Four confusions worth clearing up now

The model does not pick a word. It outputs a distribution; a separate decoding step picks. Temperature and top-p live there, not in the model.
Low loss is not the same as being right. The objective rewards matching the training distribution, including its errors and biases. Nothing in cross-entropy knows what is true.
A base model does not answer questions. It continues text. Ask it something and it may reply with more questions — the answering behavior comes later, from supervised finetuning and RLHF.
Perplexity across tokenizers is meaningless. Different vocabularies chop text into different numbers of tokens, so per-token numbers aren't comparable.

Explain it yourself

Explain to a friend who has never studied maths why training a model to guess the next word ends up teaching it facts and grammar. Then explain what the loss number means in plain words, and why a probability of zero on the true token would be infinitely bad. If you stall on the loss, reread the surprise curve section.

Recap — the key ideas

A language model outputs a probability distribution over the entire vocabulary at every position — a forecast, never a single word.
Softmax turns raw scores (logits) into that distribution: exponentiate, then divide by the total. Only the differences between logits matter.
Cross-entropy loss is the negative log of the probability given to the token that actually came next — literally "how surprised was the model." It is exactly maximum likelihood, and its floor is the entropy of language itself.
Perplexity is $e^{\text{loss}}$ : the effective number of options the model is choosing among. An untrained model's perplexity equals its vocabulary size $V$ — the sanity check at step 0.
Every token position is its own training example, labelled by the text itself (self-supervision), with a causal mask stopping the model from seeing the future and teacher forcing supplying the true prefix.
Feeding predictions back in makes the predictor a writer (autoregressive generation); how you pick each token — greedy, sampling, top-k, top-p, temperature — is a decoding choice made after training.

Practice — and how to make it stick

Learn it the way that actually works

• Retrieval practice: before scrolling back, try to say out loud what softmax does, what cross-entropy measures, and what perplexity means. Struggling to recall beats rereading.
• Spaced repetition: mark this topic complete to add it to your Review queue, so it resurfaces right before you would have forgotten it.
• Interleaving: mix these with Optimization and Probability & Statistics rather than grinding one topic — the loss you just met is exactly the number gradient descent minimizes.

Edit and run the lab below. If you get stuck or hit an error, ask Ada on the right — she can see your code and your terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Build the training signal end to end. Run the starter to see the loss and perplexity of a toy forecast. Then do the three TODOs: (1) make the model more confident in the correct token and check the loss falls, (2) compute the perplexity and confirm it equals exp(loss), (3) set the true token to the one the model doubts most and explain in a comment why the loss explodes.

editor

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Then take it further, away from the keyboard and back to it:

By hand: a model gives the true token a probability of $0.25$ . What is the loss, and what perplexity would that imply if every position were like this? (Answer: $-\log 0.25 = 1.386$ , perplexity $4$ — as if choosing among four options.)
Break it on purpose: in the bigram cell above, change rng = np.random.default_rng(0) to a different seed and rerun a few times. Notice how much the generated sentence varies — that variability is sampling, and temperature would widen or narrow it.
Extend the bigram model: make it a trigram model that conditions on the previous two words. Does the generated text get more coherent? What happens to the number of contexts it has never seen — and why does that make smoothing necessary?
Read like a scientist: skim the GPT-2 paper's introduction and find where it argues that a model trained only on next-token prediction picks up tasks it was never taught. Compare that claim to the list in the section above.
Build it for real: work through Karpathy's Let's build GPT video, which starts from exactly the counting bigram model above and grows it into a Transformer.

The one thing this page left as a black box is how the model turns a context into logits — how it decides that after The capital of France is, the token " Paris" deserves a high score. That's the next lesson: Self-Attention from Scratch.