How LLMs Are Trained: The Big Picture

The whole journey behind models like Claude, Gemini, and GPT — from raw internet text to a helpful assistant — explained from zero, then the three training stages that make it happen.

beginner#llm#overview#pretraining#rlhf

Start here — what "training a language model" even means

You've talked to Claude, Gemini, or ChatGPT. Under the hood, each is a : a giant network of numbers that reads text and writes text. The surprising part is how it got so capable, because at its core it was taught to do just one boringly simple thing — and everything else emerged from that.

The one-sentence version

An LLM is trained to predict the next word (really, the next token) over and over, across a huge slice of the internet. That's it. Predicting the next word well enough — for billions of examples — forces the model to secretly learn grammar, facts, translation, arithmetic, and a shadow of reasoning, because all of those help it guess better.

are the pieces text gets chopped into. So "the model predicts the next token" just means: given the text so far, guess what comes next.

Think of it like autocomplete that read the whole library:

Your phone's autocomplete guesses the next word from the last few. An LLM is that idea taken to an absurd extreme: it read a large chunk of all the books, code, and web pages humanity has written, and got extremely good at "what comes next." Push that far enough and "finish this sentence" turns into "answer this question" and "write this program."

Where we're going

This page is the map. It gives you the whole pipeline in plain language; the rest of this track zooms into each stop — tokenization, embeddings, next-token prediction, self-attention, and on through pretraining and alignment. Flip the Depth switch for the more formal version of anything here.

The three stages, end to end

A model like Claude isn't trained in one shot. It goes through three stages, and each one changes what the model is:

Pretraining → a model that knows a lot but only autocompletes.
Instruction tuning → a model that follows instructions.
Preference tuning (RLHF) → a model that's helpful, honest, and safe.

Let's walk through each.

Stage 1 — Pretraining: learn to predict the next token

The model starts as random numbers that output gibberish. In pretraining, we show it trillions of tokens of text and, at every position, ask: what's the next token? When it's wrong, we nudge its numbers a tiny bit to be less wrong — the exact gradient-descent loop you may already know. Repeat for months across thousands of GPUs.

is what comes out. It's brilliant at completing text and has absorbed huge amounts of knowledge — but ask it a question and it might just continue with more questions, because all it learned to do is continue text.

Try to recall

Why does 'just predict the next token' end up teaching a model facts and grammar?

Hint: Think about what information helps you guess the next word correctly.

The architecture doing the predicting is the Transformer, whose key trick — self-attention — lets every token look at every other token to decide what's relevant.

Feel it yourself — a tiny next-token model

Real LLMs have billions of parameters; this toy has none — it just counts which character follows which, then predicts. Same idea, microscopic scale:

Python · runs in your browser

What this does: A tiny 'language model' that learns which character tends to follow which by counting, then reports the most likely next character. Real LLMs do exactly this — predict the next token — just with billions of learned parameters instead of a count table.

Stage 2 — Instruction tuning: turn the autocompleter into an assistant

The base model can continue text but won't reliably answer. So we fine-tune it on a smaller, curated set of instruction → response examples written by humans: "Summarize this: … → …", "Write a poem about … → …". This stage is called (supervised fine-tuning).

Think of it like a brilliant new hire on day one:

The base model is a genius who has read everything but has never had a job. SFT is the first week of onboarding: showing it, by example, "when someone asks you something, here's the shape of a good reply." It doesn't add much new knowledge — it teaches behavior.

After SFT you have a model that follows instructions. But it still has no sense of which of two decent answers is better, safer, or more honest. That's stage three.

Stage 3 — Preference tuning (RLHF): make it helpful, honest, harmless

Here we teach the model human preferences. People (and increasingly, other AIs) compare pairs of the model's answers — "which response is better?" Those comparisons train a , and then the assistant is optimized to produce answers the reward model rates highly. This whole loop is .

Why this stage matters so much

SFT teaches "answer the question." RLHF teaches "answer it the way people actually want — helpful, truthful, and not harmful." It's the difference between a model that can respond and one you'd actually trust as an assistant. It's also where a lot of a model's "personality" and safety behavior comes from.

A base model straight out of pretraining is asked 'What is 2+2?' and replies 'What is 3+3? What is 4+4?'. Why?

Putting the whole pipeline together

Recap — the key ideas

An LLM is a giant network trained to predict the next token — that one objective forces it to learn language, facts, and reasoning.
Stage 1 — Pretraining: predict the next token over trillions of tokens → a knowledgeable base model that only autocompletes.
Stage 2 — Instruction tuning (SFT): fine-tune on human instruction→response examples → a model that follows instructions.
Stage 3 — Preference tuning (RLHF/DPO): optimize toward human preferences via a reward model → a helpful, honest, harmless assistant.
The engine under all of it is the Transformer and its self-attention.

Explain it yourself

In your own words, explain to a friend why a model trained only to 'predict the next word' can end up answering questions — and what the three training stages each add. If you can't name what SFT and RLHF each change, reread those sections.

Practice — build the foundation

Learn it the way that sticks

• Retrieval: try to name the three stages and what each one changes before scrolling back up.
• Spaced repetition: mark this complete to add it to your Review queue.
• Interleaving: revisit this alongside Optimization — pretraining is gradient descent at scale.

By hand: take the sentence "Large language models predict the next token." and split it into what you think the tokens are. Then check your intuition against a real tokenizer at tiktokenizer.vercel.app — you'll be surprised where the splits fall. (Next lesson: Tokenization.)
Extend the code above: make it actually generate text — start from a character and repeatedly sample the next one from the probabilities. Watch a "language model" babble.
Read like a scientist: skim the abstract of the InstructGPT paper and find the sentence that describes stages 2 and 3. Notice the paper's own three-step figure — it's this page in academic form.
Watch and build: work through the first two videos of Karpathy's Zero to Hero to build a tiny model with your own hands.

Ready to go deeper? Start the descent into the machine with Tokenization — Text into Tokens.