Tokenization — Text into Tokens

How words become the integer tokens a model actually reads — characters, words, and Byte-Pair Encoding — built from zero, one merge at a time.

beginner#tokenization#bpe#vocabulary

Start here — a model cannot read letters

A neural network is a machine that multiplies numbers. That's genuinely all it does. It has no notion of the letter k, no notion of a space, no notion of the word "cat."

So before a language model can read the sentence you typed, something has to convert that sentence into a list of numbers. That converter is the tokenizer, and this page is about how it works.

The one-sentence version

A tokenizer chops text into pieces called tokens, and every distinct piece is assigned a number. The model only ever sees the numbers. Everything you type is turned into a list of integers on the way in, and a list of integers is turned back into text on the way out.

Three words to pin down right away, because everything below uses them:

A is one chunk of text.
The is the full list of chunks the tokenizer knows about.
A is that token's position in the list — the integer handed to the model.

Think of it like a numbered menu at a restaurant:

You don't hand the kitchen a paragraph describing your meal. You say "number 14." The menu is the vocabulary, "number 14" is the token ID, and the dish is the token. The kitchen works entirely in numbers; the menu is the only thing that connects those numbers back to real food.

Here is the whole idea in eight lines of Python. Run it — this is a real (if very crude) tokenizer:

Python · runs in your browser

What this does: Builds the tiniest possible tokenizer — one token per character — by listing every distinct character as a vocabulary and using each character's position in that list as its ID. It then encodes a sentence into integers and decodes those integers back, showing that the round trip returns the original text exactly.

Where this sits in the pipeline

Tokenization is step one of the whole LLM pipeline. The tokens it produces are turned into vectors next (Embeddings), and predicting the next token is the entire training objective (Language modeling). Get tokenization wrong and everything downstream inherits the damage. Flip the Depth switch at the top for the formal version of anything here.

The real question — how big should a chunk be?

Encoding text as numbers is easy. The hard part — the only hard part — is deciding where to cut. Should a chunk be one letter? One word? Something in between?

That single decision controls two things that pull in opposite directions, and understanding that tug-of-war is understanding tokenization. Let's take the two obvious answers in turn and watch each one fail.

Attempt 1 — one token per character

The simplest possible rule: every character is its own token. That's exactly what the code above did.

Why this is tempting

The vocabulary is tiny. English needs maybe 100 entries — 26 lowercase letters, 26 uppercase, 10 digits, some punctuation, a space. And it can spell anything: no word is ever unknown, because every word is just a sequence of letters you already have.

Character tokenization, counted out

Take the sentence Tokenization turns text into numbers. (37 characters including spaces and the full stop).

Vocabulary size: count the distinct characters. There are 17 of them.
Sequence length: count all the characters. That's 37 tokens for one short sentence.
So: a 17-entry vocabulary buys us a 37-token sequence.

Now scale that up. A 1,000-word essay is roughly 5,000 characters — so the model must process a 5,000-token sequence to read one essay.

And that's the fatal problem. Two costs blow up as sequences get longer:

Compute. The attention mechanism inside a Transformer compares every token to every other token, so its cost grows with the square of the sequence length. Double the tokens, quadruple the work.
Reach. A model has a fixed budget of tokens it can look at (its context window). Spending 5 tokens on the word "hello" means fewer real ideas fit inside that budget.

There's a subtler cost too: the model has to learn that c, a, t in that order means a small furry animal — it must rediscover spelling before it can even start on meaning.

Try to recall

Character tokenization gives you a tiny vocabulary and can spell any word. So what is wrong with it?

Hint: Think about how long the resulting list of numbers is.

Attempt 2 — one token per word

The opposite extreme: cut on spaces, and let every word be a token. Tokenization turns text into numbers. becomes just 5 tokens. Wonderful — sequences are seven times shorter.

Why this also fails

Now the vocabulary has to contain every word you will ever see. English alone has hundreds of thousands, plus names, plus typos, plus code, plus every other language. And no matter how big you make the list, someone will type a word that isn't on it — and the model has no way to represent it at all.

A word the tokenizer has never seen is , and the usual fix is to replace it with a single placeholder token, .

Think of it like a phrasebook with no alphabet:

A phrasebook lists whole phrases. It's fast when your sentence is in the book — and useless the moment it isn't, because you have no way to build a new phrase from parts. Every missing word comes out as the same shrug. That's UNK: antidisestablishmentarianism, Kalamazoo, and teh all become the identical token, and the model loses the information that they were ever different.

Worse, word-level tokenization throws away real structure. walk, walked, walking, and walker share an obvious stem, but as four unrelated vocabulary entries the model has to learn each one from scratch, separately.

Approach	Vocabulary size	Sequence length	Unknown words
Character	Tiny (~100)	Very long	Impossible — can spell anything
Word	Huge (100k+) and still incomplete	Short	Common — collapse to `UNK`
Subword	Moderate (~30k–200k)	Moderate	Impossible — falls back to pieces

That last row is the answer, and the rest of this page is about how to build it.

The trade-off, seen as a curve

Here's the tug-of-war made concrete. I ran a small tokenizer over a 37-word paragraph, starting from pure characters and letting it glue frequent pairs together one step at a time. The x-axis is how many glued-together chunks we've added to the vocabulary; the y-axis is how many tokens the paragraph then takes.

Bigger vocabulary buys shorter sequences— interactive, drag & zoom

Loading chart…

Real numbers from the algorithm you will build below, run on a 37-word paragraph. With no learned chunks the paragraph is 211 character-tokens; after 30 learned chunks it is 64 — approaching the 37-token word-level floor, but without ever needing an unknown-word placeholder. The curve is steep at first (a few chunks buy a lot) and then flattens (diminishing returns), which is exactly why real tokenizers stop somewhere in the tens of thousands.

Read the shape, not the numbers

Every learned chunk you add makes the vocabulary one entry bigger and the sequence a bit shorter. The first few chunks are bargains — they cover the most common patterns and cut a lot. Later ones cover rarer patterns and cut less. Somewhere on that flattening curve is the sweet spot, and picking it is the tokenizer designer's entire job.

You switch a model from a 30,000-token vocabulary to a 200,000-token vocabulary. What is the most likely effect?

Byte-Pair Encoding — the algorithm that found the middle

is how essentially every modern LLM solves this. It was borrowed from a 1990s data-compression trick and applied to translation vocabularies by Sennrich and colleagues in 2016.

The whole algorithm, in one sentence

Start with every character as its own token. Then find the pair of neighbouring tokens that occurs most often in your text, glue those two into one new token, and add it to the vocabulary. Repeat a few thousand times. Stop when the vocabulary is as big as you wanted.

That's it. There is no cleverness beyond "glue the most common pair, over and over." The remarkable part is what falls out of it: nobody tells BPE about words, prefixes, or suffixes, yet common words end up as single tokens, and rare words end up split into recognisable pieces like un + predict + able.

Think of it like inventing shorthand as you take notes:

You start writing everything out letter by letter. You notice you keep writing "th," so you invent a squiggle for it. Then you notice "the" appears constantly, so the squiggle grows to cover the whole word. Over a semester you build up a personal shorthand — and crucially, its symbols are the things you actually write often, not the things a dictionary says are important. If a word you've never abbreviated shows up, you can still write it out longhand. BPE is that process, automated.

Step 1 — count every adjacent pair

Let's do it by hand on a corpus small enough to see completely. Four words, with how many times each appears:

low ×5, lower ×2, newest ×6, widest ×3.

Split every word into characters and stick a marker _ on the end to record where words finish (so the tokenizer can tell low at the end of a word from low inside slower). Now count every adjacent pair, weighted by how often its word appears:

Every adjacent pair, counted— interactive, drag & zoom

Loading chart…

Rows are the first character of the pair, columns the second, and each cell is how many times that pair appears across the whole corpus. Bright cells are frequent pairs. The bottom row of the grid — the marker _ — is empty because nothing ever follows the end of a word. Three cells tie at 9, and the winner of that tie becomes the first merge.

The same counts, ranked:

The eight most frequent pairs— interactive, drag & zoom

Loading chart…

Exact counts from the four-word corpus. The pair es appears 9 times: 6 from newest and 3 from widest. Three pairs tie for first place, so the tokenizer needs a tie-break rule — here, whichever it happened to see first.

Step 2 — glue the winner, and repeat

Ten merges of BPE, worked all the way through

Starting point: every word is a list of characters plus the end marker. The corpus is 95 symbol-slots in total (counting each word as many times as it appears).

Each step below finds the most frequent pair and glues it. Watch the total shrink:

Merge e+s (seen 9×) → 86 slots. The token es now exists.
Merge es+t (9×) → 77. Note it merged a previously merged chunk — that's how chunks grow.
Merge est+_ (9×) → 68. The suffix "est at the end of a word" is now a single token. Nobody told it that -est is a superlative ending; it just is frequent.
Merge l+o (7×) → 61.
Merge lo+w (7×) → 54. The token low exists.
Merge n+e (6×) → 48.
Merge ne+w (6×) → 42. The token new exists.
Merge new+est_ (6×) → 36. The whole word newest_ is now one token.
Merge low+_ (5×) → 31. The standalone word low_ is one token.
Merge w+i (3×) → 28.

After ten merges the corpus went from 95 slots to 28 — and here is how each word is now segmented:

low → low_ — one token.
lower → low e r _ — four tokens (the stem survived as a unit).
newest → newest_ — one token.
widest → wi d est_ — three tokens (rare stem in pieces, common suffix as a unit).

Look at what emerged without any linguistic knowledge: the frequent words became single tokens, and the rare one got split into a stem plus a reusable suffix. That's the entire magic of BPE.

Try to recall

After merging e+s into es, BPE's very next merge was es+t. Why can a merge use a chunk that was itself created by an earlier merge?

Hint: What exactly is BPE counting at each step?

Step 3 — build it yourself

Here is the algorithm in full. It is about twenty lines, and it reproduces every number in the worked example above exactly:

Python · runs in your browser

What this does: A complete BPE trainer. It splits four words into characters, counts every adjacent pair weighted by word frequency, merges the most frequent pair, and repeats ten times — printing the shrinking corpus size at each step and the final segmentation of each word. This is the real algorithm, just on a corpus small enough to read.

from collections import Counter

corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3}   # word -> how often it appears
words = {tuple(list(w) + ["_"]): c for w, c in corpus.items()}   # split into chars + end marker

def count_pairs(words):
    """How often does each adjacent pair occur, weighted by word frequency?"""
    p = Counter()
    for symbols, freq in words.items():
        for a, b in zip(symbols, symbols[1:]):
            p[(a, b)] += freq
    return p

def apply_merge(words, pair):
    """Glue every occurrence of `pair` into a single token."""
    a, b = pair
    out = {}
    for symbols, freq in words.items():
        new, i = [], 0
        while i < len(symbols):
            if i < len(symbols) - 1 and symbols[i] == a and symbols[i + 1] == b:
                new.append(a + b)   # the two become one
                i += 2
            else:
                new.append(symbols[i])
                i += 1
        out[tuple(new)] = freq
    return out

size = lambda w: sum(len(s) * f for s, f in w.items())
print("starting size:", size(words), "symbol slots")

merges = []
for step in range(1, 11):
    pair, count = count_pairs(words).most_common(1)[0]   # the most frequent pair
    words = apply_merge(words, pair)
    merges.append(pair)
    print(f"{step:2d}. merge {pair[0]!r}+{pair[1]!r} (seen {count}x) -> {size(words)} slots")

print("\nfinal segmentation:")
for symbols, freq in words.items():
    print("  ", " ".join(symbols), f"(x{freq})")

Encoding new text with a learned tokenizer

Training produced an ordered list of merges. Encoding a new word means replaying that list, in order, on the word's characters. Order matters enormously: es has to exist before est can be formed.

Encoding words the tokenizer has never seen

Using the ten merges learned above, encode some new words:

lowest → apply e+s, then es+t, then est+_, then l+o, then lo+w → low est_ — two tokens, both learned from other words. The tokenizer has never seen "lowest" and handles it perfectly.
newer → new e r _ — the stem is one token, the ending spelled out.
slower → s low e r _ — the unfamiliar leading s survives as a lone character.
glow → g low_ — the letter g never appeared in the training corpus at all, yet nothing breaks. It stays a single-character token.

That last case is the whole point. There is no UNK. A word the tokenizer has never encountered simply decomposes into smaller pieces, and in the worst case into individual characters. Rare words cost more tokens; they are never impossible.

Python · runs in your browser

What this does: Takes the ten merge rules learned above and replays them, in order, on words the tokenizer never trained on. Notice that lowest becomes two familiar chunks, and that glow keeps the letter g as its own token rather than failing — subword tokenizers degrade gracefully instead of producing an unknown-word placeholder.

Try to recall

A subword tokenizer meets a word it has never seen. What happens?

Hint: Compare with what a word-level tokenizer would do.

Why must the merges be replayed in the exact order they were learned?

Bytes, not characters — how real tokenizers guarantee coverage

There's one gap left. "Every character is in the vocabulary" is easy to promise for English. But Unicode has around 150,000 characters — every alphabet, every emoji, every mathematical symbol. Putting them all in the base vocabulary is wasteful; leaving them out reintroduces UNK.

The fix used by GPT-2 and essentially every model since is to run BPE not over characters but over .

Why bytes solve it completely

Every piece of text on earth — English, Japanese, emoji, code, a corrupted file — is stored as a sequence of bytes, and there are only 256 possible byte values. Put all 256 in the base vocabulary and you have provably covered every possible input, forever, with a starting alphabet smaller than the English one you'd have used anyway.

Think of it like LEGO bricks versus pre-built models:

Stocking one of every finished LEGO set is impossible and still won't cover what someone wants to build. Stocking every brick covers everything — anything buildable is buildable from bricks. Bytes are the bricks of text: 256 of them, and every document in every language is made of them.

Python · runs in your browser

What this does: Shows that text is really a sequence of bytes, and that non-English characters simply take more of them. Each byte is a number from 0 to 255, so a 256-entry base vocabulary can represent absolutely any text — which is why byte-level BPE never needs an unknown-token placeholder.

Notice what that prints: café is 4 characters but 5 bytes, and the emoji is 1 character but 4 bytes. Non-English text costs more tokens, and that is a direct, measurable consequence of this design — a Japanese sentence can cost several times more tokens than its English translation, meaning more compute, more money, and less of it fits in the context window.

Where GPT-2's vocabulary size comes from

GPT-2's tokenizer has exactly 50,257 entries. That oddly specific number is just three things added up:

256 — one entry per possible byte value, the base alphabet that guarantees coverage.
50,000 — the number of merges the trainers chose to run. This is the dial from the curve earlier.
1 — a single special token marking the end of a document.

$256 + 50{,}000 + 1 = 50{,}257$ . Every LLM vocabulary size decomposes like this: an alphabet, plus merges, plus a handful of special tokens. Newer tokenizers just turn the middle dial up — cl100k_base (GPT-4) runs to roughly 100,000 entries and o200k_base to roughly 200,000, buying shorter sequences at the cost of a bigger embedding table.

Special tokens — the ones no text can produce

Alongside learned chunks, every tokenizer reserves a few hand-made . They carry structure rather than content:

End-of-text — marks a document boundary during pretraining, so the model learns that one document has finished and an unrelated one is starting.
Chat role markers — signal where a system message, a user turn, or an assistant turn begins. This is how a chat model knows who is speaking.
Padding — filler that makes every sequence in a batch the same length, masked out so it never affects the result.

Special tokens are a security boundary

They are given IDs no ordinary text can produce, and tokenizers refuse to emit them from user input by default. If they could be typed, a user could paste the assistant-turn marker into a message and forge a turn — the text equivalent of SQL injection. This is a real class of prompt-injection attack, and it's why the guarantee that user text never encodes to a special-token ID matters.

Why tokenization leaks — the famous model failures

Tokenization is supposed to be invisible plumbing. It isn't. A surprising share of LLM weirdness traces straight back to it, and knowing this makes you dramatically better at diagnosing model behaviour.

Counting letters. Ask a model how many rs are in "strawberry" and it may get it wrong. It never saw the letters — it saw two or three chunks. Asking it to count letters is like asking you to count the brushstrokes in a printed word.

Arithmetic. If 1234 is one token and 5678 is two, the model sees no consistent digit structure to add column by column. GPT-4's three-digit cap exists precisely to reduce this.

Reversing a string. Same cause: reversing characters requires access to characters, which the model does not have directly.

Trailing whitespace. Ending your prompt with a space can genuinely change the output, because the and the are different tokens and a dangling space leaves the model mid-chunk, in a state it rarely saw in training.

Non-English cost. As the byte cell showed, the same meaning in a non-Latin script can cost several times more tokens — a real, unequal tax on compute, price, and context length.

Debugging rule of thumb

When a model fails at something that looks trivially easy — spelling, counting, character manipulation, arithmetic on long numbers — check the tokenization first. Paste the exact input into tiktokenizer.vercel.app and look at the actual chunks. The explanation is often sitting right there.

A model is asked to count the letter 'r' in 'strawberry' and answers incorrectly. What is the most likely underlying cause?

What vocabulary size actually costs the model

Explain it yourself

Explain to a friend why LLMs use chunks of words rather than whole words or single letters, and describe how BPE decides what the chunks should be — without using any formulas. Then explain why a model might miscount the letters in a word. If you stall on any of the three, that is exactly the section to reread.

Recap — the key ideas

Models only process numbers. A tokenizer chops text into tokens, and each token's position in the vocabulary is the integer the model receives.
Character-level tokenization gives a tiny vocabulary but very long sequences; word-level gives short sequences but a huge, always-incomplete vocabulary plus the UNK problem. Neither works.
Subword tokenization is the compromise, and BPE is how it is learned: start from single characters, repeatedly glue the most frequent adjacent pair, stop at the target vocabulary size.
Common words end up as one token, rare words split into reusable pieces — with no unknown token possible, because every character is a fallback.
Real tokenizers merge bytes, not characters: 256 base entries provably cover every language and every emoji. GPT-2's 50,257 is $256 + 50{,}000$ merges $+\,1$ special token.
A regex pre-split stops merges crossing word boundaries; GPT-4's version caps number tokens at three digits to help arithmetic.
Vocabulary size $V$ is a real cost — it scales the embedding table and the output softmax — and tokenization is the hidden cause of letter-counting, arithmetic, whitespace, and non-English-cost failures.

Practice — and how to make it stick

Learn it the way that actually works

• Retrieval practice: before scrolling back, try to state the BPE loop in one sentence and name the three parts of GPT-2's 50,257. Struggling to recall beats rereading.
• Spaced repetition: mark this topic complete to add it to your Review queue, resurfacing right before you would forget.
• Interleaving: mix these with Embeddings and The Big Picture rather than grinding tokenization alone — the token/vector boundary is where all three meet.

By hand: run three merges of BPE on the corpus bat ×4, bats ×3, cat ×5, cats ×2. Which pair wins first, and does a plural-s token emerge?
Check your intuition: paste strawberry, 1234567, hello, and a sentence in a non-Latin script into tiktokenizer.vercel.app. Count the tokens and predict, before looking, where the splits will fall.
Read the source: skim §3.2 of the BPE paper and find the sentence describing the merge loop — it is the algorithm you built above, in four lines of prose.
Go further: work through Karpathy's minbpe to build a byte-level tokenizer that reproduces GPT-4's output exactly.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Train BPE yourself, then measure what it bought you. Run the starter to see 12 merges on a small corpus and how the corpus shrinks. Then do the TODOs: (1) print the merge list and find the step where a whole word became one token; (2) change NUM_MERGES to 3 and then to 25, and compare the final sizes — where do the returns start diminishing? (3) add the word cheapest to the corpus and check whether it reuses the est chunk learned from the other words.

editor

from collections import Counter

corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3, "slowest": 4}
NUM_MERGES = 12

words = {tuple(list(w) + ["_"]): c for w, c in corpus.items()}
size = lambda w: sum(len(s) * f for s, f in w.items())
start = size(words)

def count_pairs(words):
    p = Counter()
    for symbols, freq in words.items():
        for a, b in zip(symbols, symbols[1:]):
            p[(a, b)] += freq
    return p

def apply_merge(words, pair):
    a, b = pair
    out = {}
    for symbols, freq in words.items():
        new, i = [], 0
        while i < len(symbols):
            if i < len(symbols) - 1 and symbols[i] == a and symbols[i + 1] == b:
                new.append(a + b)
                i += 2
            else:
                new.append(symbols[i])
                i += 1
        out[tuple(new)] = freq
    return out

merges = []
for _ in range(NUM_MERGES):
    pairs = count_pairs(words)
    if not pairs:
        break
    pair, _count = pairs.most_common(1)[0]
    words = apply_merge(words, pair)
    merges.append(pair)

print(f"{start} slots -> {size(words)} slots after {len(merges)} merges")
for symbols, freq in words.items():
    print("  ", " ".join(symbols), f"(x{freq})")

# TODO 1: print `merges` and find which step first produced a complete word token.
# TODO 2: try NUM_MERGES = 3, then 25. Plot or compare the final sizes.
# TODO 3: add "cheapest": 3 to `corpus` and see whether it reuses the est chunk.

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Next: now that text is a list of integers, find out how each integer becomes a meaningful vector in Embeddings — Tokens as Vectors.