Embeddings — Tokens as Vectors

Turning each token into a learned vector, and why similar meanings end up close together — built from zero, starting with why a token's ID number is useless on its own.

beginner#embeddings#vectors#representation

Start here — the problem nobody warns you about

In the previous lesson your text got chopped into tokens, and each token was handed an ID number — a row number in the tokenizer's vocabulary list. Something like:

"the cat sat"  →  ["the", " cat", " sat"]  →  [464, 3797, 3332]

So now we have numbers. A neural network is a machine that does arithmetic on numbers. Job done?

No. And the reason why is the whole point of this lesson.

A token ID is a name, not a quantity

Token 3797 is " cat" and token 3332 is " sat". Those IDs came from alphabetical-ish bookkeeping, not from meaning. So 3797 + 1 = 3798 is not "one more than a cat" — it's some unrelated token. And 3797 / 2 is meaningless. The ID is a label, like a jersey number or a house address. Doing math on jersey numbers tells you nothing about the players.

Think of it like library shelf numbers:

Every book in a library has a shelf code. Book #4021 sits next to book #4022, but that adjacency is about where they were filed, not about what they say — a cookbook can sit beside a physics text. If you fed shelf numbers into a machine and asked "which two books are most alike?", you'd get nonsense. That's exactly what happens if you feed raw token IDs into a network.

A network needs numbers where arithmetic means something — where "close together" really does mean "similar". Manufacturing those numbers is what an is for.

How to read this page

It teaches from first principles — no linear algebra assumed. Flip the Depth switch at the top for the formal notation and the derivations; they open automatically once you've finished the prerequisites. Nothing is hidden for good.

Attempt #1 — one-hot vectors (and why they fail)

Here's the first idea everyone has, and it's worth walking through because its failure tells you what embeddings must fix.

If the ID number itself is misleading, don't use it as a number at all. Instead give every token its own private slot: a long list of zeros with a single 1 in the token's position. That's a .

One-hot encoding a 5-word vocabulary

Say our entire vocabulary is [cat, dog, king, queen, apple] — 5 words, so every vector has 5 slots:

token	one-hot vector
cat	`[1, 0, 0, 0, 0]`
dog	`[0, 1, 0, 0, 0]`
king	`[0, 0, 1, 0, 0]`
queen	`[0, 0, 0, 1, 0]`
apple	`[0, 0, 0, 0, 1]`

Now no ID is being treated as a quantity — king is not "3 units of something". Each token just owns a slot. Good so far.

But now ask the machine: is king more similar to queen or to apple?

Recall from Linear Algebra that the way to compare two vectors is the dot product — multiply matching slots, add up the results:

king · queen = 0×0 + 0×0 + 1×0 + 0×1 + 0×0 = 0
king · apple = 0×0 + 0×0 + 1×0 + 0×0 + 0×1 = 0

Both zero. Identical. The encoding says king is exactly as related to queen as it is to apple — which is to say, not at all.

That's the fatal flaw. One-hot vectors are maximally ignorant: every token is equidistant from every other token, so the representation contains no information beyond "these are different things." Every scrap of meaning would have to be re-learned from scratch, separately, for every single word.

And there's a second, more mundane problem: real vocabularies have tens of thousands of tokens. A one-hot vector for GPT-2's vocabulary is 50,257 numbers long, of which 50,256 are zero. That is a spectacularly wasteful way to say "token number 3797".

Try to recall

Why does the dot product between any two different one-hot vectors always come out to zero?

Hint: Think about where the 1s are.

Attempt #2 — short lists of learned numbers

So we want the opposite of one-hot: instead of one long, mostly-empty vector where each slot means "is it this exact word", we want a short, dense vector where each slot means something shared across many words.

Describe tokens by traits, not by identity

Don't give each word its own private slot. Give every word a score on the same handful of underlying traits — something like "how royal is it", "how alive is it", "how edible is it", "how formal is it". Now king and queen score similarly on most traits and differ on one, so they land near each other. apple scores completely differently, so it lands far away. Similarity falls out of the numbers for free, without anyone programming it in.

Think of it like a mixing board:

Picture a sound desk with 300 sliders. Every word is one setting of all 300 sliders. Two words that sound alike in meaning have nearly the same slider positions — a few nudged here and there. The word isn't stored as a name any more; it's stored as a configuration. And configurations can be compared, averaged, and subtracted, which names cannot.

That list of trait scores is the token's , and the number of slots in it is the .

A hand-built toy embedding space

Let's build a tiny embedding space by hand, with just four traits, so you can see every number. (Real ones have hundreds or thousands, and nobody hand-picks them — but the shape of the idea is identical.)

The four traits are: person-ness, royalty, maleness, fruit-ness.

token	person	royalty	male	fruit
king	0.9	0.9	0.4	0.0
queen	0.9	0.9	−0.4	0.0
man	0.9	0.1	0.4	0.0
woman	0.9	0.1	−0.4	0.0
apple	0.1	0.0	0.0	0.9

Read a row as a recipe. king is "very much a person, very royal, leaning male, not a fruit." queen is the same recipe with the maleness slider pushed the other way. apple shares almost nothing with any of them.

Now redo the similarity question. Compare king with queen slot by slot: 0.9 vs 0.9, 0.9 vs 0.9, 0.4 vs −0.4, 0.0 vs 0.0. Three of four traits agree exactly. Compare king with apple: nothing lines up.

The representation now carries meaning. Nobody added a rule saying "kings and queens are related" — it simply follows from the numbers.

Notice two things that changed:

Short instead of long. Four numbers instead of five slots here; in practice, ~768 numbers instead of ~50,000. Embeddings are .
Shared instead of private. Every token is scored on the same traits, so tokens can be compared, and what the model learns about one trait helps it with every token that uses it.

Try to recall

One-hot vectors are long and sparse; embeddings are short and dense. Which of those two properties is the one that actually lets the model see that two tokens are related?

Hint: Think about what a 0 tells you versus what a 0.9 tells you.

Seeing the space

Here's a toy embedding space drawn as a picture. Each dot is a token, placed where its vector puts it. Words that mean similar things cluster; unrelated words sit far apart.

A toy embedding space, flattened to 2-D— interactive, drag & zoom

Loading chart…

Four clusters emerge with no one labelling them: royalty, fruit, animals, and motion verbs. Real embedding spaces have hundreds of dimensions, so pictures like this are always a squashed 2-D shadow of the real thing — which is why the axes have no names. Drag and zoom to explore.

Don't over-read the axes

In a plot like this the horizontal and vertical directions mean nothing on their own — they're just whatever two directions the flattening procedure happened to pick. Only relative distances are meaningful. A common beginner mistake is to announce that "the x-axis is clearly formality"; usually it isn't.

The embedding matrix — where the vectors actually live

So each of the ~50,000 tokens gets its own vector of ~768 numbers. Stack all those vectors on top of each other and you get a big table: one row per token, one column per trait. That table is the , and it is the very first layer of a language model.

The first layer of an LLM is a lookup table

People expect the input layer of a neural network to be doing something clever. It isn't. It's a filing cabinet: token 3797 arrives, the model pulls out row 3797, and hands that list of 768 numbers to the rest of the network. That's the entire operation. The cleverness is not in the lookup — it's in the fact that every number in that cabinet was learned during training.

The size of the cabinet is decided by two numbers you pick before training: how many tokens are in the vocabulary, and how many traits each one gets. Multiply them and you have the parameter count of the whole layer.

For GPT-2 small that's 50,257 tokens × 768 dimensions = 38.6 million numbers — roughly 31% of the model's 124 million total parameters, all sitting in a lookup table before any "real" computation happens.

Rough sizes

GPT-2 small: 50,257 × 768
A modern 7B model: ~32,000 × 4,096
Bigger models mostly grow the dimension, not the vocabulary

Let's watch the lookup happen, and confirm the equivalence with our own eyes:

Python · runs in your browser

What this does: Builds a tiny embedding matrix for a 5-token vocabulary, looks up one token's vector by row index, and then proves the row-lookup gives exactly the same answer as multiplying by a one-hot vector — which is why the embedding layer counts as an ordinary linear layer.

Measuring closeness — cosine similarity

We keep saying similar tokens end up "close together". Time to make that precise, because the obvious definition of closeness turns out to be the wrong one.

Direction matters, length doesn't

Two vectors can point the same way but have very different lengths — one is a long arrow, one is a short arrow, both aimed north. In embedding space, the direction is what encodes meaning; the length tends to encode boring things like how often the token appears in training. So we compare angles, not distances. Two vectors pointing the same way are similar, no matter how long they are.

Think of it like a compass reading versus a hike:

Two hikers both set off due north-east — one walks 2 km, the other 20 km. They went the same way; they just went different amounts. If you want to know whether two people had the same idea, ask for their compass bearing, not their step count. Cosine similarity asks for the bearing.

The measure that reads out the angle is , and it's built from the dot product you already know.

Before the formula, get the geometry into your hands. Drag the two vectors below and watch the dot product change — biggest when they align, zero at a right angle, negative when they oppose. That behaviour is the similarity score:

Dot product — drag the two vectors— GeoGebra, drag & exploreOpen on GeoGebra →

Loading interactive visualization…

Drag the tips of vectors u and v and watch the dot product update. Notice three things: it peaks when they point the same way, hits zero when they are perpendicular, and goes negative when they oppose. Also notice it grows when you simply make a vector longer — that length sensitivity is exactly what cosine similarity divides away.

Cosine similarity, by hand

Take two 2-D vectors, a = [3, 4] and b = [4, 3].

Step 1 — dot product (multiply matching slots, add):

a · b = 3×4 + 4×3 = 12 + 12 = 24

Step 2 — lengths (square the entries, add, take the square root):

length of a = √(3² + 4²) = √(9 + 16) = √25 = 5
length of b = √(4² + 3²) = √25 = 5

Step 3 — divide the dot product by both lengths:

cos(a, b) = 24 / (5 × 5) = 24 / 25 = 0.96

0.96 is near the maximum of 1, so these two vectors point in almost the same direction. Now try c = [−4, 3]:

a · c = 3×(−4) + 4×3 = −12 + 12 = 0, so cos(a, c) = 0 — perfectly unrelated, at a right angle.

The division in step 3 is the crucial bit: it cancels out how long the arrows are, leaving only how much they agree in direction.

Now let's compute real similarity scores on the hand-built toy space from earlier:

Python · runs in your browser

What this does: Computes cosine similarity for every pair of tokens in our four-trait toy embedding space and prints the table. Watch the block structure appear: the four people-words score high with each other, apple sits apart from all of them, and king lands closer to queen than to woman.

The same table as a picture — bright means similar, dark means unrelated:

Cosine similarity between toy embeddings— interactive, drag & zoom

Loading chart…

Two bright blocks appear on the diagonal — the four people-words, and the two fruit-words — with a dark region between them. Nobody drew those blocks; they are a direct consequence of the numbers in the embedding table. Note the faint asymmetry inside the people block: king–queen scores 0.82, but king–woman only 0.56.

Two embedding vectors have a cosine similarity of 0. What does that tell you?

Where do the numbers come from?

We hand-built that toy table. Nobody does that in practice — the traits are never named, never designed, and mostly never interpretable. So how are they filled in?

The answer rests on one observation about language, old enough to predate neural networks entirely:

The distributional hypothesis

A word is characterised by the company it keeps. If you never learn a definition of tesgüino but you read "put tesgüino in a glass", "tesgüino makes you drunk", and "we brew tesgüino from corn", you've worked out roughly what it is — because it appeared where words like beer appear. Meaning can be inferred from context alone, and context is something a computer can count.

So the recipe is: give each token a vector, then adjust those vectors until tokens that show up in similar contexts have similar vectors. That's the whole training principle.

The classic implementation is , which trains a small network on a fill-in-the-blank game: given a word, predict the words around it. The predictions themselves are thrown away — the vectors learned along the way are the actual product. A close cousin, GloVe, gets there differently, by factorising a giant table of how often each pair of words co-occurs.

Think of it like learning a language from subtitles alone:

Drop into a foreign film with no dictionary. You can't look anything up — but you notice that one word keeps appearing right before people drink, and another keeps appearing when they leave a room. After ten thousand scenes you have a decent working sense of both, purely from where they show up. That is word2vec's entire education, and it's the model's too.

Try to recall

Word2vec is trained to predict a word's neighbours, yet nobody uses its predictions. What is the actual product?

Hint: Think about what has to exist inside the model for the prediction to work at all.

Directions carry meaning — the analogy trick

Here is the result that made embeddings famous. Take the vectors for king, man, and woman, and do arithmetic on them:

king − man + woman  ≈  queen

That is not a metaphor. It is a literal subtraction and addition of lists of numbers, and the result lands next to the vector for queen.

A difference is a direction, and directions mean things

Subtracting man from king cancels out everything the two share — person-ness, adulthood — and leaves behind the one thing that distinguishes them: royalty. That leftover is a direction in the space, a "make it royal" arrow. Add it to woman and you arrive at the royal version of a woman. The space has organised itself so that consistent semantic relationships become consistent geometric offsets.

The analogy, done by hand in our toy space

Use the four-trait vectors from earlier. Each list is [person, royalty, male, fruit]:

king = [0.9, 0.9, 0.4, 0.0]
man = [0.9, 0.1, 0.4, 0.0]
woman = [0.9, 0.1, −0.4, 0.0]

Step 1 — king − man, slot by slot:

[0.9−0.9, 0.9−0.1, 0.4−0.4, 0.0−0.0] = [0.0, 0.8, 0.0, 0.0]

Look at what survived: only the royalty slot. Person-ness and maleness cancelled, because king and man agree on both. What's left is a pure "royalty" direction.

Step 2 — add woman:

[0.0+0.9, 0.8+0.1, 0.0+(−0.4), 0.0+0.0] = [0.9, 0.9, −0.4, 0.0]

Step 3 — look it up. That is exactly the vector we wrote down for queen.

The analogy as a parallelogram— interactive, drag & zoom

Loading chart…

The arrow from man to woman and the arrow from king to queen are parallel and the same length — that shared offset is the gender direction. Adding it to any point moves you along the same semantic relationship, which is why the arithmetic works.

Python · runs in your browser

What this does: Does the famous king − man + woman analogy as literal NumPy arithmetic on the toy embedding table, then ranks every token in the vocabulary by cosine similarity to the result. Queen comes out on top — because the subtraction isolated a royalty direction and adding it to woman landed on queen's vector.

import numpy as np

words = ["king", "queen", "man", "woman", "apple", "banana"]
#                person royalty  male  fruit
E = np.array([[   0.9,    0.9,   0.4,   0.0],   # king
              [   0.9,    0.9,  -0.4,   0.0],   # queen
              [   0.9,    0.1,   0.4,   0.0],   # man
              [   0.9,    0.1,  -0.4,   0.0],   # woman
              [   0.1,    0.0,   0.0,   0.9],   # apple
              [   0.05,   0.0,   0.0,   0.88]]) # banana
idx = {w: i for i, w in enumerate(words)}

# The analogy is plain vector arithmetic
target = E[idx["king"]] - E[idx["man"]] + E[idx["woman"]]
print("king - man       =", np.round(E[idx["king"]] - E[idx["man"]], 3), " <- a pure 'royalty' direction")
print("+ woman          =", np.round(target, 3))

# Rank the whole vocabulary by closeness to that result
def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

scores = sorted(((cosine(target, E[i]), w) for i, w in enumerate(words)), reverse=True)
print("\nnearest tokens to (king - man + woman):")
for s, w in scores:
    print(f"  {w:>7}: {s:.3f}")

In our toy space, king − man came out as [0.0, 0.8, 0.0, 0.0] — non-zero in only the royalty slot. Why did the other three slots vanish?

From word vectors to LLM embeddings — three things change

Everything so far describes classic word embeddings. Inside a modern language model the idea is the same, but three details differ, and each one matters.

1. They're token embeddings, not word embeddings. The rows correspond to whatever the tokenizer produces — subword pieces like " un", "believ", "able", not dictionary words. A rare word is assembled from several rows rather than getting its own.

2. They're learned jointly, not in advance. There is no separate word2vec stage. The embedding matrix is initialised randomly along with everything else, and it's trained by the same next-token-prediction loss as the rest of the model. Its geometry is whatever makes next-token prediction easiest.

3. They are only the starting point — and this is the big one. A row in the embedding matrix is static: token " bank" gets the identical vector whether the sentence is about a river or a mortgage. The model can't have it both ways at layer zero.

Static in, contextual out

Think of the embedding vector as the token's dictionary entry — every sense mushed together into one average. The attention layers above are what read the room: each layer lets a token look at its neighbours and update its own vector, so by the middle of the network the two " bank" tokens have drifted to genuinely different places. Embeddings give you word-out-of-context; attention converts that into word-in-this-context. If you remember one thing from this page, make it this — it is precisely the gap that attention exists to close.

Python · runs in your browser

What this does: Demonstrates the limitation that motivates attention. Two sentences use the token 'bank' with completely different meanings, but the embedding lookup is pure table access — so both get byte-for-byte the same vector. Only the layers above can tell them apart.

There's one more gap the embedding layer leaves open. A lookup table has no idea where in the sentence a token sits — "dog bites man" and "man bites dog" produce the same multiset of vectors. Models therefore add a on top of the token embedding, so position and identity travel together into the first attention layer.

Why does a Transformer add positional encodings to token embeddings?

Explain it yourself

Explain to a friend why a language model can't just use token ID numbers directly, and what an embedding gives it instead. Then explain why the model still needs attention on top, using the two meanings of the word bank. If you stall on either half, that's exactly the section to reread.

Recap — the key ideas

A token ID is a name, not a quantity — arithmetic on it is meaningless, so it can't be fed to a network directly.
One-hot vectors fix the arithmetic problem but are long, sparse, and maximally ignorant: every pair of tokens is equally unrelated.
An embedding is a short, dense vector of learned numbers. Tokens are described by shared traits, so similar tokens land near each other automatically.
The embedding matrix is a $|V| \times d$ lookup table — the first layer of an LLM, and mathematically a linear layer applied to a one-hot vector.
Cosine similarity measures closeness by angle, not distance, so vector length (which mostly tracks token frequency) doesn't distort the score.
The numbers come from the distributional hypothesis: tokens appearing in similar contexts get similar vectors. Word2vec learned this explicitly; an LLM gets it for free from next-token prediction.
Directions carry meaning — relationships appear as roughly consistent offsets, which is why king − man + woman lands near queen (with an asterisk on how that demo is evaluated).
Embeddings are static: the same token always gets the same row. Turning that into context-dependent meaning is exactly what attention does.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: attempt each problem before scrolling back up — pulling an answer out of memory beats re-reading it on the page.
• Spaced repetition: mark this topic complete and it joins your Review queue, resurfacing just before you'd forget it.
• Interleaving: mix these with problems from Tokenization and Linear Algebra rather than grinding embeddings alone — messier practice, sturdier memory.

By hand: write down 3-D vectors for cat, dog, and car using traits of your own choosing, then compute all three pairwise cosine similarities. Did you get the ordering you intended? If not, which trait needs rebalancing?
Break the analogy: in the lab below, change woman so the analogy stops working, and articulate precisely why it broke.
Read like a scientist: skim §6.4 of Jurafsky & Martin Ch. 6 and find their argument for cosine over raw dot product. It's the frequency point from the advanced panel above, in the textbook's own words.
Go bigger: in a Colab notebook, load real pretrained vectors (gensim.downloader.load("glove-wiki-gigaword-100")) and run most_similar on a few words. Then try an analogy with most_similar(positive=["king", "woman"], negative=["man"]) — and check whether it still works once you allow the input words as candidates.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Add a new token to the embedding table and make the geometry work. Step 1: run this as-is to see the current nearest neighbours of 'king'. Step 2: add a row for 'prince' that should be close to 'king' but less royal, and a row for 'grape' that should sit with the fruit. Step 3: rerun and check the neighbour lists came out the way you intended — if they didn't, which trait value is wrong?

editor

import numpy as np

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

def neighbours(word, k=3):
    i = words.index(word)
    scored = sorted(((cosine(E[i], E[j]), w) for j, w in enumerate(words) if j != i), reverse=True)
    return scored[:k]

for w in words:
    print(f"{w:>7} -> ", ", ".join(f"{n} ({s:.2f})" for s, n in neighbours(w)))

# TODO 1: append a row for "prince"  -> very much a person, quite royal, male, not a fruit
# TODO 2: append a row for "grape"   -> barely a person, not royal, neutral gender, very fruity
# Hint: words.append("prince") and E = np.vstack([E, [0.9, 0.7, 0.4, 0.0]])
# TODO 3: rerun and confirm prince's nearest neighbour is king, and grape's is apple or banana.

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Ready for the next step? You now have tokens turned into meaningful vectors. Next, see what the model actually does with them in Next-Token Prediction — and then watch those static vectors become context-aware in Self-Attention from Scratch.