Linear Algebra

The math of lists of numbers and the machines that transform them — taught from zero, then built up to the ideas behind neural networks, embeddings, and PCA.

beginner#math#vectors#matrices#eigenvalues

Start here — what this is really about

Forget the symbols for a moment. Linear algebra is just two ideas:

A neat way to write down a list of numbers (that's a vector).
A neat way to describe a machine that turns one list of numbers into another (that's a matrix).

That's the whole game. Everything else — the Greek letters, the formulas — is shorthand invented so people don't have to write full sentences every time. We'll meet each symbol after you already understand the idea it stands for, never before.

How to read this page

The page adapts to you. By default it teaches from first principles. When you're ready — or once you've completed the prerequisites — flip the Depth switch at the top to reveal the formal notation, derivations, and code. Nothing is hidden for good; deeper material sits behind the "Go deeper" panels so you can open it the moment you're curious.

Vectors — a list of numbers

A is nothing fancier than an ordered list of numbers, like [3, 2] or [255, 100, 50].

You already use them constantly:

A location on a map: 3 steps east, 2 steps north → [3, 2].
The color of a pixel: red 255, green 100, blue 50 → [255, 100, 50].
A movie's genre mix: 80% action, 20% comedy → [0.8, 0.2].

Think of it like an arrow:

A vector [3, 2] is an arrow starting at the origin (the point 0, 0) and reaching over to the point 3 across and 2 up. The list of numbers and the arrow are the same thing seen two ways — numbers for computing, arrow for picturing. Keeping both pictures in your head at once is the single most useful habit in this whole subject.

Each number in the list is a , and the number of components is the vector's . A plain single number (like 5) is called a .

Try to recall

Why is [3, 2] considered different from [2, 3]?

Hint: Look at the definition of a vector.

The dot product — "do these point the same way?"

Here's the first genuinely powerful move. Take two vectors, multiply the numbers in matching slots, and add up the results. That single number is the .

Dot product, step by step

Let's compute [1, 2] · [3, 4]:

Multiply the first slots: 1 × 3 = 3
Multiply the second slots: 2 × 4 = 8
Add them up: 3 + 8 = 11

So [1, 2] · [3, 4] = 11. That's it — a list-in, list-in, single-number-out operation.

Why should you care about one number? Because that number tells you how much the two vectors agree in direction:

Big positive → they point roughly the same way.
Zero → they're at right angles (totally unrelated).
Negative → they point in opposing directions.

This one operation is everywhere in ML

"How similar are these two things?" is the question machine learning asks constantly — how similar are two documents, two faces, two words. The honest answer, almost every time, is a dot product. It's the similarity score in a recommendation engine, the "attention" one word pays to another in a language model, and the core computation inside every single neuron. Master this and a huge amount of ML stops looking like magic.

Matrices — a machine that transforms vectors

A is a grid of numbers. But the number grid is not the point — what a matrix does is the point.

A matrix is a function for whole spaces

An ordinary function like $f(x) = 2x$ takes one number and returns another. A matrix does the same thing, but for entire vectors: feed a vector in, get a (usually different) vector out. Geometrically, that transformation rotates, stretches, squishes, or shears all of space at once. "Multiplying by a matrix" and "applying that transformation" are the same sentence.

Think of it like a coffee grinder for space:

Beans go in, grounds come out — the grinder is the fixed machine, the beans are what changes. A matrix is the machine; vectors are the beans. The same matrix applied to different vectors is the same transformation applied to different points.

Don't just take my word for it — see it. Pick a transformation below and watch the entire grid of space bend. The two coloured arrows (î and ĵ) always land on the matrix's two columns — that's the whole secret of what a matrix is.

Linear transformations & eigenvectors in 2D— GeoGebra, drag & exploreOpen on GeoGebra →

Loading interactive visualization…

Type a 2×2 matrix in the top-left box and drag the point around the circle to watch where it lands. Then click 'Show eigenvectors' to reveal the special directions that only stretch, never turn — the eigenvectors you'll meet next.

Applying a 2×2 matrix by hand

Take the matrix that doubles the x-axis and leaves y alone, applied to the vector [3, 2]:

\begin{bmatrix} 2 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 3 \\ 2 \end{bmatrix}

Work row by row — each output number is a dot product of a matrix row with the input vector:

Top row [2, 0] · [3, 2] = 2×3 + 0×2 = 6
Bottom row [0, 1] · [3, 2] = 0×3 + 1×2 = 2

Out comes [6, 2] — the point got stretched to twice as far along x, exactly as promised. Notice the whole thing was just dot products, the operation you already learned.

Try to recall

A matrix transforms a vector by taking, for each output slot, the ___ of a matrix row with the input vector.

Hint: It's the operation from the previous section.

If A is 3×4 and B is 4×2, what is the shape of AB?

Python · runs in your browser

What this does: Builds a tiny 'linear layer' — a matrix A times an input vector x, the exact operation inside every neural-network layer — and confirms it doubles x as designed. Then it shows the shape rule: a 4×8 matrix times an 8×2 works only because the inner 8s match, giving a 4×2.

Eigenvectors — the directions a transform never turns

Most vectors get knocked off their line when a matrix transforms them — they rotate to a new direction. But for any given transformation, a few special directions only get stretched or shrunk, never rotated. Those are the , and the amount each one is stretched is its .

Think of it like the axis of a spinning globe:

Spin a globe. Almost every point moves — but the two points on the axis stay put. The axis is the transformation's eigenvector: the one direction the spin leaves pointing the same way. Find a transformation's eigenvectors and you've found its "natural axes," the directions it treats most simply.

Python · runs in your browser

What this does: Asks NumPy for the eigenvalues and eigenvectors of a symmetric 2×2 matrix, then checks the defining equation A·v = λ·v for the first pair — it prints A @ v0 and λ0·v0 so you can see they land on the same vector, proving v0 is a direction the matrix only scales, never rotates.

Putting it to work — PCA (best opened once the ideas above click)

Suppose your data has 1000 numbers per example but really only varies in a handful of meaningful ways. finds those few directions of greatest variation so you can compress the data while keeping what matters.

The one-sentence version

PCA asks: "Along which directions does my data actually spread out the most?" — and answers it by taking the top eigenvectors of the data's covariance. The big-eigenvalue directions are where the action is; the tiny-eigenvalue directions are mostly noise you can drop.

What do the top eigenvectors of a data covariance matrix represent?

Where this shows up in CV & ML

Every one of these is an idea from above in disguise:

A fully-connected layer is $\mathbf{y} = W\mathbf{x} + \mathbf{b}$ — a matrix transform plus a shift.
A convolution is a structured matrix multiply (the same weights slid across the image).
Attention is a batch of dot products (similarities) followed by a weighted sum.
Image rotations and warps are matrix multiplies in coordinate space.

Explain it yourself

In your own words, explain what it means to multiply a vector by a matrix — using the coffee-grinder or arrow picture, not symbols. If you stall, that's the exact spot to reread.

Recap — the key ideas

A vector is an ordered list of numbers — and equally, an arrow in space.
The dot product multiplies matching slots and sums them; it measures how much two vectors point the same way, and it's the atom of ML similarity.
A matrix is a machine that transforms vectors; each output entry is a dot product of a matrix row with the input.
Eigenvectors are the directions a transform only scales (by their eigenvalue) — a transformation's natural axes.
PCA/SVD use those eigenvectors to find the directions data varies along, so you can compress it.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: do the problems below before rereading — pulling an answer from memory beats recognizing it on the page.
• Spaced repetition: mark this topic complete and it's added to your Review queue, resurfacing right before you'd forget.
• Interleaving: mix these with problems from Calculus rather than grinding one type — messier practice, sturdier memory.

By hand: compute the eigenvalues of [[2,0],[0,3]] and of [[0,1],[1,0]]. Which matrix has a direction that stays put, and which flips things?
From scratch: implement PCA in NumPy and reduce MNIST digits to 2-D; scatter-plot them and see the clusters.
Prove it numerically: verify $(AB)^\top = B^\top A^\top$ on random matrices.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Verify the eigenvector equation A·v = λ·v yourself. Run the starter to see it hold for the first eigenpair, then do the TODO: change A to [[0., 1.], [1., 0.]] (a flip) and rerun. What happens to the eigenvalues, and why does a pure flip have no direction that stays put?

editor

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Ready for the next step? Continue to Calculus, then Optimization — where these vectors and matrices start to learn.