Unsupervised Learning

Clustering, PCA, and dimensionality reduction — finding structure without labels. Taught from zero: sort data into groups with k-means, squeeze it down with PCA, and learn how to check whether the structure you found is real.

intermediate#clustering#pca#kmeans#dimensionality-reduction

Start here — learning with no answer key

Everything you've met so far in machine learning came with an answer key. You showed the model a photo and told it cat. You showed it a house and told it $400,000. The model's whole job was to copy that mapping. Those answers have a name — — and learning from them is called supervised learning.

Now take the answer key away.

You're handed a spreadsheet of 50,000 customers, or a folder of a million photos, or a pile of server logs, and nobody tells you what any of it is. There's nothing to predict, because there's no correct answer to predict. All you have is the data itself.

The one-sentence version

Unsupervised learning is finding structure in data that came with no answers. Not "predict this label" but "what's in here?" — which things belong together, which directions actually matter, and what a thousand confusing numbers really boil down to.

covers a family of methods, but nearly all of them are doing one of two things:

Grouping — put similar things in the same pile. That's .
Squeezing — describe each thing with far fewer numbers, without losing what matters. That's .

This page teaches both, from zero.

Think of it like moving into a house full of unlabelled boxes:

The previous owner left a hundred boxes with nothing written on them. Nobody can tell you what's inside. So you start opening: kitchen stuff here, tools there, books over there. You invented the categories — they weren't given to you, you discovered them from the contents. That's clustering. Then you notice each box only really differs in a couple of ways (which room, how heavy), so you write two numbers on each lid instead of listing every item. That's dimensionality reduction.

How to read this page

It starts from first principles and assumes no prior knowledge. Flip the Depth switch at the top when you want the formal notation, the objective functions, and the proofs — they open automatically once you've finished the prerequisite (Machine Learning). Nothing is hidden for good; the deeper material sits behind the "Go deeper" panels.

Try to recall

What makes a problem unsupervised rather than supervised?

Hint: Think about what the training data does and does not contain.

Clustering — putting similar things in the same pile

Look at these 36 points. Nobody labelled them, but your eye instantly does the work: there are clearly three groups. Clustering is the attempt to get an algorithm to see what you just saw.

36 unlabelled points — and the three groups an algorithm found— interactive, drag & zoom

Loading chart…

Colours are not given labels: they are what k-means decided after four passes over the data. The orange crosses are the three cluster centres it settled on. Drag to rotate the view or zoom in on a group.

Why would you want this? Because "which pile does this belong in?" is a genuinely useful question:

Customer segmentation — group shoppers by behaviour, then market to each group differently.
Anomaly detection — a point that belongs to no pile is worth a human's attention (fraud, a broken sensor, an intrusion).
Compressing data — replace a million colours with the 16 most representative ones.
Organising a mess — group support tickets, news articles, or genes into themes nobody wrote down in advance.

k-means — the workhorse

The most-used clustering algorithm in the world is also one of the simplest. Here is the entire idea, with no math at all:

Decide how many piles you want. Call that number k.
Drop k flags anywhere on the map.
Assign: every point joins whichever flag is closest to it.
Update: every flag walks to the middle of the points that just joined it.
Go back to step 3. Repeat until the flags stop moving.

That's . Steps 3 and 4 alternate, and each one makes the grouping a little better, until it can't improve any more.

Think of it like ice-cream trucks on a crowded beach:

Three trucks park at random spots. Every sunbather walks to the nearest truck (assign). Each driver looks at their queue and thinks "I should be parked in the middle of these people," and moves there (update). Now some sunbathers are closer to a different truck, so they switch queues — and the drivers move again. After a few rounds nobody wants to switch and no driver wants to move. The trucks have found the three crowds.

The flags have a proper name: each one is a , and it's literally just the average of its members.

Two rounds of k-means, entirely by hand

Six points on a line — as simple as data gets:

1, 2, 3, 10, 11, 12

We want k = 2 piles. Suppose the two centroids start (badly) at 1 and 2.

Round 1 — assign. Each point joins the nearer centroid:

Point 1: distance 0 to centroid-1, distance 1 to centroid-2 → joins centroid-1.
Point 2: distance 1 vs 0 → joins centroid-2.
Points 3, 10, 11, 12: all nearer to 2 than to 1 → all join centroid-2.

Round 1 — update. Each centroid moves to the average of its members:

centroid-1 holds 1 → new position 1.
centroid-2 holds 2, 3, 10, 11, 12 → new position (2+3+10+11+12)/5 = 7.6.

Round 2 — assign. With centroids now at 1 and 7.6:

1, 2, 3 are nearer to 1 (distances 0, 1, 2 versus 6.6, 5.6, 4.6).
10, 11, 12 are nearer to 7.6.

Round 2 — update. centroid-1 → (1+2+3)/3 = 2. centroid-2 → (10+11+12)/3 = 11.

Round 3 — assign. Nothing changes: 1, 2, 3 still belong to 2, and 10, 11, 12 still belong to 11. The centroids don't move. Converged.

Starting from a genuinely lopsided guess, two rounds were enough to find the obvious answer: one pile near 2, one pile near 11.

Try to recall

In k-means, what exactly does a centroid move to on the update step?

Hint: It is the simplest summary of a set of points you can think of.

Now watch it happen. This is k-means in about fifteen lines — no library, just the assign/update loop you read above:

Python · runs in your browser

What this does: Implements k-means from scratch on 36 points that form three blobs. Each pass prints the inertia (total squared distance from points to their own centre) so you can watch it drop and then stop, and the final plot shows the three groups with orange X marks at the centroids. Try changing k to 2 or 5 and rerun to see how the picture and the inertia change.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(7)
true_centers = np.array([[1.5, 1.2], [6.2, 5.8], [8.0, 1.0]])
X = np.vstack([rng.normal(c, 0.75, size=(12, 2)) for c in true_centers])

k = 3
centroids = X[rng.choice(len(X), k, replace=False)]   # step 0: k random points as flags

for step in range(8):
    # 1. ASSIGN: each point joins the nearest centroid
    d2 = ((X[:, None, :] - centroids[None, :, :]) ** 2).sum(axis=-1)
    labels = d2.argmin(axis=1)
    # 2. UPDATE: each centroid moves to the mean of its members
    new_centroids = np.array([X[labels == j].mean(axis=0) for j in range(k)])
    inertia = d2[np.arange(len(X)), labels].sum()
    print(f"step {step}: inertia = {inertia:7.2f}")
    if np.allclose(new_centroids, centroids):
        print("centroids stopped moving -> converged")
        break
    centroids = new_centroids

plt.figure(figsize=(5, 3.6))
plt.scatter(X[:, 0], X[:, 1], c=labels, cmap="viridis", s=28)
plt.scatter(centroids[:, 0], centroids[:, 1], marker="X", s=180, c="#d97757", label="centroids")
plt.legend(); plt.title("k-means, k=3"); plt.tight_layout()

The printed inertia falls from about 361 to 44 to 30, then the centroids stop moving. That plateau is convergence.

You run k-means twice on the same data with the same k and get two different clusterings. What is the most likely explanation?

Choosing k — the number nobody gives you

k-means needs you to name k up front. But if you knew how many natural groups your data had, you'd already know a lot about it. So how do you choose?

The tempting move is "pick the k with the lowest inertia" — and that's a trap.

Why lowest inertia is a trap

Inertia can always be lowered by adding another cluster. Push $k$ up to the number of data points and every point becomes its own cluster sitting exactly on its own centroid, giving an inertia of zero — and telling you absolutely nothing. Inertia is a measure of tightness, not of truth.

So instead of the lowest point, you look for the elbow: the value of k after which extra clusters stop buying you much.

Elbow plot — inertia against k for the 36-point dataset— interactive, drag & zoom

Loading chart…

Going from k=2 to k=3 slashes the inertia from 188 to 30. Going from 3 to 4 saves only 6, and every step after that saves less still. The sharp bend at k=3 is the elbow — and 3 is the number of blobs actually in the data.

Compute it yourself — and note the detail that matters most, the restarts:

Python · runs in your browser

What this does: Runs k-means for every k from 1 to 8 and prints the inertia of each, so you can find the elbow numerically instead of by eye. Note the best-of-10-restarts line: because k-means is sensitive to its random start, a single run per k can produce a bumpy, misleading curve.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(7)
true_centers = np.array([[1.5, 1.2], [6.2, 5.8], [8.0, 1.0]])
X = np.vstack([rng.normal(c, 0.75, size=(12, 2)) for c in true_centers])

def kmeans_inertia(X, k, seed):
    r = np.random.default_rng(seed)
    C = X[r.choice(len(X), k, replace=False)]
    for _ in range(20):
        d2 = ((X[:, None, :] - C[None, :, :]) ** 2).sum(-1)
        lab = d2.argmin(1)
        newC = np.array([X[lab == j].mean(0) if (lab == j).any() else C[j] for j in range(k)])
        if np.allclose(newC, C):
            break
        C = newC
    d2 = ((X[:, None, :] - C[None, :, :]) ** 2).sum(-1)
    return d2.min(1).sum()

ks = range(1, 9)
# best of 10 restarts per k -- k-means is seed-sensitive, so never trust one run
inertias = [min(kmeans_inertia(X, k, s) for s in range(10)) for k in ks]
for k, v in zip(ks, inertias):
    print(f"k={k}: inertia = {v:7.1f}")

plt.figure(figsize=(5, 3))
plt.plot(list(ks), inertias, "o-", color="#d97757")
plt.xlabel("k"); plt.ylabel("inertia"); plt.title("Elbow plot"); plt.tight_layout()

The elbow is a judgement call, and on real data it's often no sharper than a gentle curve. A more principled companion is the , which asks a different question — not "how tight are the clusters?" but "is each point in the right one?"

Python · runs in your browser

What this does: Computes the silhouette score by hand for six points that form two obvious clusters — for each point it measures the average distance to its own group (a) and to the nearest other group (b), then scores (b-a)/max(a,b). All the scores come out near 1, which is what clean, well-separated clusters look like. Try moving one point to the middle, like [3.5, 3.5], and watch its score collapse toward 0.

Try to recall

Why can you not choose k by simply picking the value with the lowest inertia?

Hint: Imagine setting k equal to the number of data points.

When k-means is the wrong tool

k-means is fast and it is everywhere, but it makes strong assumptions, and it fails quietly when they don't hold — it always returns k tidy-looking clusters, even when the answer is nonsense.

The four ways k-means misleads people

It assumes round, similarly-sized blobs. Because every point goes to the nearest centre, cluster boundaries are always straight lines. Crescents, rings, and long thin filaments get chopped straight through the middle.
It is not scale-invariant. A feature measured in metres and one measured in millimetres do not contribute equally — the big-numbered feature dominates the distance and effectively decides the clustering on its own. Standardise your features first.
It forces every point into a cluster. There is no "this is an outlier" option, so a single extreme point can drag a whole centroid off course.
It needs k in advance, and it will happily give you 5 clusters from data that has 2.

When those assumptions break, reach for a different algorithm:

Algorithm	How it groups	Reach for it when
k-means	Nearest centroid	Round, similar-sized blobs; large datasets; you can guess k
Hierarchical	Repeatedly merges the two closest groups	You want a tree of nested groupings and to pick k afterwards
DBSCAN	Grows clusters through dense neighbourhoods	Odd shapes, unknown k, and you want outliers labelled as noise
Gaussian mixture	Fits overlapping bell-shaped blobs	Clusters overlap, are elongated, or you want a probability per point

Your two features are annual income (roughly 20,000 to 200,000) and number of purchases (roughly 1 to 30). You run k-means without preprocessing. What happens?

Dimensionality reduction — fewer numbers, same story

Now the second half of unsupervised learning. Forget groups; here the question is about width.

A 28×28 grayscale image of a handwritten digit is 784 numbers. But handwritten digits don't vary in 784 independent ways — they vary in slant, thickness, loop size, and a handful of other things. Most of those 784 numbers are near-duplicates of their neighbours. The data is described with 784 numbers but really lives in a much smaller space.

that's far higher than necessary costs you real money: more memory, slower models, noisier distances, and more training data needed to learn anything. Shrinking it honestly is one of the highest-leverage moves in a data pipeline.

Think of it like photographing a chair:

A chair is a 3-D object, but a photograph of it is flat — 2-D — and you can still instantly tell it's a chair. That works only because you chose a good angle. Shoot it from directly above and you get an unrecognisable blob; shoot from a three-quarter view and every important feature survives. Dimensionality reduction is choosing that camera angle automatically: throw away a dimension, but throw away the least informative one.

What makes an angle good

A good viewing angle is the one that keeps things spread out. From a bad angle, distinct parts of the object land on top of each other and become impossible to tell apart. From a good angle, the shadow is wide and detailed. "Spread out" has a precise name in statistics — variance — so the search for the best angle becomes: find the direction in which the data varies most. That single sentence is PCA.

PCA — finding the directions data actually varies along

takes a cloud of points and finds its natural axes, in order of importance. The first — the — is the single direction of greatest spread. The second is the direction of greatest remaining spread, at right angles to the first. And so on.

Here's the recipe, in plain words:

Centre the data — subtract the average, so the cloud sits around the origin.
Compute the covariance matrix — a small table of how every pair of features moves together.
Take its eigenvectors — those are the principal directions, and each one's eigenvalue is how much variance lives along it.
Keep the top few, project the data onto them, and you're done.

Steps 3 and 4 are the eigenvectors you already met, doing a real job.

PCA on four points, with clean numbers

Four points: (5, 6), (1, 2), (4, 3), (2, 5).

Step 1 — centre. The mean is $\left(\frac{5+1+4+2}{4}, \frac{6+2+3+5}{4}\right) = (3, 4)$ . Subtract it from every point:

(2, 2), (-2, -2), (1, -1), (-1, 1)

Step 2 — covariance. For centred data, each entry is an average of products over the 4 points:

Top-left (feature 1 with itself): $\frac{2^2 + (-2)^2 + 1^2 + (-1)^2}{4} = \frac{10}{4} = 2.5$
Bottom-right (feature 2 with itself): same arithmetic → $2.5$
Off-diagonal (feature 1 times feature 2): $\frac{(2)(2) + (-2)(-2) + (1)(-1) + (-1)(1)}{4} = \frac{4 + 4 - 1 - 1}{4} = 1.5$

So the covariance matrix is:

\Sigma = \begin{bmatrix} 2.5 & 1.5 \\ 1.5 & 2.5 \end{bmatrix}

The positive off-diagonal says the two features tend to rise together.

Step 3 — eigenvectors. For a matrix of this shape the answer is exact and pretty:

eigenvalue 4, direction $\frac{1}{\sqrt{2}}(1, 1)$ — the up-and-right diagonal.
eigenvalue 1, direction $\frac{1}{\sqrt{2}}(-1, 1)$ — the perpendicular one.

Check the first one — apply the matrix to that direction and it comes back only stretched, never turned, which is exactly the eigenvector condition:

\begin{bmatrix} 2.5 & 1.5 \\ 1.5 & 2.5 \end{bmatrix}\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 4 \\ 4 \end{bmatrix} = 4\begin{bmatrix} 1 \\ 1 \end{bmatrix}

Step 4 — read it off. The eigenvalues total $4 + 1 = 5$ , so the first direction holds $4/5 =$ 80% of the variance and the second holds 20%. Keeping only the first component turns each 2-number point into 1 number — for example the centred point (2, 2) becomes its dot product with $\frac{1}{\sqrt{2}}(1,1)$ , which is $\frac{2 + 2}{\sqrt{2}} \approx 2.83$ — and we've kept 80% of the story with half the numbers.

The same thing on a real cloud of 40 points. Notice the long axis threads straight through the length of the cloud, and the short one crosses it at a right angle:

A data cloud and its two principal components— interactive, drag & zoom

Loading chart…

Orange is PC1 — the single direction of greatest spread, holding 97.7% of this data's variance. Teal is PC2, at right angles to it, holding the remaining 2.3%. Each line is drawn with a length proportional to the spread along it, which is why PC2 is barely a stub: projecting onto PC1 alone would lose very little.

PCA in five steps and nine lines of NumPy:

Python · runs in your browser

What this does: Runs PCA from scratch on 40 two-dimensional points: centre the data, build the covariance matrix, eigen-decompose it, then project onto the single top direction and rebuild. It prints the explained variance ratio (about 98% on the first component) and the tiny error left after rebuilding from just one number per point instead of two.

import numpy as np

rng = np.random.default_rng(7)
t = rng.normal(0, 2.0, 40)
X = np.stack([t + rng.normal(0, 0.35, 40), 0.55 * t + rng.normal(0, 0.35, 40)], axis=1)

mu = X.mean(axis=0)          # 1. CENTER: subtract the mean of every column
Xc = X - mu
cov = Xc.T @ Xc / len(Xc)    # 2. COVARIANCE: how features move together
vals, vecs = np.linalg.eigh(cov)     # 3. EIGEN-DECOMPOSE (eigh: symmetric matrices)

order = vals.argsort()[::-1]         # eigh returns ascending -- flip to biggest-first
vals, vecs = vals[order], vecs[:, order]

print("eigenvalues (variance per direction):", np.round(vals, 3))
print("explained variance ratio:", np.round(vals / vals.sum(), 4))
print("PC1 direction:", np.round(vecs[:, 0], 3))

z = Xc @ vecs[:, :1]                 # 4. PROJECT onto the top-1 direction: 2 numbers -> 1
X_rebuilt = z @ vecs[:, :1].T + mu   # 5. REBUILD and see how much we lost
print("shape after PCA:", z.shape)
print("mean squared reconstruction error:", round(float(((X - X_rebuilt) ** 2).mean()), 4))

Halving the size of every point cost a mean squared error of about 0.04 — on data that spans roughly ten units. That is the trade PCA offers, made numerical.

Reading a scree plot — how many components to keep

With two features "keep 1 of 2" is not much of a decision. The tool earns its keep on wide data. Below is a 10-feature dataset that was secretly generated from just 3 underlying factors plus noise — exactly the situation PCA is built to expose:

Scree plot — variance explained by each of 10 components— interactive, drag & zoom

Loading chart…

Bars show each component's share of the variance; the line shows the running total. Three components carry 98.1% of everything, and components 4 through 10 are flat noise at roughly 0.3% each. The cliff after component 3 recovers the true number of hidden factors — you can drop 7 of the 10 dimensions and lose almost nothing.

A is read the same way as the elbow plot: find where the curve goes flat, and keep everything before it.

Try to recall

A colleague runs PCA on a dataset without standardising, and finds PC1 explains 99.8% of the variance. Why should you be suspicious?

Hint: What does variance depend on besides the data's shape?

You reduce 100 features to 10 with PCA and the explained variance ratios sum to 0.95. What does that mean?

Beyond straight lines — t-SNE and UMAP

PCA can only rotate and flatten. When the interesting structure is curved, you need methods that bend.

Where this shows up in real ML

Every one of these is a technique from this page, wearing a job title:

Vector search and RAG — cluster millions of embeddings so a query only searches a few nearby groups instead of all of them.
Anomaly and fraud detection — flag whatever sits far from every cluster centre, or whatever reconstructs badly after PCA.
Image compression and denoising — keep the top components, discard the rest; the discarded tail is mostly noise.
Preprocessing — PCA before a classical model kills redundant correlated features and speeds training up.
Exploration — project a new dataset to 2-D and look at it before writing a single model.
Self-supervised learning — modern pretraining is unsupervised learning's grown-up form: no human labels, structure invented from the data itself.

Explain it yourself

Explain to a friend, without any formulas, what k-means does each round and why PCA is like choosing the best camera angle. Then answer the harder one: what is each method assuming about the data, and when would that assumption be wrong? If you stall on the assumptions, that is the section to reread.

Recap — the key ideas

Unsupervised learning finds structure in data with no labels — mainly by grouping it (clustering) or squeezing it (dimensionality reduction).
k-means repeats two steps: assign each point to the nearest centroid, then move each centroid to the mean of its members. It's minimising inertia, the total squared distance from points to their own centre.
It converges, but only to a local minimum — so use k-means++ seeding and multiple restarts, and keep the lowest-inertia run.
Choosing k can't be done by lowest inertia (that always says "more clusters"). Use the elbow of the inertia curve plus the silhouette score.
k-means assumes round, similar-sized, comparably-scaled blobs. Standardise your features, and switch to DBSCAN, hierarchical clustering, or a Gaussian mixture when the shapes are odd.
PCA centres the data, takes the eigenvectors of the covariance matrix as new axes, and keeps the few with the biggest eigenvalues — the directions of greatest variance.
The explained variance ratio and its scree plot tell you how many components are worth keeping; PCA is linear, scale-sensitive, and blind to your labels.
t-SNE and UMAP handle curved structure and make beautiful maps — but cluster sizes and between-cluster distances on those maps should not be trusted.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: close the page and try to write out the two steps of k-means and the four steps of PCA from memory before rereading. Struggling to recall builds far more durable memory than re-reading does.
• Spaced repetition: mark this topic complete and it joins your Review queue, resurfacing right before you'd forget it.
• Interleaving: mix these exercises with Linear Algebra eigenvector problems and Feature Engineering scaling questions rather than grinding one type — messier practice, sturdier memory.

By hand: run k-means with k = 2 on the points 2, 4, 6, 20, 22, 24, starting from centroids at 4 and 6. How many rounds until it stops? Now start from 2 and 24 — does it reach the same answer?
By hand: compute the covariance matrix of the centred points (1, 1), (-1, -1), (1, -1), (-1, 1). What are its eigenvalues, and what does the answer say about the principal directions of a perfectly square cloud?
From scratch: implement the silhouette score and use it to pick k on the three-blob dataset above. Does it agree with the elbow?
The failure case: the lab below. Generate two interleaved crescents, run k-means, and see it fail — then work out exactly which assumption it broke.
On real data: load a dataset with 10+ numeric features, standardise it, run PCA, and plot the scree curve. How many components reach 95%? Then re-run without standardising and compare — the difference is usually shocking.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Watch k-means fail. The data below is two interleaved crescents, which any human eye separates instantly, but k-means cuts straight through both because it can only draw straight boundaries between centroids. Run it first and look at the plot. Then do the TODOs: (1) set k = 4 and see whether more clusters help or just make a finer mess; (2) print the cluster sizes and think about what a correct answer would look like. Finally, answer in your own words: which of k-means four assumptions does this data break, and which algorithm from the table above would you reach for instead?

editor

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
th = np.linspace(0, np.pi, 60)
moon_a = np.stack([np.cos(th), np.sin(th)], 1)
moon_b = np.stack([1 - np.cos(th), 0.5 - np.sin(th)], 1)
X = np.vstack([moon_a, moon_b]) + rng.normal(0, 0.06, (120, 2))

def kmeans(X, k, seed=0):
    r = np.random.default_rng(seed)
    C = X[r.choice(len(X), k, replace=False)]
    for _ in range(30):
        lab = ((X[:, None, :] - C[None, :, :]) ** 2).sum(-1).argmin(1)
        newC = np.array([X[lab == j].mean(0) if (lab == j).any() else C[j] for j in range(k)])
        if np.allclose(newC, C):
            break
        C = newC
    return C, lab

k = 2                      # TODO (1): try k = 4
C, lab = kmeans(X, k)

plt.figure(figsize=(5, 3.2))
plt.scatter(X[:, 0], X[:, 1], c=lab, cmap="coolwarm", s=18)
plt.scatter(C[:, 0], C[:, 1], marker="X", s=160, c="k")
plt.title(f"k-means on two crescents (k={k})"); plt.tight_layout()

print("cluster sizes:", np.bincount(lab))   # TODO (2): what would a correct split look like?

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Next up: put labels back on the data and learn to judge a model honestly in Evaluation & Metrics, or stack simple models into strong ones in Ensembles & Boosting.