Generative & Diffusion Models

Teaching a computer to invent brand-new images — built from zero. Start with the one-line trick behind Stable Diffusion (add noise, then learn to take it away), then climb to the math that makes it work.

expert#diffusion#gan#vae#generative

Start here — what a generative model even is

Most machine learning you have seen so far is a judge: you show it a photo and it answers a question — cat or dog? tumor or healthy? That is called discriminative — it discriminates between things that already exist.

A does the opposite and much harder job: it is an artist. Nobody hands it a photo. It has to conjure one out of thin air — a face that has never existed, a landscape nobody photographed. This is what powers Stable Diffusion, Midjourney, and DALL·E.

Think of it like the difference between a critic and a painter:

A film critic can tell you a good movie from a bad one, but has never had to make one. A painter starts with a blank canvas and must produce something from nothing. Discriminative models are critics; generative models are painters. Painting is the far harder skill — which is why generative models took decades longer to get good.

To "paint from nothing," the model needs to learn what the whole world of images looks like — which pixel patterns are plausible photos and which are random garbage. Then it can reach into that learned world and pull out a fresh sample. That act of pulling out a new example is called , and it is the whole point of everything below.

How to read this page

It starts from first principles — no probability background assumed. Flip the Depth switch at the top when you want the formal equations and the derivations behind them; they will also open automatically once you have finished the prerequisites (CNNs and Probability & Statistics).

Three ways to fake a photo

Before diffusion won, two other families of "painter" were tried. You do not need the details yet — just the one-line gist of each, so you can see why diffusion took over.

VAE, GAN, Diffusion — in one breath each

A VAE squashes each image down to a short summary and learns to expand summaries back into images. Reliable, but its images come out a little blurry.
A GAN pits two networks against each other — a forger and a detective — until the forger fools the detective. Razor-sharp images, but the training is a knife-edge and often collapses.
A Diffusion model learns to remove noise, one gentle step at a time. Slow to run, but stable to train and stunning in quality. This is today's champion.

Model	Core idea	Strength	Weakness
VAE	Encode to a compact summary, decode back	Stable, tidy summary space	Blurry samples
GAN	Forger vs. detective, adversarial game	Sharp, one-shot sampling	Unstable, can collapse
Diffusion	Undo noise step by step	Top quality + variety, stable	Slow (many steps)

Try to recall

What is the core task a generative model has to do that a classifier never does?

Hint: Think painter versus critic.

The big idea — destroy, then learn to rebuild

Here is the entire trick behind diffusion, and it is almost silly how simple it sounds:

Take a photo and slowly smother it in noise until it is pure TV static. Train a network to undo one small step of that smothering. To make a new image, start from fresh static and let the network un-smother it, step by step, until a picture appears.

Think of it like cream swirling into coffee:

Pour cream into coffee and stir: it starts as a sharp white blob and diffuses into a uniform tan — easy to watch, and the end state (uniform) carries no memory of the blob. Now imagine a video of that, played backwards: the uniform coffee spontaneously gathers the cream back into a crisp blob. That backwards video is impossible for coffee — but a diffusion model learns to fake it for images. Forward (adding noise) is easy and fixed; the magic is learning to run it in reverse.

Why break it just to fix it?

Un-smothering static into a photo in one leap is impossibly hard — there are too many choices. But removing a tiny bit of noise from a slightly noisy image is easy: you can almost see the picture underneath. Diffusion turns one impossible problem into a thousand easy ones, chained together. Each step barely does anything; a thousand of them turn noise into art.

So there are two processes, and you must keep them straight:

The forward process — add noise. Fixed, dumb, no learning. We define it ourselves.
The reverse process — remove noise. This is the neural network, and the only thing we train.

The next two sections take them one at a time.

Step 1: the forward process — adding noise

The forward process is a recipe we write down. It takes a clean image and, over many steps, dissolves it into random noise. Crucially, nothing is learned here — it is a fixed procedure, like a kitchen timer.

is the technical name: at each step we take the current (slightly noisy) image and add a little more random noise, and that is the only thing the next image depends on.

The noise we add is — the classic bell curve. Drag the sliders below to feel what a Gaussian is: the peak is where values cluster, and the width is how violently they spread.

The Gaussian (normal) distribution— GeoGebra, drag & exploreOpen on GeoGebra →

Loading interactive visualization…

Move the mean slider to slide the bell left and right, and the standard-deviation slider to make it narrow and tall or wide and flat. Diffusion adds noise drawn from exactly this bell curve at every step — start narrow (a tiny nudge) and picture it widening until the noise drowns the image.

One noising step, by hand

Forget images for a second — noise a single pixel whose brightness is $x = 0.80$ .

The recipe says: keep most of the pixel, then sprinkle a little noise on top. Say we keep 98% of it and add noise with strength 0.14.
Draw one random number from the bell curve — suppose we get $\epsilon = -0.5$ (a slightly-below-average draw).
New pixel $= \sqrt{0.98}\times 0.80 + 0.14\times(-0.5) = 0.99\times 0.80 - 0.07 = 0.79 - 0.07 = 0.72$ .

The pixel barely moved — from 0.80 to 0.72. That is one step. Do it a thousand times, each nudging a little, and the original 0.80 is long forgotten: the pixel is now just a random draw from the bell curve.

Run the forward process yourself. Watch a clean signal (stand-in for an image) dissolve into static as the step counter climbs:

Python · runs in your browser

What this does: Takes a clean 1-D "image" (a smooth wave) and applies the diffusion forward shortcut x_t = sqrt(ᾱ_t)·x0 + sqrt(1-ᾱ_t)·ε at four timesteps. Watch the wave fade and the noise take over as t grows — exactly what diffusion does to a real photo, one pixel at a time. The printed ᾱ values show the surviving fraction of the original dropping toward 0.

Try to recall

In the forward process, which part is learned by the neural network?

Hint: Re-read the first sentence of this section.

Step 2: the reverse process — learning to denoise

Now the actual learning. We want to walk the chain backwards: given a noisy image, produce a slightly less noisy one. Do that repeatedly and static becomes a picture.

Predict the noise, not the picture

Here is the clever reframing that makes it all click. Instead of asking the network the vague question turn this static into a masterpiece, we ask a razor-sharp one: this image is a clean picture plus some noise I sprinkled on — point to exactly the noise. If the network can name the noise that was added, we simply subtract it and the picture underneath emerges. Naming noise is a concrete, learnable target; hallucinating a masterpiece is not.

Think of it like finding the smudges on a window:

You are not asked to repaint the scene behind a dirty window — only to point at the smudges. Wipe exactly those off, and the clean view is already there. The network's whole job is smudge-detection: it looks at a noisy image and predicts where the noise is. Remove it and the image is revealed.

The noise-predictor is a network — usually a — that takes the noisy image and the step number, and outputs its best guess of the noise that was added.

One denoising step, conceptually

Suppose at step $t$ the noisy pixel is $x_t = 0.72$ , and the network predicts the noise that was added was $\hat\epsilon = -0.5$ .

The forward formula said $x_t = (\text{kept signal}) + (\text{noise strength})\times \epsilon$ .
The network just handed us its guess of that $\epsilon$ . So we can subtract the noise term it accounts for, rescale, and land on a cleaner estimate.
Repeat at step $t-1$ , $t-2$ , … each step peeling off a thin layer of noise, until $x_0$ — a clean pixel — remains.

The point: each step is a small, well-posed subtraction, not a wild guess. The heavy lifting is the network naming $\hat\epsilon$ accurately.

Here is that loss as runnable code — one full training step on a toy example, with a stand-in model, so you can see the objective is genuinely just an MSE on the noise:

Python · runs in your browser

What this does: Runs one real DDPM training step on a toy signal: pick a random timestep, noise the clean signal with the forward formula, let a stand-in "untrained model" guess the noise, and score it with the MSE loss ||ε - ε_pred||² — the entire objective that trains Stable Diffusion, in a handful of lines. A trained network would drive this loss toward 0.

And here is the real version in PyTorch — the loop that actually trains modern diffusion models. It is the toy above with a genuine network in place of the zero-guess:

Python · needs a GPU — run on Colab

import torch, torch.nn.functional as F

def train_step(model, x0, betas):
    B = x0.size(0)
    t = torch.randint(0, len(betas), (B,), device=x0.device)   # random step per image
    alpha_bar = torch.cumprod(1 - betas, dim=0)
    a_bar = alpha_bar[t].view(B, 1, 1, 1)

    eps = torch.randn_like(x0)                                 # true noise
    x_t = a_bar.sqrt() * x0 + (1 - a_bar).sqrt() * eps          # forward noising (the shortcut)
    eps_pred = model(x_t, t)                                    # U-Net predicts the noise
    return F.mse_loss(eps_pred, eps)                            # exactly the loss above

What does the network ε_θ(x_t, t) predict in a standard DDPM?

Sampling — turning static into an image

Training is done. To actually make a picture, we start from pure static and run the reverse process to the end.

A thousand tiny wipes

Begin with a fresh sheet of random noise — literally $x_T$ drawn from the bell curve, resembling nothing. Ask the network what is the noise here, subtract a sliver of it, and you get a marginally cleaner image. Ask again, subtract again. After enough passes the static resolves into a coherent picture the model has never seen before. That is sampling: no input photo, just noise in and art out.

Try to recall

Sampling starts from what, and why is it slow?

Hint: Count how many times the network runs.

Latent diffusion — why Stable Diffusion is fast

Running all this on a full 512×512 image means denoising ~260,000 pixels, a thousand times over. That is brutally expensive. Stable Diffusion's key trick fixes it.

Think of it like editing the thumbnail, not the billboard:

Rather than repaint a giant billboard stroke by stroke, you edit a small thumbnail and enlarge it at the end. Latent diffusion runs the whole noisy dance on a tiny compressed version of the image, then blows the result back up to full resolution only once, at the very end.

Diffuse in a compressed space

A VAE first squashes the image into a much smaller grid — a — that keeps the meaningful content but throws away redundant pixel detail. Diffusion runs entirely in that small space, then the VAE decoder expands the finished latent back to a full image. Far fewer numbers to denoise means far less compute, at basically the same quality.

Why this was the breakthrough

Compressing to a latent grid (Stable Diffusion uses an 8× downscale per side) cuts the number of elements to denoise by roughly 64×. That is the difference between diffusion needing a data-center and running on a gaming GPU — the reason image generation reached everyone's laptop.

Why does Stable Diffusion run the diffusion process in a latent space instead of pixel space?

Conditioning & guidance — steering with text

So far the model paints something, but we cannot tell it what. Text-to-image adds a steering wheel.

Whisper the prompt at every step

Turn the prompt (a corgi on a skateboard) into a bundle of numbers with a text encoder like CLIP, and feed that bundle into the denoiser at every step. Now, when the network predicts the noise, it predicts the noise for a corgi-on-a-skateboard image — nudging each denoising step toward pictures that match your words.

The prompt enters the U-Net through , so every region of the image can attend to the words that matter to it. (This is the attention mechanism, connecting text to pixels.)

But conditioning alone is often too timid — the image only loosely follows the prompt. The fix is a surprisingly simple hack:

Try to recall

What does turning up the classifier-free guidance scale w trade off?

Hint: Prompt-adherence versus something else.

Common pitfalls

Things that bite everyone

Sampling is slow — do not use 1000 steps in production; switch to a fast sampler (DDIM, DPM-Solver) at 20–50 steps.
Forgetting the timestep — the network must be told which step $t$ it is on (via a sinusoidal time embedding), or it cannot know how much noise to expect.
A mis-scaled noise schedule — the right $\beta_t$ curve for 64×64 is wrong for 512×512; the image ends up not fully noised or over-noised.
Judging quality by eye only — use a quantitative score like to compare models honestly.

Explain it yourself

Explain diffusion to a friend using the cream-in-coffee picture, with no equations. Why do we bother destroying an image with noise just to learn to rebuild it — and why is predicting the noise easier than predicting the picture? If you stall on either, that is the exact spot to reread.

Recap — the key ideas

A generative model invents brand-new examples (sampling), where a classifier only labels existing ones.
Of the three families — VAE, GAN, Diffusion — diffusion won on quality and training stability.
Forward process: a fixed, unlearned recipe that adds a little Gaussian noise each step until the image is pure static. A closed-form shortcut jumps to any step in one line.
Reverse process: the only trained part. A U-Net predicts the noise in a noisy image; the loss is a plain MSE between true and predicted noise — no adversary, which is why it trains stably.
Sampling starts from pure noise and denoises step by step; it is slow because the network runs once per step.
Latent diffusion runs the whole process in a small compressed space, making Stable Diffusion fast enough for a laptop.
Text conditioning feeds the prompt in via cross-attention; classifier-free guidance (scale $w$ ) dials up how hard the image obeys the prompt.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: attempt the exercises below before rereading — pulling an answer from memory beats recognizing it on the page.
• Spaced repetition: mark this topic complete to add it to your Review queue, resurfacing right before you would forget.
• Interleaving: mix these with CNN and Attention & Transformers problems rather than grinding one topic in a block.

Edit and run the code below — and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Noise an image forward, and watch it become pure staticPython · runs in your browser · Ada can see your terminal

Your task: Implement the diffusion forward step yourself, then answer the question in the TODO. Fill in the one missing line in noise_image so it returns sqrt(ᾱ_t)·x0 + sqrt(1-ᾱ_t)·ε. Run it and read the printed mean and std as t grows.

editor

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Mini project

Train a tiny DDPM on MNIST or Fashion-MNIST; save samples across training and watch digits emerge from noise.
Implement DDIM sampling and compare 1000-step vs 50-step generation — measure both quality and wall-clock time.
Add class conditioning and classifier-free guidance; sweep the guidance scale $w$ and see prompt-adherence trade against diversity.
Compute FID against the real data to put a number on quality.

Explore the 3D frontier next in NeRF & Gaussian Splatting.