Optimization

How a model actually learns — taught from the ground up. Start with the idea of "walking downhill," then build up to gradient descent, momentum, and the Adam optimizer that trains modern networks.

intermediate#gradient-descent#sgd#adam#optimization

Start here — what "training a model" actually means

A model is just a big machine with a lot of knobs. Each knob is a number you can turn. When the knobs are set well, the machine gives good answers; when they're set badly, it gives nonsense.

are those knobs. Training is the process of turning the knobs until the model stops making so many mistakes. That's the entire goal — everything below is just how we decide which way to turn each knob.

To turn the knobs sensibly, we need a single number that says "how wrong is the model right now?" That number is the . High loss = very wrong; low loss = doing well.

Think of it like tuning a guitar by ear:

You pluck a string (make a prediction), hear how far off it is from the right note (the loss), and turn the tuning peg a little (adjust a parameter). Then you listen again and repeat. Training a neural network is the same loop — just with millions of pegs and a number instead of your ear.

This page adapts to you

It starts from first principles. Flip the Depth switch at the top for the formal update rules and derivations — and they'll open automatically once you've completed the prerequisites (Calculus and Linear Algebra).

The core idea — walk downhill

Here's the whole trick. Picture the loss as a landscape: a hilly surface where every location is one setting of the knobs, and the height is how wrong the model is there. Training means finding the lowest valley.

Feel the slope, step downhill, repeat

You're standing on a foggy hillside and can't see the valley. But you can feel which way the ground slopes under your feet. So you take a small step in the downhill direction, then feel the slope again, and step again. Do this enough times and you arrive at the bottom. That's the whole algorithm.

The "which way is downhill, and how steep" information is the . Following it downhill, step by step, is called .

One step downhill, by hand

Take the simplest possible loss, $f(x) = x^2$ — a U-shaped valley with its bottom at $x = 0$ . Suppose we start at $x = 4$ (up the right slope).

The slope of $x^2$ at a point is $2x$ . At $x=4$ , that's $2 \times 4 = 8$ — a steep, positive slope (uphill to the right).
Downhill is the opposite of the slope, so we move left. Take a small step: new $x = 4 - (\text{step size}) \times 8$ .
With a step size of $0.3$ : new $x = 4 - 0.3 \times 8 = 1.6$ . We went from 4 down toward 0 — closer to the bottom.

Repeat that a dozen times and $x$ slides right down into the valley. You just ran gradient descent.

Try to recall

Why do we step in the opposite direction of the gradient?

Hint: The gradient points toward the steepest increase of the loss.

See it happen

Run this — it walks downhill on $f(x)=x^2$ from $x=4$ . This is the single most important loop in all of deep learning, in six lines:

Python · runs in your browser

What this does: Runs 12 steps of gradient descent on f(x)=x² starting at x=4, and plots the parabola with the steps marching down into the valley at 0. Try changing lr: 0.9 overshoots and ping-pongs across the valley, 0.05 barely crawls. This little loop is the core of how every model trains.

Gradient Descent— interactive, try itOpen in lab →

Click anywhere to drop a new starting point.

Learning rate0.15

Momentum0.70

Brighter = higher loss. Watch how a high learning rate overshoots, and how momentum powers through the small bumps toward a minimum.

Drop a starting point on the loss surface and watch the optimizer roll downhill. Turn the learning rate up until it overshoots, add momentum, and see exactly what the equation above does — step by step.

The learning rate — how big a step to take

Notice the lr in the code — the size of each step. It has its own name, the , and it's a — a dial you set, not one the model learns.

Goldilocks steps

Too big a step and you leap clear over the valley and bounce up the far wall — the loss bounces around or blows up. Too small and you inch along, taking forever to reach the bottom. The art is a step size that's just right: bold enough to make progress, gentle enough not to overshoot.

Go back to the code cell above and try it yourself — this is the best way to build intuition:

Set lr = 0.9 and watch the steps overshoot, ping-ponging across the valley.
Set lr = 0.05 and watch them crawl, barely moving.

Try to recall

You set the learning rate high and the loss suddenly becomes 'NaN' (not a number). What almost certainly happened?

Hint: Think about the too-big-step picture.

Mini-batches — a fast, noisy estimate of the slope

To know the exact downhill direction, you'd measure the loss over every training example — millions of them — before taking a single step. That's far too slow.

A quick poll instead of a full census

Instead of surveying the entire dataset to find the slope, grab a small handful of examples and estimate the slope from just those. Each estimate is a little noisy (it wobbles depending on which examples you grabbed), but you get to take a step almost immediately — and hundreds of quick, slightly-wrong steps beat one perfect step.

That small handful is a , and training this way is called (Stochastic Gradient Descent). One full pass through all the mini-batches is an .

Method	Batch size	Trade-off
Batch GD	All data	Accurate slope, but slow and memory-heavy
SGD	1 sample	Very noisy, very fast, can escape shallow traps
Mini-batch	32–1024	The practical sweet spot everyone uses

The noise is a feature, not a bug

That wobble in mini-batch gradients actually helps. It jostles the optimizer out of shallow dead-ends and bad flat spots it might otherwise get stuck in, and it nudges the model toward solutions that generalize better to new data.

What is the main practical advantage of mini-batch SGD over full-batch gradient descent?

Momentum — build up speed, smooth out the wobble

Plain SGD has an annoying habit: in a long, narrow valley it zig-zags across the walls instead of running straight down the middle, wasting steps.

Think of it like a heavy ball rolling downhill:

A ball with weight doesn't jitter side to side — it builds up speed in the direction it's been consistently rolling and coasts right through small bumps and ripples. gives the optimizer that same heaviness: it remembers where it's been heading and keeps rolling that way.

The effect: it accelerates along directions the gradient keeps agreeing on, and cancels out the back-and-forth zig-zag directions. In practice this makes training noticeably faster and steadier.

Adam — a personalized step size for every knob

So far, one learning rate controls all the knobs. But some parameters need big adjustments and others need tiny ones. What if each knob got its own step size, tuned automatically?

Per-knob cruise control

watches each parameter's recent gradients and adapts: a knob whose gradient has been large and erratic gets smaller, more careful steps; a knob with small, steady gradients gets larger ones. You still set one overall learning rate, but Adam personalizes it for every parameter.

To do this, Adam tracks two running averages per parameter: the (direction, like momentum) and the (size and volatility).

Here's what it looks like in real PyTorch — the loop that trains most modern models:

Python · needs a GPU — run on Colab

import torch

opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=epochs)

for epoch in range(epochs):
    for x, y in loader:
        opt.zero_grad()                # clear old gradients
        loss = criterion(model(x), y)  # how wrong are we?
        loss.backward()                # compute the gradients
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()                     # turn the knobs
    sched.step()                       # anneal the learning rate

Common mistakes that bite everyone

Forgetting opt.zero_grad() — gradients accumulate across steps and training explodes.
A learning rate that's too high (loss → NaN) or too low (loss plateaus instantly).
Skipping a warmup + decay schedule for Transformers.
Applying weight decay to biases and LayerNorm parameters (usually you shouldn't).

In Adam, what role does the second moment estimate v_t play?

Learning-rate schedules — change the step size over time

A single fixed learning rate is rarely best: you often want bold steps early and gentle ones near the end. A does exactly that.

Explain it yourself

Explain gradient descent and the learning rate to a friend using the foggy-hillside picture — no formulas. What goes wrong if the steps are too big? If you can't explain the learning rate cleanly, that's the spot to reread.

Recap — the key ideas

A model is a machine with millions of knobs (parameters); training turns them to shrink the loss (how wrong it is).
Gradient descent = feel the slope (gradient) and step downhill, over and over.
The learning rate is the step size — the most important dial; too big overshoots, too small crawls.
Mini-batch SGD estimates the slope from a small handful of examples: fast, a little noisy, and the noise actually helps.
Momentum rolls like a heavy ball — building speed and smoothing the zig-zag.
Adam gives every parameter its own adaptive step size; AdamW + a warmup/cosine schedule is the modern default.

Practice — and how to make it stick

Learn it the way that actually works

• Retrieval practice: attempt the problems below before rereading — struggling to recall beats re-reading.
• Spaced repetition: mark this topic complete to add it to your Review queue, resurfacing right before you'd forget.
• Interleaving: mix these with Linear Algebra and Calculus problems rather than doing them in a block.

Implement plain SGD, SGD+momentum, and Adam from scratch and race them on a 2-D loss like the Rosenbrock function; plot the trajectories and see momentum cut the corners.
Sweep the learning rate across [1e-1, 1e-2, 1e-3, 1e-4] on a small CNN and plot the loss curves — find where it overshoots and where it crawls.
Add a cosine schedule with warmup and observe the effect on final accuracy.

Next: understand where these gradients actually come from in Backpropagation.