Backpropagation

Reverse-mode autodiff — how gradients flow through a network. Taught from zero, starting with a graph of five little arithmetic steps, then built up to the algorithm that assigns blame to every weight in a billion-parameter model.

intermediate#autograd#chain-rule

Start here — the question backprop answers

You already know from Optimization that training means turning knobs to make the loss smaller, and that to turn a knob you need to know which way and how much. That "which way and how much" number is the gradient.

Here's the problem nobody warns you about. A modern network has millions of knobs, buried dozens of layers deep, and the loss is one single number computed at the very end. So the real question is:

The final answer was wrong by this much. How much of that was each individual knob's fault?

That question has a name — — and is the algorithm that answers it. Efficiently. For every knob at once. It is the idea that made neural networks with hidden layers trainable in the first place, and the version everyone uses today was popularized in a four-page 1986 paper.

One number in, millions of numbers out

Backprop takes one number — the loss — and hands back one gradient per parameter, telling each weight exactly how the loss would change if that weight were nudged up a hair. The astonishing part is the price: getting all million answers costs roughly the same as computing the loss once. That single fact is why deep learning is possible at all.

Think of it like a factory tracing back a defective product:

A product comes off the end of a long assembly line, and it's 3 millimetres too wide. You walk backwards down the line asking each station: "given that your output was off by this much, how far off was your input, and how much did your own setting contribute?" Each station only needs to understand its own little job. Chain those local answers together, station by station, and by the time you reach the front of the line every worker knows their exact share of the blame — without anyone having to understand the whole factory.

How to read this page

It starts from first principles with five numbers you can check by hand. Flip the Depth switch at the top for the formal notation, matrix forms, and derivations — and they open automatically once you've finished the prerequisites (Neural Networks and Calculus). Nothing is hidden for good.

Backprop is not the optimizer

This trips up almost everyone. Backpropagation computes the gradients; it does not change a single weight. Gradient descent, SGD, or Adam take those gradients and do the actual updating. In PyTorch the split is literal: loss.backward() is backprop, opt.step() is the optimizer. Two different jobs, two different lines of code.

The computational graph — a network is just small steps wired together

Before we can push blame backwards, we need to be precise about what "the network" even is. And the answer is nicer than you'd expect: a network is a pile of tiny arithmetic operations wired together. Add. Multiply. Take a max. Apply a squashing function. Nothing in there is harder than middle-school arithmetic — there's just a lot of it.

Drawing those steps as boxes with wires between them gives a .

Think of it like a recipe written as a flowchart:

"Cream the butter and sugar" → "beat in the eggs" → "fold in the flour" → "bake." Each arrow hands the result of one step to the next. A network is the same flowchart, just with multiply-and-add instead of creaming and folding — and dozens of layers instead of four steps.

Let's take the smallest example that shows every idea we need. Three inputs, two operations:

d = a + b, \qquad e = d \times c

The forward pass, by hand

Set $a = 2$ , $b = 5$ , $c = -3$ . Now walk left to right, doing one step at a time and writing down what each step produced:

The add node. $d = a + b = 2 + 5 = 7$ .
The multiply node. $e = d \times c = 7 \times (-3) = -21$ .

That's the whole forward pass: $e = -21$ . Notice we wrote down the intermediate value $d = 7$ rather than throwing it away — in a moment you'll see that the backward pass is helpless without it.

That left-to-right sweep is the . Here is the same computation as a picture — the black labels are what the forward pass computes, and the orange labels are what the backward pass will compute in a moment. Don't worry about the orange numbers yet; just note that every wire ends up carrying two things: a value forward, and a gradient backward.

The computational graph for e = (a + b) x c— interactive, drag & zoom

Loading chart…

Forward pass, left to right (black): each node computes its value from the values wired into it. Backward pass, right to left (orange): each node receives a gradient and hands one back to each of its inputs. Same wires, opposite directions. Drag to pan, scroll to zoom.

Try to recall

Why does the forward pass bother storing the intermediate value d, instead of just returning the final answer?

Hint: Look at what the multiply node would need in order to answer questions about its own inputs.

Local gradients — every node only needs to know its own job

Now the key move, and honestly the whole idea of backpropagation.

No node in the graph knows anything about the network. The add node has never heard of the loss. It doesn't know it's inside a neural network, doesn't know what layer it's in, doesn't know what happens after it. It knows exactly one thing: if my inputs wiggle, how does my output wiggle? That's its .

Tiny, ignorant, and that's the point

Each node is a specialist with a one-line job description. "I add." "I multiply." Ask an adder how its output responds to its first input and it says "one-for-one" — and it can answer that without knowing a single thing about the other three million nodes. Global understanding never has to exist anywhere. It emerges from chaining local answers.

Think of it like a chain of currency booths:

Booth 1 changes dollars to euros at 0.9. Booth 2 changes euros to yen at 160. Neither clerk knows about the other, and neither needs to. Each knows only their own rate. Walk your money through both and the end-to-end rate — $0.9 \times 160 = 144$ — appears on its own, just by multiplying the local rates. Backprop is that walk, done backwards.

The local gradients of our two nodes

The add node, $d = a + b$ , with $a = 2$ , $b = 5$ :

Push $a$ up by $0.001$ : $d$ goes from $7$ to $7.001$ . It moved by exactly the same amount, so the local gradient is $1$ .
Same for $b$ : local gradient $1$ .
So an add node's local gradients are $1$ and $1$ — always, regardless of the input values.

The multiply node, $e = d \times c$ , with $d = 7$ , $c = -3$ :

Push $d$ up by $0.001$ : $e$ goes from $-21$ to $7.001 \times (-3) = -21.003$ . It moved $-0.003$ , which is $-3$ times the nudge. Local gradient: $-3$ — which is exactly $c$ , the other input.
Push $c$ up by $0.001$ : $e$ goes to $7 \times (-2.999) = -20.993$ , a move of $+0.007$ . Local gradient: $7$ — which is $d$ , again the other input.
So a multiply node's local gradients are the values it was multiplied by. It swaps them.

Three gates, three personalities (worth memorizing)

These three shapes cover most of what you'll ever hand-derive, and reading a backward pass gets much faster once you know them by feel:
• Add is a distributor. It copies the incoming gradient, unchanged, to every input. Local gradients are all $1$ .
• Multiply is a swapper. Each input's gradient is the incoming gradient times the other input. A tiny input therefore hands its partner a tiny gradient — which is precisely why badly scaled inputs stall training.
• Max is a router. It sends the whole gradient to whichever input won, and exactly zero to the losers. This is why a dead ReLU stays dead: it lost, so it receives nothing, so it never gets a chance to change.

The

Activation Functions— interactive, try itOpen in lab →

f(x) — GELUf′(x) — derivative

Switch between activations and watch the derivative curve underneath — that curve IS the local gradient this node will hand backward. Notice how sigmoid and tanh flatten to nearly zero at the edges: a node sitting out there multiplies the incoming gradient by almost nothing and effectively cuts the wire. ReLU's derivative is a clean 1 or a hard 0 — pass or block.

nodes in a real network work the same way, just with a curvier local gradient.

Try to recall

A multiply node computes y = u x v with u = 0.001 and v = 500. Which input will receive a large gradient, and which a tiny one?

Hint: A multiply node's local gradient for one input is the value of the other input.

The backward pass — start at 1 and multiply your way home

We have local gradients everywhere. Now we chain them. The rule is short enough to fit on a sticky note:

Each node takes the gradient handed to it from above, multiplies by its own local gradient, and hands the result down to each of its inputs.

Where does the very first gradient come from? From the output itself. How sensitive is $e$ to $e$ ? Change $e$ by one and $e$ changes by one, so we seed the whole process with $\partial e/\partial e = 1$ .

The backward pass, by hand

Working right to left through our graph, with the local gradients we just computed:

Seed the output. $\dfrac{\partial e}{\partial e} = 1$ .
Through the multiply node. Incoming gradient is $1$ $1$ .
- To $d$ : $1 \times (\text{local } c) = 1 \times (-3) = \mathbf{-3}$ .
- To $c$ : $1 \times (\text{local } d) = 1 \times 7 = \mathbf{7}$ .
Through the add node. Incoming gradient is the $-3$ $- 3$ that just arrived at $d$ $d$ .
- To $a$ : $-3 \times 1 = \mathbf{-3}$ .
- To $b$ : $-3 \times 1 = \mathbf{-3}$ .

Done. Every input now has its gradient: $\partial e/\partial a = -3$ , $\partial e/\partial b = -3$ , $\partial e/\partial c = 7$ . Those are the orange numbers in the graph above.

Check one by brute force. Nudge $a$ from $2$ to $2.001$ . Then $d = 7.001$ and $e = 7.001 \times (-3) = -21.003$ . The output fell by $0.003$ for a nudge of $0.001$ — a rate of $-3$ . The backward pass was right, and it got all three answers in a single sweep, while brute force needs one full re-run of the network per input.

Read the sign, then the size

$\partial e/\partial a = -3$ says two things. The minus says increasing $a$ decreases the output — so if you wanted the output up, you'd move $a$ down. The 3 says the effect is three times as strong as the nudge. In training, the loss plays the role of $e$ : sign tells you which way to turn the knob, size tells you how much that knob matters.

Run it and see the two sweeps side by side. Change $a$ , $b$ , or $c$ and watch how the gradients move — the multiply node's gradients depend on the values, so they change every time:

Python · runs in your browser

What this does: Runs the forward pass through e = (a+b)*c storing each intermediate value, then runs the backward pass right to left, multiplying each incoming gradient by that node's local gradient. It finishes by re-deriving each gradient with brute-force numerical nudging, so you can see the two agree to several decimal places.

a, b, c = 2.0, 5.0, -3.0

# ---------- forward pass: compute and REMEMBER ----------
d = a + b            # the add node
e = d * c            # the multiply node
print(f"forward:  a={a}  b={b}  c={c}  ->  d={d}  ->  e={e}")

# ---------- backward pass: seed with 1, then multiply local gradients ----------
grad_e = 1.0                 # d(e)/d(e) — the output's sensitivity to itself

# multiply node: swaps its inputs
grad_d = grad_e * c          # local gradient w.r.t. d is the OTHER input, c
grad_c = grad_e * d          # local gradient w.r.t. c is the OTHER input, d

# add node: distributes unchanged
grad_a = grad_d * 1.0
grad_b = grad_d * 1.0

print(f"backward: de/da={grad_a}  de/db={grad_b}  de/dc={grad_c}")

# ---------- brute-force check: nudge each input and measure ----------
def f(a, b, c):
    return (a + b) * c

h = 1e-6
print("\nmeasured by nudging each input:")
print(f"  de/da ~ {(f(a+h, b, c) - e) / h:.4f}   (backprop said {grad_a})")
print(f"  de/db ~ {(f(a, b+h, c) - e) / h:.4f}   (backprop said {grad_b})")
print(f"  de/dc ~ {(f(a, b, c+h) - e) / h:.4f}   (backprop said {grad_c})")

In a computational graph, a node's gradient with respect to one of its inputs equals:

A real neuron, end to end

Our toy graph had no weights. Let's do the smallest thing that actually learns: one neuron with a weight, a bias, a squashing function, and a loss.

z = wx + b, \qquad \hat{y} = \sigma(z), \qquad L = (\hat{y} - y)^2

In words, left to right: multiply the input by the weight and add the bias; squash the result into the range $0$ to $1$ with the ; then score how far the prediction landed from the truth by squaring the miss.

One neuron, forward then backward

Take $w = 0.5$ , $x = 2$ , $b = 1$ , and a target of $y = 0$ .

Forward:

$z = 0.5 \times 2 + 1 = 2.0$
$\hat{y} = \sigma(2.0) \approx 0.8808$
$L = (0.8808 - 0)^2 \approx 0.7758$

Badly wrong — we wanted $0$ and predicted $0.88$ .

Backward, seeding with $\partial L/\partial L = 1$ and walking back through each node:

Through the squared loss. Local gradient of $(\hat{y}-y)^2$ with respect to $\hat{y}$ is $2(\hat{y}-y) = 2 \times 0.8808 \approx 1.7616$ . So $\partial L/\partial \hat{y} \approx 1.7616$ .
Through the sigmoid. Its local gradient is the tidy $\hat{y}(1-\hat{y}) = 0.8808 \times 0.1192 \approx 0.1050$ . Multiply: $\partial L/\partial z \approx 1.7616 \times 0.1050 \approx 0.1850$ .
Through the multiply-and-add. The bias is added, so it distributes: $\partial L/\partial b \approx 0.1850$ . The weight is multiplied by $x$ , so it swaps: $\partial L/\partial w \approx 0.1850 \times 2 = 0.3699$ .

Both gradients are positive, meaning raising $w$ or $b$ would raise the loss — so gradient descent will lower both, pushing the prediction down toward the target of $0$ . Exactly right.

Spot the villain: 0.1050

Look at step 2. The sigmoid multiplied the gradient by roughly one tenth on its way through — and $z = 2$ isn't even far out on the curve. At $z = 6$ that factor drops below $0.0025$ . Stack ten such layers and the gradient reaching the first one has been multiplied by something like $10^{-10}$ : the early layers receive essentially nothing and stop learning. That is the vanishing-gradient problem, and it is the single biggest reason ReLU replaced sigmoid in hidden layers.

Run the neuron and watch each stage's gradient appear. Then try the experiment that makes the point stick: change b to 6.0. The forward pass barely notices — the prediction just saturates from $0.88$ up to $0.999$ , which is more wrong, not less. But every gradient collapses by roughly a hundredfold. The neuron is now maximally wrong and almost completely unable to learn.

What to look for

dL/dz is the gradient after passing through the sigmoid. Compare it to dL/dyhat, the gradient before. The ratio between them is the sigmoid's local gradient — and it is never greater than $0.25$ , no matter what.

Python · runs in your browser

What this does: Runs a single sigmoid neuron forward on w=0.5, x=2, b=1 with target 0, then walks the gradient back through the loss, the sigmoid, and the linear step, printing the gradient at every stage. Change b to 6.0 and rerun: the prediction gets WORSE (0.88 to 0.999) while every gradient shrinks about a hundredfold — that is saturation, and it is what stops deep sigmoid networks from training.

import math

w, x, b, y = 0.5, 2.0, 1.0, 0.0     # try b = 6.0 to see saturation

# ---------- forward ----------
z = w * x + b
yhat = 1.0 / (1.0 + math.exp(-z))    # sigmoid
loss = (yhat - y) ** 2
print(f"forward:  z={z:.4f}  yhat={yhat:.4f}  loss={loss:.4f}")

# ---------- backward ----------
dL_dyhat = 2.0 * (yhat - y)          # local gradient of the squared loss
dyhat_dz = yhat * (1.0 - yhat)       # local gradient of the sigmoid
dL_dz    = dL_dyhat * dyhat_dz       # chain them
dL_dw    = dL_dz * x                 # multiply node swaps: w's partner is x
dL_db    = dL_dz * 1.0               # add node distributes unchanged

print(f"backward: dL/dyhat={dL_dyhat:.4f}")
print(f"          dyhat/dz={dyhat_dz:.4f}   <- the sigmoid's shrink factor")
print(f"          dL/dz={dL_dz:.4f}")
print(f"          dL/dw={dL_dw:.4f}   dL/db={dL_db:.4f}")

# ---------- one gradient-descent step, to prove the signs are useful ----------
lr = 1.0
w_new, b_new = w - lr * dL_dw, b - lr * dL_db
z_new = w_new * x + b_new
loss_new = (1.0 / (1.0 + math.exp(-z_new)) - y) ** 2
print(f"\nafter one step: loss {loss:.4f} -> {loss_new:.4f}   (change: {loss_new - loss:+.6f})")
print("the gradients pointed the right way — but notice HOW MUCH progress they bought")

Try to recall

The sigmoid's local gradient is yhat(1 - yhat). What is the largest value that can ever take, and where?

Hint: Try a few values of yhat between 0 and 1.

Passes needed to get every gradient: forward mode vs reverse mode— interactive, drag & zoom

Loading chart…

Forward-mode autodiff needs one sweep per parameter, so its cost tracks model size. Reverse mode (backpropagation) needs one sweep per output — and a loss is a single number, so it is always exactly one, at any scale. Note the log axis: the gap at a billion parameters is a factor of a billion.

Why is reverse-mode (backprop) the right choice for training neural networks, rather than forward mode?

Vanishing and exploding gradients — the price of multiplying

The chain rule multiplies one factor per layer. That is elegant, and it is also a trap, because repeated multiplication is unstable. Multiply many numbers slightly below $1$ and the product races to zero; multiply many slightly above $1$ and it races to infinity. There is no gentle middle.

Think of it like compound interest, running in reverse:

A 5% annual gain over 30 years multiplies your money by 4.3. A 5% annual loss over 30 years leaves you with 21% of it. Same modest per-year factor, wildly different 30-year outcome — because the effect compounds. Gradient flow through 30 layers compounds in exactly the same way, except the "years" are layers and the compounding happens on every single training step.

Gradient magnitude reaching layer 1, as depth grows— interactive, drag & zoom

Loading chart…

Each curve assumes every layer multiplies the gradient by a constant factor. At 0.6 per layer the gradient vanishes into numerical noise by layer 25; at 1.5 it explodes by four orders of magnitude; only a factor near 1.0 survives the trip. Keeping that per-layer factor near 1 is the entire job of careful initialization, normalization layers, and residual connections. Log scale on the vertical axis.

Everything in the modern training toolkit is about this one plot

Ask why any of these exist and the answer is "to hold that per-layer factor near 1.0":
• ReLU instead of sigmoid — local gradient of exactly $1$ where active, instead of at most $0.25$ .
• Careful initialization (Xavier, He) — weight scales chosen so each layer's factor starts near $1$ .
• Normalization layers — re-centre activations each layer so nodes don't drift into saturated regions.
• Residual connections — add a shortcut path whose local gradient is exactly $1$ , giving the gradient a clean route home no matter how deep the network is.
• Gradient clipping — a blunt safety net: if the gradient explodes anyway, cap its size before the optimizer acts on it.

Autograd — what a framework does so you don't have to

You have now hand-derived what PyTorch does automatically. Worth being precise about what it automates:

It records the graph as you run. Every operation on a tensor with requires_grad=True appends a node to a tape, capturing the operation and the values it needs for its local gradient. That's the forward pass.
loss.backward() walks the tape in reverse. Each recorded node applies exactly the rule from the sticky note — upstream gradient times local gradient — accumulating results into each tensor's .grad.
opt.step() is a separate matter entirely. Backprop filled in .grad; the optimizer decides what to do with it.

is neither guessing nor doing algebra on your formula. It is exactly the bookkeeping you just did by hand, done at scale and without arithmetic slips.

Python · needs a GPU — run on Colab

import torch

# Exactly our toy graph, with PyTorch tracking the gradients
a = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(5.0, requires_grad=True)
c = torch.tensor(-3.0, requires_grad=True)

d = a + b            # the add node
e = d * c            # the multiply node
e.backward()         # seeds de/de = 1 and walks the tape backwards

print(a.grad, b.grad, c.grad)   # tensor(-3.) tensor(-3.) tensor(7.)

# The same thing inside a real training loop
for x, y in loader:
    opt.zero_grad()              # clear .grad — it ACCUMULATES, remember
    loss = criterion(model(x), y)
    loss.backward()              # <- backpropagation: fills every .grad
    opt.step()                   # <- the optimizer: uses them

The five autograd bugs you will actually hit

Forgetting opt.zero_grad(). Gradients accumulate by design (that's the sum-over-paths rule). Skip the reset and you're stepping on the sum of every batch so far.
Calling .backward() twice without retain_graph=True. The tape is freed after the first pass — the error message is confusing, the cause is not.
Accumulating loss for logging with total += loss instead of loss.item(). That keeps the whole graph alive for every batch and quietly eats all your memory.
In-place ops (x += 1, relu_()) on a tensor the backward pass still needs, which corrupts a stored local gradient. PyTorch usually catches this and complains.
Silently broken gradient flow — a .detach(), a torch.no_grad(), or a non-differentiable step in the middle. The classic tell is .grad that is None or all zeros for a parameter you expected to be learning.

Try to recall

Your model trains, the loss goes down, but one particular layer's weights never change at all. What would you check first?

Hint: Think about what has to reach a weight before the optimizer can move it.

Now build a whole network with no framework at all — forward pass, backward pass, and training loop written by hand. If you can read this cell, you understand backpropagation:

Python · runs in your browser

What this does: Trains a 2-layer neural network on the XOR problem using nothing but NumPy — no PyTorch, no autograd. The backward pass is written out by hand using exactly the rules from this lesson: the loss node, the sigmoid's local gradient, and the matrix forms for a linear layer. It prints the loss falling and the final predictions landing near the correct 0/1 targets, then plots the loss curve.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)

# XOR: not solvable by a single linear layer — it needs a hidden layer
X = np.array([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
Y = np.array([[0.], [1.], [1.], [0.]])

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

# --- parameters (small random init, so no unit starts saturated) ---
W1 = rng.normal(0, 1.0, (2, 8)); b1 = np.zeros((1, 8))
W2 = rng.normal(0, 1.0, (8, 1)); b2 = np.zeros((1, 1))

lr, losses = 0.5, []
for step in range(4000):
    # ---------- forward: compute and remember every intermediate ----------
    z1 = X @ W1 + b1
    a1 = sigmoid(z1)
    z2 = a1 @ W2 + b2
    a2 = sigmoid(z2)
    loss = np.mean((a2 - Y) ** 2)
    losses.append(loss)

# ---------- backward: upstream gradient x local gradient, layer by layer ----------
    dL_da2 = 2 * (a2 - Y) / len(X)      # local gradient of the mean squared error
    dL_dz2 = dL_da2 * a2 * (1 - a2)     # through the sigmoid: local gradient a(1-a)
    dL_dW2 = a1.T @ dL_dz2              # matrix form of "upstream x the other input"
    dL_db2 = dL_dz2.sum(axis=0, keepdims=True)   # add node: distributes, so just sum

dL_da1 = dL_dz2 @ W2.T              # W transposed sends blame back a layer
    dL_dz1 = dL_da1 * a1 * (1 - a1)     # through the first sigmoid
    dL_dW1 = X.T @ dL_dz1
    dL_db1 = dL_dz1.sum(axis=0, keepdims=True)

# ---------- the optimizer's job, not backprop's: step downhill ----------
    W2 -= lr * dL_dW2; b2 -= lr * dL_db2
    W1 -= lr * dL_dW1; b1 -= lr * dL_db1

if step % 1000 == 0:
        print(f"step {step:5d}   loss {loss:.4f}")

print(f"step  4000   loss {losses[-1]:.4f}")
print("\ntargets:    ", Y.ravel())
print("predictions:", np.round(a2.ravel(), 3))

plt.figure(figsize=(5, 3))
plt.plot(losses, color="#d97757")
plt.xlabel("training step"); plt.ylabel("loss"); plt.title("Hand-written backprop on XOR")
plt.tight_layout()

Gradient checking — how to know your backward pass is right

Any hand-derived gradient can be verified in three lines: nudge one parameter by a tiny $h$ (around $10^{-5}$ ), re-run the forward pass, and compare $(L(w+h) - L(w-h)) / 2h$ against what your backward pass claimed. If they agree to several decimals, your derivation is correct. This is slow — one forward pass per parameter — so it is a debugging tool, never a training method. But when a from-scratch model won't learn, it is the fastest way to find out whether the bug is in your gradients or somewhere else entirely.

You write a custom layer's backward pass by hand, and gradient checking shows your analytic gradient is exactly half the numerical one for a particular input. What is the most likely cause?

Explain it yourself

Explain backpropagation to a friend using the factory-assembly-line picture and no symbols. Cover three things: what a local gradient is, where the very first gradient comes from, and why the answers get multiplied together rather than added. If you stall on any one of those, that is exactly the section to reread.

Recap — the key ideas

Backprop answers one question: how much of the final error is each parameter's fault? It computes gradients; the optimizer is what actually changes the weights.
A network is a computational graph of tiny operations. The forward pass computes values left to right and stores the intermediates — which is why training costs so much memory.
Every node knows only its local gradient: how its own output responds to its own input. No node knows about the network.
The backward pass seeds with $1$ at the loss and applies one rule at every node: upstream gradient x local gradient, handed down to each input.
Add nodes distribute gradient, multiply nodes swap their inputs, max nodes route it all to the winner. When a value forks forward, its gradients add backward — hence += and opt.zero_grad().
Backprop is reverse-mode autodiff: cost scales with outputs, and a loss is one number — so all million gradients cost about one extra forward pass.
Multiplying one factor per layer makes deep gradients vanish or explode. ReLU, good initialization, normalization, and residual connections all exist to keep that per-layer factor near $1$ .

Practice — and how to make it stick

Learn it the way that actually works

• Retrieval practice: before scrolling back up, try to state the backward rule for a multiply node from memory, then check. Pulling it out of your head beats rereading it, even when the recall feels uncomfortable.
• Spaced repetition: mark this topic complete to add it to your Review queue, where it resurfaces right before you would have forgotten it.
• Interleaving: mix these exercises with Calculus chain-rule problems and Optimization update-rule problems rather than grinding one type. Messier practice, sturdier memory.

By hand, no code: for $f(x, y) = (x + y) \times \max(x, y)$ at $x = 3$ , $y = 1$ , draw the graph and compute $\partial f/\partial x$ and $\partial f/\partial y$ . Watch carefully what the max node does with the gradient — and note that $x$ forks into two paths, so its two contributions must be summed.
Gradient checking: take the XOR network above and verify one entry of dL_dW1 numerically with the two-sided formula. Do the numbers match to five decimals?
Break it on purpose: in the XOR cell, change the initialization to rng.normal(0, 20.0, ...). Training should stall completely. Explain why using the sigmoid's local gradient — then fix it by switching the hidden activation to ReLU (np.maximum(0, z), with local gradient (z > 0)).
Build micrograd: write a tiny Value class that overloads + and *, records its children, and implements .backward() by walking the graph in reverse topological order. It is about 100 lines, and it is genuinely the same algorithm PyTorch runs. Karpathy's Zero to Hero walks through it step by step.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and your terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Finish the backward pass for f(x, y) = (x + y) * max(x, y). The forward pass and the numerical check are already written. Fill in the four TODO lines using the rules from this lesson — add distributes, multiply swaps, max routes everything to the winner and nothing to the loser — and remember that x feeds BOTH nodes, so its two gradient contributions must be added together. Your printed gradients should match the measured ones. Bonus: swap to x = 1.0, y = 3.0 and confirm the max node now sends its gradient the other way.

editor

x, y = 3.0, 1.0

# ---------- forward pass ----------
s = x + y                 # the add node
m = max(x, y)             # the max node
f = s * m                 # the multiply node
print(f"forward:  s={s}  m={m}  f={f}")

# ---------- local gradients ----------
# multiply node f = s * m  ->  swaps its inputs
df_ds = m
df_dm = s

# max node: 1 to the winner, 0 to the loser
dm_dx = 1.0 if x >= y else 0.0
dm_dy = 1.0 if y > x else 0.0

# ---------- backward pass ----------
grad_f = 1.0
grad_s = 0.0   # TODO: upstream (grad_f) times the multiply node's local gradient for s
grad_m = 0.0   # TODO: upstream (grad_f) times the multiply node's local gradient for m

# x reaches f through TWO paths: via the add node AND via the max node -> add them
grad_x = 0.0   # TODO: (grad_s * 1.0) + (grad_m * dm_dx)
grad_y = 0.0   # TODO: (grad_s * 1.0) + (grad_m * dm_dy)

print(f"backward: df/dx={grad_x}   df/dy={grad_y}")

# ---------- numerical check: these are the answers to match ----------
def F(x, y):
    return (x + y) * max(x, y)

h = 1e-6
print(f"measured: df/dx~{(F(x+h, y) - F(x, y)) / h:.4f}   df/dy~{(F(x, y+h) - F(x, y)) / h:.4f}")

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Next: stop writing backward passes by hand and let a framework do it, in PyTorch — then see what the gradients get used for in Regularization and Normalization and Initialization.