Calculus

Derivatives, chain rule, and gradients — the language of learning. Taught from zero, starting with what a slope is, and built up to the exact quantity every neural network follows downhill.

beginner#math#derivatives#gradients

Start here — what this is really about

Calculus has a fearsome reputation, and almost all of it is undeserved. The part of calculus that machine learning actually runs on is one single question, asked over and over:

If I nudge this number a tiny bit, how much does that number change?

That's it. That's the whole subject, for our purposes. A model has millions of knobs; training needs to know, for each knob, which way and how hard to turn it. "How much does the output change when I nudge this input" is exactly the question calculus answers, and the answer has a name: the derivative.

How to read this page

The page adapts to you. By default it teaches from first principles — no symbols until you already understand the idea they stand for. When you're ready, flip the Depth switch at the top to reveal the formal notation, derivations, and edge cases. Nothing is hidden for good; the deeper material sits behind the "Go deeper" panels so you can open it the moment you're curious.

What you can safely skip

Classical calculus courses spend half their time on integration — adding up infinitely many tiny pieces to get areas and volumes. It's beautiful, and it matters a lot in probability. But for understanding how models learn, you need the other half: differentiation. This page is about differentiation, and it goes deep on it. You'll meet integrals again in Probability & Statistics.

Step 1 — Slope, the ancestor of everything here

Before derivatives, there's a much older and simpler idea: the slope of a straight line. If you understand "rise over run," you already have the seed of all of calculus.

A answers: for every one step I take to the right, how far up (or down) do I go?

Think of it like a wheelchair ramp:

A ramp that climbs 1 metre over 12 metres of floor has a slope of $1/12$ — gentle. A ramp that climbs 1 metre over 2 metres of floor has a slope of $1/2$ — steep enough to be dangerous. Same idea, one number, and you instantly know how hard the climb is.

Slope between two points, by hand

Take two points on a line: $(2, 5)$ and $(6, 13)$ . (Each pair is x across, y up.)

Run — how far right did we move? $6 - 2 = 4$ .
Rise — how far up did we move? $13 - 5 = 8$ .
Slope — rise divided by run: $8 / 4 = 2$ .

So the line climbs 2 units up for every 1 unit right. Notice we needed two points to get this number: slope is a comparison between two places.

Try to recall

A line passes through (0, 10) and (5, 0). Is its slope positive or negative, and what is it?

Hint: Rise over run — and watch the sign of the rise.

Step 2 — The derivative: slope at a single point

Straight lines are easy: the slope is the same everywhere. Real functions curve. On a curve, the steepness changes as you move along it — a hill is steep near the bottom and flattens at the top.

So here's the problem: slope needs two points, but we want the steepness at one exact spot. How do you get "rise over run" when the run is zero?

Zoom in until the curve looks straight

Pick a point on a curve and zoom in. Zoom again. Zoom again. Almost every curve you'll meet becomes indistinguishable from a straight line if you zoom in far enough — the same way the Earth's surface looks flat from where you're standing even though the planet is round. The derivative is the slope of that straight line you see when you've zoomed all the way in.

The practical version: take a second point very close by, compute the ordinary rise-over-run between them, and then keep sliding that second point closer and closer. The slope value settles down on a specific number. That number is the .

Think of it like a speedometer versus a road trip average:

Drive 120 km in 2 hours and your average speed is 60 km/h — that's rise over run, computed between two points in time. But your speedometer doesn't show the trip average; it shows how fast you're going right now, at this instant. The average speed is the slope between two points; the speedometer reading is the derivative. Calculus is the machinery that turns the first into the second.

Watching the slope settle down

Let's find how steep $f(x) = x^2$ is exactly at $x = 1$ . We can't use one point, so we'll use a second point a distance $h$ away and shrink $h$ .

The slope between $x = 1$ and $x = 1 + h$ is $\dfrac{f(1+h) - f(1)}{h}$ .

$h$ (gap)	$f(1+h)$	slope $= \frac{f(1+h)-1}{h}$
1	$2^2 = 4$	$(4-1)/1 = 3$
0.5	$1.5^2 = 2.25$	$(2.25-1)/0.5 = 2.5$
0.1	$1.1^2 = 1.21$	$(1.21-1)/0.1 = 2.1$
0.01	$1.01^2 = 1.0201$	$(1.0201-1)/0.01 = 2.01$
0.001	—	$2.001$

The numbers are marching toward 2 and never wandering off. So the derivative of $x^2$ at $x = 1$ is exactly $2$ : right at that spot, the curve climbs 2 units for every 1 unit right.

Here is that table as a picture. Every dot is one row above — the gap $h$ on the horizontal axis, the measured slope on the vertical. Notice how the dots flatten onto the dashed line at 2 as $h$ shrinks toward the left:

Shrinking the gap: secant slopes converging on the derivative— interactive, drag & zoom

Loading chart…

Each dot is the ordinary rise-over-run slope of f(x) = x² measured between x = 1 and x = 1 + h. As the gap h shrinks (moving left), the measured slope settles onto exactly 2 — the derivative. This settling-down is the entire trick of calculus. Drag and zoom to explore.

Now do it with your hands. Drag the slider that controls the gap and watch the secant line (the one through two points) swing around until it becomes the tangent line (the one touching at a single point). The tangent's slope is the derivative:

The definition of the derivative — secant becoming tangent— GeoGebra, drag & exploreOpen on GeoGebra →

Loading interactive visualization…

Drag the h slider toward zero and watch the green secant line through two points rotate into the tangent line touching at one point. The equation of the secant updates live, so you can watch its slope converge on the derivative — exactly the table you just worked through, in motion.

Try to recall

Why can we not just plug a gap of exactly zero into rise-over-run to get the slope at a point?

Hint: Look at what the run would be, and what you would be dividing by.

Let's confirm the worked example numerically — shrink $h$ in code and watch the answer arrive:

Python · runs in your browser

What this does: Estimates the derivative of f(x)=x² at x=1 by measuring rise-over-run for smaller and smaller gaps h, printing each estimate next to its error. Watch the estimates march toward 2 — then watch the last few get WORSE, because dividing two nearly-equal numbers by a tiny number amplifies floating-point rounding error. That is why real frameworks never estimate derivatives this way.

Why we never compute derivatives this way in practice

That last row is the lesson. As $h$ shrinks past about $10^{-8}$ , f(x+h) and f(x) become so close that subtracting them throws away most of their significant digits, and then dividing by a tiny $h$ magnifies whatever rounding noise is left. Numerical differencing is a debugging tool (a gradient check), never the real thing. Deep-learning frameworks instead apply the rules below symbolically — which is exact, and which we'll see is also far cheaper.

Step 3 — The rules: derivatives without limits

Nobody computes limits by hand every time. Mathematicians worked out the limit once for each basic function shape and wrote down the answers. In practice, differentiating is a lookup-plus-recipe job.

The three you'll use constantly:

Rule	If your function is…	…its derivative is	Sanity check
Constant	$f(x) = c$	$0$	A flat line has no slope
Power	$f(x) = x^n$	$n\,x^{\,n-1}$	$x^2 \to 2x$ , matching our table at $x=1$
Sum	$f(x) = g(x) + k(x)$	$g'(x) + k'(x)$	Slopes just add

Why the power rule gives 2x for x squared

Picture $x^2$ as the area of a square with side $x$ . Grow the side by a hair $h$ and you glue on two thin strips — one along the top, one along the side — each of area $x \cdot h$ , plus a negligible little corner square of area $h^2$ . So the area grew by about $2xh$ , meaning it grew at a rate of $2x$ per unit of side length. That's the derivative, and the tiny corner you ignored is exactly what vanishes in the limit.

Differentiating a small polynomial, term by term

Find the derivative of $f(x) = 3x^2 + 5x - 7$ .

$3x^2$ — the power rule gives $2 \cdot x^1 = 2x$ for $x^2$ ; the constant $3$ rides along untouched, giving $6x$ .
$5x$ — this is $5x^1$ , so the power rule gives $1 \cdot x^0 = 1$ , times the $5$ : just $5$ . (Sensible: a term that's a straight line has a constant slope of 5.)
$-7$ — a constant. Flat. Contributes $0$ .
Add them up (the sum rule): $f'(x) = 6x + 5$ .

Check it at a point: at $x = 2$ , $f'(2) = 17$ . Nudge the input from $2$ to $2.001$ and $f$ goes from $15$ to $15.017003$ — a rise of about $0.017$ over a run of $0.001$ , which is $17.003$ . The derivative predicted the change.

What is the derivative of f(x) = 4x³ − 2x + 9?

What the derivative tells you about a curve

The sign of the derivative is a compass, and the zeros of the derivative are where the interesting places are:

$f'(x) > 0$ → the function is rising here. Step right and the value grows.
$f'(x) < 0$ → the function is falling here. Step right and the value shrinks.
$f'(x) = 0$ → the function is momentarily flat — a peak, a valley, or a plateau. This is where training stops.

See both curves at once. The orange curve is $f(x) = x^3 - 3x$ ; the teal one is its derivative $f'(x) = 3x^2 - 3$ . Look at where the teal curve crosses zero and check what the orange curve is doing at exactly those spots:

A function and its derivative, stacked— interactive, drag & zoom

Loading chart…

Orange is f(x) = x³ − 3x; teal is its derivative f'(x) = 3x² − 3. Where teal crosses zero (at x = −1 and x = +1), orange is perfectly flat — a hilltop and a valley floor. Where teal sits below zero, orange is falling; where teal is above zero, orange is rising. A trained model sits where its gradient has hit zero.

Try to recall

During training, the loss stops improving and the gradient is measured at almost exactly zero. Name two different situations that could produce this.

Hint: A derivative of zero means flat — but flat where?

Step 4 — The chain rule: the engine of deep learning

If you remember one thing from this page, make it this one. A neural network is functions stacked on functions stacked on functions: the input goes through layer 1, whose output goes through layer 2, and so on for dozens of layers. To train it, we need to know how a change deep inside affects the final loss at the very end.

The rule that answers that is the , and backpropagation is nothing but the chain rule applied carefully and efficiently.

Rates multiply along a chain

Suppose a bicycle's pedals turn a middle gear 3 times for every 1 pedal turn, and that middle gear turns the wheel 2 times for every 1 of its own turns. How many wheel turns per pedal turn? You multiply: $3 \times 2 = 6$ . Rates chain together by multiplication, not addition. That's the entire chain rule.

Think of it like a currency conversion:

Dollars → euros at 0.9, euros → yen at 160. Dollars → yen is $0.9 \times 160 = 144$ . Each conversion is a rate, and you compose them by multiplying. A neural network is a long chain of such conversions, and the chain rule tells you the end-to-end exchange rate between any early parameter and the final loss.

The chain rule, by hand, checked numerically

Let $g(x) = 2x + 1$ (the inner function) and $f(u) = u^2$ (the outer one). The composed function is $h(x) = f(g(x)) = (2x + 1)^2$ . Find $h'(x)$ at $x = 3$ .

Rate of the inner stage. $g'(x) = 2$ — nudge $x$ by a hair and $g$ moves twice as much.
Rate of the outer stage, at the value the inner one produced. At $x = 3$ , $g(3) = 7$ . The outer function $f(u) = u^2$ has $f'(u) = 2u$ , so at $u = 7$ that rate is $14$ . This is the step everyone forgets: the outer rate must be evaluated at the inner function's output, not at $x$ .
Multiply. $h'(3) = 14 \times 2 = 28$ .

Check it. $h(3) = 7^2 = 49$ . Nudge to $x = 3.001$ : $h(3.001) = 7.002^2 = 49.028004$ . Rise $\approx 0.028$ over run $0.001$ gives $\approx 28.004$ . The chain rule was right.

Try to recall

Why must the outer function's derivative be evaluated at g(x), rather than at x itself?

Hint: What number is actually being fed into the outer function?

Now build a two-stage chain in code and propagate a gradient backward through it by hand — this is backpropagation in miniature:

Python · runs in your browser

What this does: Builds the chain h(x) = (2x+1)² as two separate stages, computes each stage's local rate, then multiplies them backward to get the end-to-end derivative — this is backpropagation in miniature. It finally checks the answer against a brute-force numerical nudge, so you can see the chain rule and the actual measured change agree.

A network computes loss = f(g(k(x))). Which product gives dloss/dx?

Step 5 — Partial derivatives and the gradient

Everything so far had one input knob. A real model has millions. What does "the slope" even mean when there are a million directions you could move in?

Wiggle one knob at a time

Freeze every knob but one. Wiggle that single knob and see how the output responds. That one number is the partial derivative with respect to that knob. Do it for each knob in turn, collect all the answers into a list, and you have the gradient — a full report card of "how sensitive is the output to each individual input."

A is written with a curly $\partial$ instead of $d$ , purely as a reminder that other variables exist and are being held still.

Think of it like standing on a hillside at a crossroads:

You're on a hillside where a road runs exactly east–west and another exactly north–south. Face east and ask how steeply the ground rises — that's the partial derivative in $x$ . Turn to face north and ask the same — that's the partial derivative in $y$ . Two separate questions, two separate numbers, and neither one is "the" steepness of the hill; together, though, they tell you everything about which way is up.

Both partial derivatives of a two-input function

Let $f(x, y) = x^2 + 3y^2$ . Find both partials at the point $(1, 2)$ .

With respect to $x$ — treat $y$ as a frozen constant. Then $3y^2$ is just some fixed number, so its derivative is $0$ , and $x^2$ differentiates to $2x$ . So $\partial f/\partial x = 2x$ , which at $x = 1$ equals $\mathbf{2}$ .
With respect to $y$ — now freeze $x$ . The $x^2$ term becomes a constant contributing $0$ , and $3y^2$ differentiates to $6y$ . So $\partial f/\partial y = 6y$ , which at $y = 2$ equals $\mathbf{12}$ .
Collect them. The gradient at $(1, 2)$ is the list $[2, 12]$ .

Read the answer. The output is six times more sensitive to $y$ than to $x$ right here. If you could only adjust one knob, $y$ is the one worth touching. That relative sensitivity is precisely the information gradient descent acts on.

Drag the point around the surface below and watch the two tangent slices — one cutting in the $x$ direction, one in the $y$ direction. Each slice is an ordinary single-variable curve, and its slope is one of the partial derivatives:

Partial derivatives on a 3D surface— GeoGebra, drag & exploreOpen on GeoGebra →

Loading interactive visualization…

Move the sliders to reposition the point on the surface, then look at the two tangent lines. Each one is the slope of the curve you get by slicing the surface along a single axis — the partial derivative in that direction. Rotate the view by dragging to see how the two slices together pin down the tangent plane.

Here is the surface most relevant to machine learning: a loss bowl, $f(x, y) = x^2 + 2y^2$ . Rotate it and notice it's steeper along $y$ than along $x$ — a stretched, elliptical valley rather than a round one. That asymmetry is what makes plain gradient descent zig-zag, and what momentum and Adam exist to fix:

A loss surface — f(x, y) = x² + 2y²— interactive, drag & zoom

Loading chart…

Drag to rotate, scroll to zoom. Height is the loss; the two floor directions are two parameters. The bowl is squashed — twice as steep along y as along x — so the gradient at most points does not aim straight at the bottom. Training walks downhill on a surface like this one, except with millions of floor directions instead of two.

Compute a gradient numerically, then use it to actually walk downhill:

Python · runs in your browser

What this does: Computes the gradient of f(x,y) = x² + 2y² by nudging each input in turn, checks it against the hand-derived answer [2x, 4y], then takes ten downhill steps and prints the loss shrinking. This tiny loop is gradient descent — the exact procedure that trains every neural network, with two parameters instead of billions.

At a point, ∂L/∂w₁ = 0.02 and ∂L/∂w₂ = −4.8. What does this tell you?

Step 6 — Why this is the language of learning

Put the pieces together and the whole training loop is calculus:

A model produces a prediction; a loss $L$ scores how wrong it is — one number.
The loss depends on every parameter through a long chain of layers.
The chain rule, applied backward through that chain, gives $\partial L / \partial \theta$ for every parameter at once. That's backpropagation.
Those partials, stacked, are the gradient $\nabla_\theta L$ — the direction of steepest increase in wrongness.
Step the opposite way. Repeat a few million times. That's Optimization.

Watch it run on a real surface — drop a starting point, and see the gradient you just learned to compute being followed downhill step by step:

Gradient Descent— interactive, try itOpen in lab →

Click anywhere to drop a new starting point.

Learning rate0.15

Momentum0.70

Brighter = higher loss. Watch how a high learning rate overshoots, and how momentum powers through the small bumps toward a minimum.

The equation above, animated. Drop a start point and watch each step move against the local gradient. Crank the learning rate up until the steps overshoot the valley — proof that the derivative is only trustworthy for small steps.

Explain it yourself

Explain to a curious 12-year-old what a derivative is, using the speedometer picture and no symbols. Then explain the chain rule with the bicycle gears. If you stall on either, that is exactly the section to reread.

Recap — the key ideas

A derivative answers one question: nudge the input a hair — how much does the output move? It is the slope of the curve at a single point, found by shrinking the gap between two points until the answer settles.
Its sign says which way the function is heading; a derivative of zero means flat — a minimum, a maximum, or a saddle.
The rules (power, sum, product, quotient) let you differentiate without ever computing a limit — exact and cheap, which is why frameworks use them instead of numerical nudging.
The chain rule multiplies the local rates along a chain of nested functions. Applied backward through a network, it is backpropagation — and the multiplying is why gradients vanish or explode.
A partial derivative wiggles one input while freezing the rest; the gradient $\nabla f$ stacks them all into a vector pointing straight uphill, with magnitude equal to the steepness.
Training is: compute $\nabla_\theta L$ , step against it, repeat. Every idea on this page exists to make that one term computable.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: attempt the problems below before scrolling back up — pulling an answer out of memory builds far more durable knowledge than recognizing it on the page.
• Spaced repetition: mark this topic complete and it joins your Review queue, resurfacing right before you'd otherwise forget it.
• Interleaving: mix these with problems from Linear Algebra and Optimization rather than grinding derivatives in one block — messier practice, sturdier memory.

By hand: differentiate $f(x) = (3x + 2)^4$ using the chain rule, then verify by nudging $x$ numerically at $x = 1$ .
By hand: find both partials of $f(x, y) = x^2 y + \sin(y)$ and write out the gradient. Which variable is the output more sensitive to at $(2, 0)$ ?
Reason about it: the derivative of $\text{ReLU}(x) = \max(0, x)$ is $1$ for $x > 0$ and $0$ for $x < 0$ . What does a derivative of exactly $0$ mean for a neuron's ability to learn? (This is the "dying ReLU" problem.)
From scratch: write a numerical_grad function and use it to gradient-check a hand-derived derivative — the standard debugging trick when your backprop math looks suspect.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and your terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Complete the chain rule for h(x) = (3x + 2)⁴. Run the starter first to see the numerical answer it should match, then fill in the two TODO lines with the local rates and check that your chain-rule value equals the measured one. Bonus: change the exponent to 5 and confirm your formula still lines up.

editor

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Ready for the next step? Take these gradients to work in Optimization, then see exactly how a network computes them in Backpropagation.