Neural Networks

The multilayer perceptron, taught from zero — one neuron is a weighted vote, an activation is a kink, and a layer is many votes at once. Build up to the forward pass, why depth buys you anything, and what the output layer actually returns.

intermediate#mlp#activations#deep-learning

Start here — what a neural network actually is

Forget the brain pictures and the tangled diagrams for a moment. A neural network is a function: numbers go in, numbers come out. What makes it special is that it's built by stacking one very small, very boring operation thousands of times.

That small operation is: multiply some numbers, add them up, and if the total is negative, throw it away.

That's genuinely it. Everything else on this page — layers, activations, the forward pass, the reason depth matters — is bookkeeping around that one move.

The one-sentence version

A neural network is a long chain of weighted votes, with a tiny bend inserted after each round of voting. The weights are numbers the network learns; the bend is what stops the whole chain from collapsing back into a single, boring vote. Chain enough of them together and the function you get can be shaped into almost anything.

is the name for that whole chain. The particular chain on this page — every unit connected to every unit in the layer before it — is called a , or MLP. It is the ancestor of every architecture you'll meet later: CNNs, Transformers, and diffusion models are all MLPs with extra structure bolted on.

How to read this page

It teaches from first principles, assuming nothing but arithmetic. Flip the Depth switch at the top when you want the formal notation and the derivations — they also open automatically once you've completed the prerequisites (Machine Learning and Optimization). Nothing is hidden for good; the deeper material just sits folded up until you want it.

The neuron — one weighted vote

Start with a single unit. A (also called a unit) does exactly three things, in order:

Weigh each input — multiply it by a number saying how much that input matters.
Total them up, and add one extra number that shifts the result.
Bend the total with a simple function, then pass it on.

Think of it like a hiring panel scoring a candidate:

Each panel member looks at one piece of evidence — years of experience, a code sample, an interview — and each has their own private sense of how much that evidence is worth. They shout their weighted opinions, the chair adds them up, and then applies a personal bar: if the total doesn't clear a certain level, this is a no. The weights are how much each piece of evidence counts. The extra number is the chair's built-in optimism or scepticism. The bar is the bend.

The "how much each input matters" numbers are the , and the extra shifting number is the .

One neuron scoring an email for spam, by hand

Our neuron sees two features of an email and must output "how spammy is this?"

$x_1 = 3$ — the email contains 3 links.
$x_2 = 2$ — it contains 2 exclamation marks.

The neuron has already learned its numbers: weights $w_1 = 0.8$ and $w_2 = 0.5$ , bias $b = -2.0$ .

Step 1 — weigh each input:

Links: $0.8 \times 3 = 2.4$
Exclamation marks: $0.5 \times 2 = 1.0$

Step 2 — total them and add the bias: $2.4 + 1.0 - 2.0 = 1.4$

Step 3 — bend it into a probability. Squashing $1.4$ through the S-shaped sigmoid function gives about $0.80$ .

So the neuron says: 80% spam. Notice what the bias did — it subtracted 2.0, meaning this neuron demands a fair amount of evidence before it starts shouting "spam". An email with no links and no shouting would score $0 + 0 - 2.0 = -2.0$ , which squashes to about $0.12$ : probably fine.

The important thing to notice: steps 1 and 2 together are just a dot product — multiply matching slots and add — the operation from Linear Algebra. A neuron is a dot product with a shift and a bend. Nothing more.

Try to recall

What does the bias do that the weights cannot?

Hint: What happens to the neuron's output when every single input is zero?

What one neuron looks like from above

A neuron with two inputs is drawing a soft dividing line across the input space. On one side its output is near 0, on the other near 1, and there's a smooth ramp in between. Here is a real one — weights $[1, 1]$ , bias $-1$ — with the output shown as colour:

What a single neuron computes across its whole input space— interactive, drag & zoom

Loading chart…

One neuron with weights [1, 1] and bias -1, drawn over every possible input. The 0.5 contour running diagonally is its decision boundary — a perfectly straight line. Change the weights and the line tilts; change the bias and it slides. That straightness is the whole limitation of a single neuron, and the reason the rest of this page exists.

Look hard at that boundary: it is a straight line. No amount of retuning $w_1$ , $w_2$ and $b$ will ever bend it. Hold onto that — it is exactly the wall we'll need layers to climb over.

Python · runs in your browser

What this does: Computes one neuron by hand in NumPy — the dot product of weights with inputs, plus the bias, squashed by a sigmoid — and reproduces the 0.80 from the worked example above. Then it feeds in a clean email so you can watch the bias alone keep the score low.

Activations — the kink that makes depth mean anything

Here is the most important fact on this page, and it is easy to miss.

If you stack two neurons with no bend between them — pure weighted sums — you have not built anything new. Two matrix multiplies in a row are exactly equal to one matrix multiply. The second layer buys you nothing at all.

Two linear layers collapsing into one, by hand

Let layer 1 be the matrix $W_1 = \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}$ and layer 2 be $W_2 = \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}$ , with input $\mathbf{x} = [1, 1]$ .

The two-layer way:

Layer 1: $W_1\mathbf{x} = [2\times1 + 0\times1,\ 0\times1 + 3\times1] = [2, 3]$
Layer 2: $W_2[2, 3] = [1\times2 + 1\times3,\ 1\times2 - 1\times3] = [5, -1]$

The one-layer way — multiply the two matrices together first:

W_2 W_1 = \begin{bmatrix} 2 & 3 \\ 2 & -3 \end{bmatrix}, \qquad (W_2W_1)\,\mathbf{x} = [2+3,\ 2-3] = [5, -1]

Identical. The "deep" network and the shallow one compute precisely the same function, for every possible input. A hundred stacked linear layers would still just be one matrix in disguise.

Why the bend is not optional

Depth without non-linearity is a lie. The only thing that stops a deep network from folding flat into a single layer is a function between the layers that a matrix cannot imitate. That function is the — and remarkably, it can be almost absurdly simple. The one that powers most of modern deep learning is just: if the number is negative, make it zero.

That one is the , and its whole definition is $\max(0, z)$ . It was a genuine turning point: rectifier units let deep networks train well without the unsupervised pre-training that had been considered necessary.

Think of it like a stack of photocopiers versus a stack of photocopiers with a paper cutter:

Put a page through three photocopiers, each set to enlarge by some percentage, and you could have got the same result with one copier set to the product of those percentages — the machines compose into a single machine. Now put a paper cutter between them that trims off anything hanging past the edge. Suddenly the order matters, information is genuinely lost and reshaped at each stage, and no single copier setting can reproduce the result. The activation is that cutter.

Try the three classic bends side by side — drag and compare their shapes, and pay attention to how flat each one goes at the extremes:

Activation Functions— interactive, try itOpen in lab →

f(x) — GELUf′(x) — derivative

The three activations that matter most. Notice how sigmoid and tanh flatten out at both ends — that flatness is where gradients go to die — while ReLU stays a clean 45-degree ramp forever on the right-hand side. That single difference is most of why modern networks use ReLU and its descendants.

Python · runs in your browser

What this does: Proves the collapse numerically. It applies two linear layers in sequence, then applies their product as one merged matrix, and shows the results are identical — so the second layer was free of charge and worth nothing. Then it drops a ReLU between them and the two stop agreeing, which is exactly the point.

You build a 20-layer network but forget the activation functions. What have you actually built?

A layer — many neurons at once

One neuron asks one question. A is a row of neurons all looking at the same inputs simultaneously, each with its own weights and its own bias — so you get several different questions answered at once.

A panel of specialists, not a committee

Give the same photo to twenty neurons and, because their weights differ, they end up asking twenty different questions: is there an edge here, is this region bright, does this look like fur? None of them is told what to look for — the questions emerge from training. The layer's output is a vector of answers, and it is a new description of the input, written in the layer's own invented vocabulary.

A three-unit layer, computed by hand

Input $\mathbf{x} = [1, 2]$ , going into a layer of three ReLU units.

Unit	Weights	Bias	Weighted sum + bias	After ReLU
1	$[1, -1]$	$0.5$	$1(1) + (-1)(2) + 0.5 = -0.5$	$0$
2	$[0.5, 0.5]$	$-1$	$0.5(1) + 0.5(2) - 1 = 0.5$	$0.5$
3	$[-1, 2]$	$0$	$(-1)(1) + 2(2) + 0 = 3$	$3$

The layer's output is $[0,\ 0.5,\ 3]$ .

Two things to notice. First, unit 1 switched itself off — its ReLU zeroed a negative total, so it contributes nothing at all for this particular input. Different inputs will switch on a different subset of units, which means the network effectively rewires itself per example. Second, we started with 2 numbers and ended with 3: a layer is free to change the width of the representation, growing or shrinking it as needed.

Try to recall

A layer takes a 2-number input and produces a 3-number output. How many weights and how many biases does it have?

Hint: Every one of the output units needs its own weight for every input.

Python · runs in your browser

What this does: Builds the exact three-unit layer from the table above and confirms it outputs [0, 0.5, 3] — watch unit 1 get switched off by the ReLU. Then it runs a whole batch of four examples through the same layer in a single matrix multiply, which is precisely what a GPU executes.

The forward pass — stacking layers into a network

Now chain the layers. Feed the input into layer 1; feed layer 1's output into layer 2; keep going until the last layer hands you an answer. That single sweep from input to output is the .

Think of it like a relay of translators, each inventing a better language:

The first translator receives raw pixels and renders them into a language of edges. The second takes edges and renders them into corners and textures. The third takes textures and renders them into a language of eyes, wheels, and fur. Nobody assigns these vocabularies — each translator invents whatever intermediate language makes the next one's job easiest, because that is what lowers the final error. By the last hand-off, the input has been rewritten into terms where the answer is a single straight cut away.

What depth actually buys

Every layer re-describes the data. A single layer can only slice the input space with flat cuts; but if the layer before it has already bent and folded the space, then a flat cut in that folded space is a curved, complicated boundary back in the original space. Depth is not about more cuts — it's about cutting in a space that earlier layers have made easy to cut.

A full forward pass through a 2-2-1 network, by hand

Input $\mathbf{x} = [1, 2]$ . Two inputs, a hidden layer of two ReLU units, one sigmoid output.

Layer 1 — weights $W_1 = \begin{bmatrix} 1 & -1 \\ 0 & 2 \end{bmatrix}$ , biases $\mathbf{b}_1 = [0, -1]$ :

Unit 1: $1(1) + (-1)(2) + 0 = -1$
Unit 2: $0(1) + 2(2) - 1 = 3$
Pre-activations $\mathbf{z}_1 = [-1, 3]$ ; after ReLU, $\mathbf{h} = [0, 3]$ .

Layer 2 — weights $W_2 = \begin{bmatrix} 2 & -0.5 \end{bmatrix}$ , bias $b_2 = 1$ :

$2(0) + (-0.5)(3) + 1 = -0.5$

Output — squash with sigmoid: $\sigma(-0.5) \approx 0.38$ .

The network's answer is 0.38. Notice that hidden unit 1 was switched off, so the entire final answer flowed through unit 2 alone. Feed a different input and a different subset would carry the signal — the network is quietly choosing a different path for every example.

Python · runs in your browser

What this does: Runs the complete 2-2-1 forward pass from the worked example, printing the pre-activations, the hidden activations, and the final probability — which should come out at 0.3775, matching the hand calculation exactly. Every deep network is this same code with more layers.

Try to recall

During a forward pass, which quantities does the framework need to keep in memory, and why?

Hint: Think about what the backward pass will need to look at later.

Why depth buys you anything — the XOR problem

Time to prove that layers really do something a single neuron cannot. The classic demonstration is XOR: output 1 when exactly one of two binary inputs is 1, and 0 otherwise.

XOR — four points that no straight line can separate— interactive, drag & zoom

Loading chart…

The two orange squares must end up on one side and the two blue circles on the other. Try it with a ruler: every straight line you draw gets at least one point wrong, because the matching classes sit diagonally opposite each other. A single neuron draws exactly one straight line, so a single neuron cannot solve XOR — no matter how long you train it.

The two classes sit on opposite diagonals. A single neuron draws one straight line, so a single neuron provably cannot solve XOR. This was the objection that helped stall neural network research for years. The escape is a hidden layer.

Solving XOR by hand with two ReLU units

Use a hidden layer of two ReLU units with weights $W_1 = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$ and biases $\mathbf{c} = [0, -1]$ , then an output that computes $h_1 - 2h_2$ .

Both hidden units compute the same sum $x_1 + x_2$ ; the only difference is that unit 2 subtracts 1 before its ReLU. Run all four inputs through:

Input	$x_1 + x_2$	$\mathbf{z}$ (after biases)	$\mathbf{h}$ (after ReLU)	$h_1 - 2h_2$
$(0,0)$	0	$[0, -1]$	$[0, 0]$	$0$
$(0,1)$	1	$[1, 0]$	$[1, 0]$	$1$
$(1,0)$	1	$[1, 0]$	$[1, 0]$	$1$
$(1,1)$	2	$[2, 1]$	$[2, 1]$	$0$

Output: $[0, 1, 1, 0]$ — exactly XOR.

Watch precisely where the magic happened. The ReLU on unit 2 clipped $-1$ up to $0$ for the first row. That single clip is the only non-linear act in the whole network, and it is what breaks the symmetry: without it, $h_1 - 2h_2$ would be a linear function of $x_1 + x_2$ , and would give $[2, 1, 1, 0]$ — steadily decreasing, unable to come back up. The hidden layer folded the input space so that $(0,0)$ and $(1,1)$ landed on the same side of a straight cut.

Python · runs in your browser

What this does: Feeds all four XOR inputs through the hand-designed two-unit hidden layer and prints [0, 1, 1, 0] — a problem that is provably impossible for a single neuron, solved by one extra layer with no training at all. The last line deletes the ReLU and shows the answer collapse to something no output weight could fix.

How hidden units build arbitrary shapes

XOR shows depth solves one problem. The deeper reason a hidden layer is so powerful: a handful of ReLUs can be combined into a bump, and bumps can be stacked into any shape you like.

Three ReLUs adding up to a bump— interactive, drag & zoom

Loading chart…

Three ReLU units, each kinking at a different place, weighted +1, -2, +1 by the output layer. Individually they are dull ramps that only ever increase; combined they produce a localized triangular bump that is flat everywhere else. Add more units with different kink positions and you can place a bump anywhere, at any height — which is how a wide hidden layer approximates any curve you like.

Each ReLU is a boring ramp that only ever goes up. But weight three of them $+1$ , $-2$ , $+1$ and they cancel each other everywhere except in a narrow window, leaving a localized bump. Give the layer more units and you can put a bump anywhere, of any width and height — and any function can be built by stacking enough bumps.

The universal approximation theorem says a one-hidden-layer network can approximate any continuous function. Why does anyone bother with depth?

The output layer — turning the last numbers into an answer

Hidden layers invent a useful description. The output layer converts that description into whatever shape your task requires, and its activation is chosen by the task rather than by fashion:

Task	Output units	Output activation	Reads as
Predict a number (regression)	1	none (identity)	the number itself
Yes/no (binary classification)	1	sigmoid	probability of "yes"
Pick one of K classes	K	softmax	a probability per class, summing to 1
K independent yes/no tags	K	sigmoid on each	independent probabilities

Why multiclass needs softmax rather than K sigmoids

Put a sigmoid on each of ten class scores and you get ten independent probabilities that might sum to 3.7 — meaningless as "which digit is this?". instead makes the classes compete: raising one score necessarily lowers the others, because everything is divided by a shared total. Use softmax when exactly one answer is correct, and independent sigmoids when several tags can be true at once.

Try to recall

Your network must tag a photo with any of {beach, sunset, people} — several can be true at once. Softmax or sigmoid on the output?

Hint: Does choosing one tag have to make the others less likely?

Where the weights come from

Everything above assumed the weights were already set. In reality they start as small random numbers and are learned, by exactly the loop from Optimization:

Forward pass — run the input through and get a prediction.
Loss — score how wrong it was with a single number.
Backward pass — compute how much each weight contributed to that wrongness. This is backpropagation, and it is just the chain rule applied to the nested formula from earlier.
Step — nudge every weight a little in the direction that lowers the loss. Repeat a few hundred thousand times.

Here is the whole thing, from scratch in NumPy, learning XOR — the problem a single neuron cannot touch:

Python · runs in your browser

What this does: Trains a small MLP on XOR completely from scratch — forward pass, cross-entropy loss, hand-written gradients, and a gradient-descent step — then plots the loss falling. It starts near 0.71 (random guessing) and ends near zero, with predictions snapping to 0, 1, 1, 0. Everything on this page, working together in 25 lines.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
X = np.array([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = np.array([[0.], [1.], [1.], [0.]])          # XOR

W1 = rng.normal(0, 1, (2, 8)); b1 = np.zeros(8)
W2 = rng.normal(0, 1, (8, 1)); b2 = np.zeros(1)
sigmoid = lambda v: 1. / (1. + np.exp(-v))

lr, losses = 0.5, []
for step in range(3000):
    z1 = X @ W1 + b1;  h = np.maximum(0., z1)   # forward
    p = sigmoid(h @ W2 + b2)
    losses.append(float(-np.mean(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))))

dz2 = (p - y) / len(X)                      # backward
    dW2, db2 = h.T @ dz2, dz2.sum(0)
    dz1 = (dz2 @ W2.T) * (z1 > 0)
    dW1, db1 = X.T @ dz1, dz1.sum(0)

W1 -= lr*dW1; b1 -= lr*db1                  # step downhill
    W2 -= lr*dW2; b2 -= lr*db2

print("start loss:", round(losses[0], 4), " final loss:", round(losses[-1], 4))
print("predictions:", np.round(p.ravel(), 3), " target: [0 1 1 0]")

plt.figure(figsize=(5, 3))
plt.plot(losses, color="#d97757")
plt.xlabel("training step"); plt.ylabel("loss"); plt.title("An MLP learning XOR")
plt.tight_layout()

The mistakes that catch everyone

Forgetting the activation. The network trains fine and quietly behaves like a linear model. If a deep net performs no better than logistic regression, check this first.
Initializing all weights to zero. Every unit in a layer then computes the same thing and receives the same gradient forever — they never differentiate. Random initialization is what breaks the symmetry.
Shape errors. Confusing $W\mathbf{x}$ with $\mathbf{x}W$ , or $(B, 1)$ with $(B,)$ , silently broadcasts into a wrong-but-runnable answer. Print shapes constantly.
Softmax applied twice — once by you, once inside the loss function. Frameworks expect raw logits.
A learning rate that kills ReLUs. Too large a step drives biases strongly negative and units die permanently. If a chunk of your network outputs exact zeros for every input, lower the learning rate.

Explain it yourself

Explain to a friend what one neuron does, and then why stacking neurons without an activation function gains you nothing at all. Use the hiring-panel and photocopier pictures, not symbols. If you cannot say clearly why the activation is what makes depth real, that is the section to reread.

Recap — the key ideas

A neuron is a weighted vote: multiply each input by a weight, add them plus a bias, then bend the total. Its decision boundary is always a straight line.
The activation function is the bend, and it is not optional — without it, any stack of layers collapses into a single linear layer. ReLU ( $\max(0, z)$ ) is the default.
A layer is many neurons sharing the same inputs, which is exactly one matrix multiply: $\mathbf{h} = \phi(W\mathbf{x} + \mathbf{b})$ .
The forward pass chains layers, each re-describing the data so the next one's job is easier. Depth means cutting in a space earlier layers already folded.
XOR is unsolvable by one neuron and trivial with one hidden layer; a handful of ReLUs combine into bumps, and bumps build any shape — the intuition behind universal approximation, which promises representability, not learnability.
The output layer matches the task: identity for regression, sigmoid for binary, softmax for mutually-exclusive classes.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: attempt the problems below before scrolling back up — pulling an answer from memory beats recognizing it on the page.
• Spaced repetition: mark this topic complete and it joins your Review queue, resurfacing right before you'd forget it.
• Interleaving: mix these with problems from Optimization and Linear Algebra rather than grinding one type — messier practice, sturdier memory.

By hand: take the 2-2-1 network from the forward-pass example and push $\mathbf{x} = [0, 0]$ through it. Which hidden unit fires now, and what is the output?
By hand: design a single neuron that computes logical AND on two binary inputs, and another that computes OR. Then convince yourself no single neuron computes XOR by trying to draw its line.
From scratch: widen the hidden layer in the training cell below from 8 units to 2, then to 64. Where does it stop learning XOR reliably, and why?
Break it deliberately: delete the ReLU from the training cell and watch the loss stall around $0.69$ — the value you get from guessing 50/50 on everything.
Count parameters: how many weights and biases are in a 784 → 128 → 64 → 10 MLP? (This is a real MNIST classifier.)

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and your terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Run the starter to watch an 8-unit hidden layer solve XOR. Then do the TODOs in order: (1) set HIDDEN to 2 and rerun with a few different seeds — change default_rng(0) to 1, 2, 3 — and see how often it fails. (2) Remove the non-linearity by making both TODO-2 changes, and watch the loss stall at 0.6931 with every prediction stuck at 0.5. Explain to yourself why that exact number, and not something lower.

editor

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Next: you've been handed the gradients on this page without explanation. Go and earn them in Backpropagation, then stop writing them by hand and let a framework do it in PyTorch.