Convolutional Neural Networks

The architecture that gave machines sight — taught from zero. Start with the idea of sliding a tiny pattern-detector over a picture, then build up to convolutions, pooling, and the residual connections behind modern vision models.

intermediate#cnn#convolution#resnet#vision

Start here — what this is really about

Forget the acronym for a moment. A convolutional neural network is a machine that learns to see by doing one small thing over and over: it slides a tiny pattern-detector across a picture and asks, at every spot, "does the thing I'm looking for show up right here?"

That's the whole idea. One detector might light up wherever there's a vertical edge. Another might fire on a patch of fur, or the corner of an eye. Stack enough of these detectors — with the simple ones feeding the complicated ones — and the machine goes from "there's an edge here" all the way up to "that's a cat." Everything else on this page — the sums, the Greek letters, the layer names — is just careful bookkeeping for that one sliding-detector move.

How to read this page

The page adapts to you. By default it teaches from first principles. When you're ready — or once you've completed the prerequisites — flip the Depth switch at the top to reveal the formal notation, derivations, and code. Nothing is hidden for good; the deeper material sits behind the Go deeper panels so you can open it the moment you're curious.

Why not just look at every pixel at once?

The obvious first idea is: flatten the image into one long list of numbers and feed it to a plain neural network — a where every input pixel connects to every neuron. It works for tiny inputs. For real images it falls apart in two ways.

Two things go wrong with connect-everything-to-everything

It's enormous. A modest photo has hundreds of thousands of pixels. Wiring each one to each neuron means an astronomical number of to store and train.

It has no idea that a cat is a cat wherever it stands. If the network learns to spot a cat in the top-left corner, that knowledge is stuck in the top-left corner. Move the cat to the bottom-right and, to the MLP, it is a completely unrelated pattern it has never seen.

A CNN fixes both by refusing to connect everything to everything. Instead it uses one small detector and slides it everywhere — so a cat-detector automatically works in every corner, and there's only one small set of weights to learn instead of a giant one.

Try to recall

Give the two separate reasons a plain MLP is a poor fit for raw images.

Hint: One is about size; one is about position.

The convolution — sliding a tiny stencil over the image

Here is the one operation the whole architecture is built from. Take a small grid of numbers — call it a (also called a ), often just $3 \times 3$ . Lay it over the top-left corner of the image, multiply each filter number by the pixel underneath it, and add all those products into a single number. Slide the filter one step to the right and do it again. Cover the whole image and you get a new grid of numbers — a that lights up wherever the filter's pattern was found.

A filter is a rubber stamp that scores a match

Think of the filter as a little stamp shaped like the thing you're hunting for — say, a vertical edge. Pressing it down anywhere gives a high score when the image underneath matches the stamp and a low score when it doesn't. Sliding the stamp across the whole picture turns one image into a heat-map of "how strongly does my pattern appear at each spot?"

Think of it like sweeping a flashlight across a dark wall:

You can't take in the whole wall at once, so you sweep a small circle of light across it, spot by spot, noticing features as they pass through the beam. The filter is that beam: it only ever sees a small patch, but by sweeping it everywhere you build up a complete picture — and crucially, you use the same beam the whole way, which is exactly the weight-sharing that makes CNNs efficient.

Running a vertical-edge filter by hand

Take a tiny $3 \times 3$ patch of a grayscale image where the left side is bright and the right side is dark — a vertical edge:

10  10   0
10  10   0
10  10   0

Slide this vertical-edge filter over it:

 1   0  -1
 1   0  -1
 1   0  -1

Multiply matching cells and add everything up:

Left column: $(10 \times 1) + (10 \times 1) + (10 \times 1) = 30$
Middle column: everything is multiplied by $0$ , contributing $0$
Right column: $(0 \times -1) + (0 \times -1) + (0 \times -1) = 0$
Total: $30 + 0 + 0 = 30$ — a big score, because there really is a bright-to-dark edge here.

Try the same filter on a flat patch (all $10$ s) and the pluses and minuses cancel to $0$ — no edge, no signal. The filter has learned to shout at edges and stay silent on flat regions.

Try to recall

Each number in a feature map is produced by which basic operation between the filter and the image patch under it?

Hint: Multiply matching cells, then add them all up.

See it happen

This runs a real vertical-edge filter over a little image whose left half is bright and right half is dark. Watch the output light up in a single column — right where the edge is:

Python · runs in your browser

Output size — how big is the feature map?

When you slide a filter, the output is a little smaller than the input (the filter can't hang off the edge), and you can change that by padding the border with zeros or by striding — jumping more than one pixel at a time. Two new knobs:

— a frame of zeros around the image so the filter can reach the very edges.
— the step size of the slide.

Same padding — keep the size unchanged

To keep the spatial size the same with stride $1$ , set $p = \lfloor k/2 \rfloor$ . For a $3 \times 3$ filter that means pad by $1$ . Plug $k=3, p=1, s=1$ into the formula and $W_{out} = W$ — the size is preserved.

A 3×3 convolution with stride 1 and padding 1 is applied to a 64×64×16 feature map, producing 32 output channels. What is the output shape?

Stacking filters into channels

One filter finds one kind of pattern. To see edges and blobs and textures, you run many filters over the same input and stack their feature maps. The number of stacked maps is the layer's . A colour image starts with 3 channels (red, green, blue); a filter there is actually $3 \times 3 \times 3$ — it looks across all input channels at once — and each filter still produces a single feature map.

Pooling — shrink the map, keep the gist

After a convolution you usually want to make the feature map smaller — fewer numbers to process, and a bit of tolerance to exactly where a feature sat. That's .

Squint at the picture

Chop the feature map into little $2 \times 2$ tiles and replace each tile with a single summary number — its maximum (max pooling, keep the strongest response) or its average. You lose fine positional detail but keep the gist, and the map gets four times smaller. It's like shrinking a photo to a thumbnail: the cat is still obviously a cat.

Think of it like asking did anyone in this room see a cat:

Max pooling over a tile is exactly that question. You don't care which pixel in the tile lit up — only whether the pattern showed up somewhere nearby. That is where a little bit of comes from: nudge the input a pixel and the tile's maximum is usually unchanged.

The building blocks, assembled

Four pieces do almost all the work in a CNN:

Convolution layer — slides learnable filters; sets the channel count via out_channels.

Activation (ReLU) — injects non-linearity with max(0, x), so stacked layers can express more than one big linear map.

Pooling — downsamples (max/avg) for a smaller map and shift-tolerance.

Batch Norm — rescales each channel's activations to a stable range; steadies and speeds up training.

A canonical block is Conv → BatchNorm → ReLU → (optional Pool), repeated and stacked — the spatial resolution shrinking and the channel count growing as you go deeper, so the network trades "where exactly" for "what is it."

Python · needs a GPU — run on Colab

import torch
import torch.nn as nn

class ConvBlock(nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.conv = nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=1, bias=False)
        self.bn   = nn.BatchNorm2d(out_ch)
        self.act  = nn.ReLU(inplace=True)

    def forward(self, x):
        return self.act(self.bn(self.conv(x)))   # Conv -> BN -> ReLU

class SmallCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            ConvBlock(3, 32), nn.MaxPool2d(2),   # 32 -> 16
            ConvBlock(32, 64), nn.MaxPool2d(2),  # 16 -> 8
            ConvBlock(64, 128), nn.AdaptiveAvgPool2d(1),
        )
        self.head = nn.Linear(128, num_classes)

    def forward(self, x):
        x = self.features(x).flatten(1)
        return self.head(x)

model = SmallCNN()
print(sum(p.numel() for p in model.parameters()), "parameters")

The feature hierarchy — simple detectors feeding complex ones

The magic is what happens when you stack these blocks. Early filters see raw pixels and can only manage simple things — edges, colour blobs. But the next layer's filters look at the first layer's feature maps, so they can combine edges into corners and textures. Deeper still, those combine into object parts, then whole objects. Each layer's — the slice of the original image it can ultimately see — grows with depth.

Depth	What the filters detect
Layer 1	Oriented edges, colour blobs
Layer 2–3	Textures, corners, simple shapes
Middle	Object parts (eyes, wheels, text)
Deep	Whole objects & scene concepts

This is the same coarse-to-fine idea as classical vision pyramids — but learned end-to-end from data rather than hand-designed.

Try to recall

Why can a layer deep in the network detect a whole object, when a first-layer filter can only find an edge?

Hint: What does each layer take as its input?

Residual connections — how we train really deep networks

Naively, deeper should mean better. But very deep plain CNNs get paradoxically worse — even on the training set — because the stacked layers struggle to pass signal and gradients cleanly through dozens of transformations. ResNet fixed this with one tiny change: let each block add its input back to its output.

Give the signal an express lane

Instead of forcing every block to reinvent the whole signal, add a shortcut that carries the input straight through, and let the block learn only the change it wants to make on top. If a block has nothing useful to add, it can output zero and the input sails past untouched. Extra depth can now only help, never hurt — and gradients get a clean highway straight back to the early layers.

Think of it like a bypass road around a town:

Traffic (the signal) can take the express bypass and arrive unchanged, or detour through the town (the block's layers) to pick something up. Because the bypass is always there, adding another town down the road never makes the trip worse — the worst a block can do is add nothing.

Why do residual connections help train very deep networks?

Common mistakes

Traps that bite everyone

Forgetting that Conv2d expects (N, C, H, W) — batch, channels, height, width — not (N, H, W, C).
Applying BatchNorm after ReLU instead of before (the usual convention is Conv → BN → ReLU).
Using a huge fully-connected head instead of global average pooling — it wastes parameters and overfits.
Not disabling bias in a conv that's immediately followed by BatchNorm — the bias is redundant, since BatchNorm re-centers anyway.
Losing track of the output-size arithmetic and hitting a shape-mismatch crash a few layers down.

Best practices that actually move the needle

Data augmentation (flips, crops, colour jitter, RandAugment) is often the single biggest accuracy lever on small datasets.
Start from pretrained ImageNet weights whenever your dataset is small — see Transfer Learning.
Use global average pooling before the classifier head instead of a giant dense layer.
Prefer modern backbones (ResNet, ConvNeXt, EfficientNet) over hand-rolled stacks for real work.

Explain it yourself

In your own words, explain convolution using the flashlight or rubber-stamp picture — no formulas. Why does sliding one small filter everywhere beat wiring every pixel to every neuron? If you stall on the why, that's the exact spot to reread.

Recap — the key ideas

A CNN learns to see by sliding small pattern-detectors (filters) across an image and stacking simple detectors into complex ones.
Convolution = lay a filter on a patch, multiply matching cells, sum to one number; slide it everywhere to get a feature map. This reuses one small set of weights (parameter sharing) instead of wiring every pixel.
Padding and stride set the output size via $W_{out} = \lfloor (W - k + 2p)/s \rfloor + 1$ ; many filters stacked give many channels.
Pooling shrinks maps and buys a little shift-tolerance; a canonical block is Conv → BatchNorm → ReLU → Pool.
Depth builds a feature hierarchy (edges → parts → objects); residual connections ( $y = \mathcal{F}(x) + x$ ) let that depth reach hundreds of layers.

Practice — and how to make it stick

Learn it the way that actually works

Three research-backed habits, built into this platform:
• Retrieval practice: attempt the problems below before rereading — pulling an answer from memory beats recognizing it on the page.
• Spaced repetition: mark this topic complete and it's added to your Review queue, resurfacing right before you'd forget.
• Interleaving: mix these with problems from Optimization and Linear Algebra rather than grinding CNNs alone — messier practice, sturdier memory.

Warm-up lab — feel a filter with your own hands. Edit the kernel below and rerun. Try a horizontal-edge detector [[1,1,1],[0,0,0],[-1,-1,-1]], or a blur np.ones((3,3))/9, and watch how the feature map changes:

Python · runs in your browser

Then build up to the real thing:

By hand: apply a $3\times3$ filter to a $5\times5$ patch with stride $2$ and padding $0$ ; predict the output size with the formula, then check it.
From scratch: implement the SmallCNN above and train it on CIFAR-10 — aim for >80% test accuracy. Add augmentation and compare.
Fine-tune: swap in a pretrained ResNet-18 and compare accuracy and training time against your from-scratch net.
Look inside: visualize the first-layer filters and a few feature maps to see the edges-to-objects hierarchy for yourself.

Where to go next

Once convolutions click, branch into Object Detection and Semantic Segmentation, or see how attention challenges the convolutional monopoly in Vision Transformers.