Regularization

Start here — the one failure that regularization exists to fix

Imagine you hand a student a hundred practice problems, each with the answer printed underneath. Two weeks later they score 100% on those hundred problems. Did they learn the subject?

You can't tell yet. There are two very different students who both score 100%:

• One understood the material and can solve problems they've never seen.
• One memorized the answer key and will fall apart the moment the numbers change.

A neural network can be either of these students, and the frustrating part is that they look identical while you're training. That's the entire problem this lesson is about.

Generalization is the goal: doing well on new data. Overfitting is the failure: acing the practice problems, flunking the exam. And regularization is the toolbox for preventing it.

Seeing overfitting happen — the two curves

You cannot fix what you cannot see, so the first skill is detecting overfitting. It takes two numbers, not one.

Split your data into two piles before training starts:

• The training set, which the model learns from.
• The validation set, which it never learns from and which you use as a stand-in for the real world.

Now track the loss on both after every epoch. That pair of curves is the single most informative plot in deep learning:

That gap between the two curves has a name.

Generalization gap is simply validation loss minus training loss. Watching it is how you diagnose everything in this lesson.

Measure it yourself

Run this. It fits polynomials of growing flexibility to eleven noisy points and reports the error on those points versus on fresh ones:

Read the printed table before moving on. Degree 3 has the lowest error on unseen points. Degree 10 has zero error on the training points — a perfect score — and roughly double the error of degree 3 on new ones. The model with the perfect training score is the worse model.

Why it happens — capacity versus data

Overfitting isn't random bad luck. It comes from a specific mismatch.

Capacity is a model's flexibility. Eleven data points and a degree-10 polynomial means eleven equations and eleven free coefficients — the model has exactly enough freedom to thread through every single point, noise included. It doesn't need to find the underlying pattern because there is a memorization solution available, and memorization drives the training loss lower than the truth does.

Drag the degree slider in the applet below. Watch how a low degree slices straight past the points, a middling degree traces the shape, and a high degree starts snaking through every point exactly:

Here is that same idea on the exact eleven points from the code cell above — three models, three levels of flexibility, one truth:

Diagnose before you treat

Regularization is a treatment for one specific disease. Applying it to the wrong one makes things worse, so always read the two numbers first:

Now the toolbox itself. There are five tools, roughly in the order you should reach for them.

Tool 1 — More data, and the cheap version: augmentation

The most effective anti-overfitting technique is not clever. It is more data.

Real labelled data is expensive. So we fake it: data augmentation takes each example and produces variations that are still correctly described by the same label.

Tool 2 — Weight decay: charge a fee for complexity

Suppose you can't get more data. The next move is to tell the optimizer that not all solutions are equally welcome.

Weight decay adds a second term to the thing being minimized: the ordinary loss, plus a charge proportional to how big the weights are.

Watch the effect on the wildly overfitting degree-10 fit from earlier — same model, same data, just a fee attached:

The printed numbers make the argument better than any picture: with no penalty the biggest coefficient is in the hundreds and the sum of squared weights is in the hundreds of thousands. A penalty of $\lambda = 10^{-3}$ — a tiny fee — collapses that to single digits and improves the error on unseen data. The model was never getting anything useful from those enormous cancelling coefficients.

Two flavours of fee: L2 and L1

Squaring the weights is one way to charge for size. Taking absolute values is another, and it behaves very differently.

L1 regularization replaces $\sum \theta_i^2$ with $\sum |\theta_i|$. The geometric picture below shows why that one change makes weights land on exactly zero:

Tool 3 — Dropout: randomly break the network while it trains

Weight decay limits how large the weights can be. Dropout attacks a different failure: neurons that only work as a fixed team.

Dropout is that idea. The mechanics have one wrinkle worth understanding before the formula.

Here is what the masks look like across a few training steps — same layer, same eight units, a different random pattern every time:

Tool 4 — Early stopping: the free one

Look back at the two-curve plot. The best model of that entire run existed at epoch 11, and training continued for 19 more epochs, steadily making it worse. Early stopping is simply: notice that, and stop.

This is the cheapest technique in the lesson: no new hyperparameter to tune beyond patience, no extra compute — in fact it saves compute — and it works alongside every other tool here.

Tool 5 — the rest of the toolbox

These come up constantly and are worth recognizing even before you use them.

Putting it together — the practical recipe

Here is the whole toolbox in one PyTorch training loop:

import torch, torch.nn as nn
from torchvision import transforms

# --- Tool 1: augmentation, applied fresh every time an example is loaded ---
train_tf = transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.7, 1.0)),
    transforms.RandomHorizontalFlip(),          # valid for cats, NOT for road signs or text
    transforms.ColorJitter(0.2, 0.2, 0.2),
    transforms.ToTensor(),
])
val_tf = transforms.Compose([transforms.Resize(256), transforms.CenterCrop(224),
                             transforms.ToTensor()])   # never augment the validation set

# --- Tool 3: dropout, declared in the model ---
model = nn.Sequential(
    backbone,
    nn.Dropout(p=0.2),
    nn.Linear(512, num_classes),
)

# --- Tool 2: weight decay, decoupled, and not applied to biases or norm parameters ---
decay, no_decay = [], []
for name, param in model.named_parameters():
    (no_decay if param.ndim <= 1 or "bias" in name else decay).append(param)
opt = torch.optim.AdamW(
    [{"params": decay, "weight_decay": 0.05}, {"params": no_decay, "weight_decay": 0.0}],
    lr=3e-4,
)

# --- Tool 5: label smoothing, built into the loss ---
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)

# --- Tool 4: early stopping with a restored best checkpoint ---
best, patience, waited = float("inf"), 10, 0
for epoch in range(max_epochs):
    model.train()                                # dropout ON
    for x, y in train_loader:
        opt.zero_grad()
        criterion(model(x), y).backward()
        opt.step()

    model.eval()                                 # dropout OFF — the classic bug if omitted
    with torch.no_grad():
        val_loss = sum(criterion(model(x), y).item() for x, y in val_loader) / len(val_loader)

    if val_loss < best:
        best, waited = val_loss, 0
        torch.save(model.state_dict(), "best.pt")     # keep the good one
    else:
        waited += 1
        if waited >= patience:
            break

model.load_state_dict(torch.load("best.pt"))     # restore the best, not the last

Practice — and how to make it stick

Start with the lab. It is the real workflow: you never pick a regularization strength by reasoning about it, you pick it by measuring.

Then work through these:

1. Force an overfit, then fix it. Train a small CNN on 500 CIFAR-10 images with no regularization and plot both loss curves until the gap is unmistakable. Now add, one at a time and re-plotting each time: random crop and flip, then weight decay, then dropout. Record how much of the gap each one closes — the ranking will surprise you, and it is the most useful thing in this list.
2. Sweep the strength. Take the winning setup and sweep weight_decay across [0, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1]. Plot final training and validation accuracy against $\lambda$ on a log axis. You should reproduce the U-shape you found in the lab, with underfitting at the right-hand end.
3. Break an augmentation on purpose. Add RandomHorizontalFlip to a digit classifier and watch accuracy on 2s, 5s and 7s specifically. This is the label-preservation trap in the wild, and seeing it once means you will never ship it.
4. Verify the dropout contract. Take a trained model, run the same input through it 100 times with model.train() and then once with model.eval(). Compare the mean of the 100 stochastic outputs to the single deterministic one, and confirm the $1/(1-p)$ scaling is doing what the code cell above claims.
5. Read the source. Skim §1 and the practical guide of the dropout paper and find where they justify $p = 0.5$ for hidden units but a much lower rate for inputs. Ask yourself why the asymmetry exists before reading their answer.

Next: see the other half of what makes deep networks trainable — Normalization and Initialization — or revisit where the shrinkage actually happens inside the optimizer in Optimization.