Transfer Learning

Pretraining, fine-tuning, and reusing learned representations — taught from zero. Start with why almost nobody trains a model from scratch, then build up to feature extraction, layer freezing, discriminative learning rates, and the parameter-efficient adapters (LoRA) behind modern fine-tuning.

intermediate#fine-tuning#pretraining

Start here — the shortcut nobody tells beginners about

Here is a secret that surprises almost everyone who finishes their first deep-learning course: in the real world, hardly anybody trains a model from scratch.

You have 400 photos of two kinds of plant disease and you want a classifier. The textbook says a good vision model needs millions of labeled images and weeks of GPU time. You have 400 photos and a laptop. The textbook answer is that you are stuck.

You are not stuck. You download a model somebody else already trained on millions of images, and you spend twenty minutes adjusting it to your 400 photos. It works — often astonishingly well. That move is , and it is the single highest-leverage trick in applied machine learning.

The one-sentence version

Learning is expensive, but most of what a model learns is not about your task — it is about the world in general (what edges look like, what fur looks like, how English sentences work). So let someone else pay for that general part once, then buy it for free and pay only for the small, specific part that is yours.

Think of it like hiring a trained chef instead of teaching someone to cook:

You need someone to make your restaurant's three signature dishes. Option A: hire a person who has never held a knife and teach them everything — knife skills, heat, timing, seasoning, plating — over several years. Option B: hire a chef with ten years of experience and spend one afternoon teaching them your three recipes.

Option B is transfer learning. The chef's knife skills are the : not specific to your menu, but essential to it, and already paid for.

Two more words you will need constantly, and they are just the two halves of that story:

— what the chef's ten years were. Someone else does this. It costs millions of dollars.
— the afternoon spent on your three recipes. This is the part you do.

How to read this page

It starts from first principles and assumes nothing beyond CNNs. Flip the Depth switch at the top for the formal statements and the derivations — they open automatically once you have finished the prerequisites, and nothing is hidden for good.

Why it works — features go from general to specific

Transfer learning would be pure superstition if there were no reason a model trained on cats and airplanes should help with plant disease. There is a reason, and it is worth understanding properly, because it also tells you when transfer will fail.

A deep network learns a ladder of features

The first layers of a trained vision network do not detect cats. They detect edges, corners, and color blobs — the visual alphabet that every image is made of. The next layers combine those into textures and simple shapes. Only near the top do you get things like a cat's face or an airplane wing.

Your plant photos are made of edges and textures too. So the bottom two-thirds of the network is already, accidentally, exactly what you needed.

This is not a hopeful guess. Yosinski and colleagues measured it layer by layer and reported that the first-layer features a network learns look like Gabor filters and color blobs, and that these appear not to be specific to a particular dataset or task but general across many. Somewhere between the bottom and the top, features must cross over from general to specific — and the whole craft of transfer learning is deciding where that line falls for your problem.

Convolution Kernels— interactive, try itOpen in lab →

Input

Filtered

Kernel (editable)

Each output pixel is the weighted sum of its 3×3 neighborhood. Zero-sum kernels (edge, Sobel) are centered at 128.

The operation those early layers are made of. Slide the tiny stencil over the image and watch what it responds to — an edge detector fires on edges no matter whether the picture is a cat, an airplane, or a diseased leaf. That task-blindness is exactly why the bottom of a pretrained network transfers.

Think of it like learning Spanish after you already speak Italian:

Nobody starting Spanish from Italian begins with the alphabet. Most of the machinery — the sounds, the Latin roots, the idea that verbs conjugate by person — carries straight over. What you actually have to learn is the difference. Now try the same trick going from Italian to Japanese: almost nothing carries over, and the head start evaporates. How much transfers depends on how related the two tasks are — a fact we will make practical shortly.

Try to recall

Why does a network trained on ImageNet photos help with medical X-rays, even though it has never seen an X-ray?

Hint: Think about what the earliest layers are actually detecting.

The warm start — see it on a loss landscape

Back in Optimization you pictured training as walking downhill on a foggy hillside. Transfer learning has an unfairly simple description in that picture.

Random init drops you on a random mountainside. Pretraining drops you in the valley.

Training from scratch starts your parameters at random numbers — a random point on the landscape, almost certainly far from anything good. A pretrained checkpoint starts you at a point that is already near the bottom of a broad, good valley, because millions of images pushed it there. Same downhill walk, same learning rate, wildly different starting distance.

Two starts, fourteen identical steps

Take a toy loss with its minimum at $(2, 1)$ :

L(w_1, w_2) = 0.6\,(w_1 - 2)^2 + 3\,(w_2 - 1)^2

Run plain gradient descent with the same learning rate ( $0.12$ ) from two different starting points:

From scratch, starting at $(-5, -3)$ — a random faraway point. Initial loss: $77.4$ .
Pretrained, starting at $(1.1, 0.35)$ — already close. Initial loss: $1.8$ .

After just 3 steps: scratch is at loss $11.6$ , pretrained is at $0.19$ . After all 14 steps: scratch reaches $0.378$ , pretrained reaches $0.006$ — about sixty times lower, from an identical number of steps of an identical algorithm.

Nothing about the optimizer changed. Only where it started.

Same optimizer, same learning rate, two starting points— interactive, drag & zoom

Loading chart…

Contours of the toy loss, with the exact gradient-descent path from each starting point. Both runs use identical steps and an identical learning rate — the only difference is the dot they began on. The pretrained run is essentially finished by step 4; the from-scratch run is still travelling at step 14. Drag to zoom in on the valley.

Run it yourself and change the starting point — the whole lesson is in this one loop:

Python · runs in your browser

What this does: Runs the exact same gradient-descent loop twice on a toy loss whose minimum is at (2, 1) — once from a random faraway start, once from a 'pretrained' start already near the valley. Same learning rate, same number of steps; only the starting point differs. Watch the final losses: the warm start lands roughly sixty times lower. That gap IS transfer learning.

Read the last line of that code again

Moving the "pretrained" start further from the minimum is exactly what happens when the source task is unrelated to yours. Transfer is not magic — it is proximity. The more your task resembles the pretraining task, the closer that starting dot lands to your valley.

The three ways to reuse a pretrained model

You have downloaded a pretrained network. There are three things you can do with it, and they sit on a single spectrum: how much of the borrowed knowledge are you allowed to overwrite?

(also called a linear probe) is the most conservative: you treat the pretrained network as a fixed function that converts an image into a feature vector, and train only a fresh classifier on top. CS231n describes it as removing the last fully-connected layer and treating the rest of the ConvNet as a fixed feature extractor for the new dataset.

is the middle road and the default: you replace the head and keep backpropagating into the body, so the borrowed features can bend toward your task.

Training from scratch is the last resort: throw the checkpoint away and start from random numbers.

Strategy	What is trainable	Data you need	Typical use
Feature extraction	The new head only	Tiny (hundreds)	Small dataset, similar domain
Partial fine-tuning	Head + the top few blocks	Moderate (thousands)	The everyday default
Full fine-tuning	Everything	Large (tens of thousands)	Big dataset, or a distant domain
From scratch	Everything, from random	Huge (millions)	No relevant checkpoint exists

Counting the knobs on a real network

Take ResNet-50, the workhorse pretrained on ImageNet. It has about 25.6 million parameters, and its final layer maps a 2048-number feature vector to the 1000 ImageNet classes.

You want 3 plant-disease classes instead. Here is what each strategy actually asks the optimizer to do:

Replace the head. Delete the $2048 \to 1000$ layer and put in a fresh $2048 \to 3$ layer. Its size is $2048 \times 3$ weights plus $3$ biases = 6,147 parameters.
Feature extraction. Freeze all 25.6M borrowed parameters. Train 6,147. That is roughly 0.02% of the network — which is why it works on 400 photos without overfitting.
Full fine-tuning. Train everything: the original head was $2048 \times 1000 + 1000 \approx 2.05$ M parameters, so the body holds about $25.6\text{M} - 2.05\text{M} \approx 23.5$ M — and all of those move, plus your 6,147 new ones.

The lesson from the arithmetic: the number of things you are asking your 400 photos to determine changes by four orders of magnitude depending on which box you tick. Overfitting is not mysterious — it is that ratio.

Trainable parameters by strategy (ResNet-50, 3 classes)— interactive, drag & zoom

Loading chart…

ResNet-50 has about 25.6M parameters in total; its last stage holds roughly 15M of them. Note the log scale — feature extraction trains about four thousand times fewer knobs than full fine-tuning. Fewer knobs means less capacity to memorize a small dataset, which is precisely the protection you want when you only have a few hundred labeled examples.

Python · runs in your browser

What this does: Counts the trainable parameters for each transfer strategy on a ResNet-50-shaped network, so you can see the four-orders-of-magnitude gap for yourself, and prints how many labeled examples each choice implies you need. The rule of thumb used here — roughly ten examples per trainable parameter for comfort — is deliberately crude; the point is the ratio, not the constant.

Try to recall

You have 400 labeled images and a pretrained ResNet-50. Why is training only the new head safer than training all 25.6 million parameters?

Hint: Count the knobs against the examples.

You replace a pretrained model's 1000-class head with a fresh 3-class head and immediately fine-tune the whole network at the same learning rate you would use for training from scratch. What is most likely to go wrong?

Which strategy should I pick? The four-quadrant rule

Two questions decide it: how much labeled data do you have, and how similar is your data to what the model was pretrained on? CS231n lays the answer out as four cases.

	Similar to pretraining data	Different from pretraining data
Small dataset	Train a linear classifier on the top-layer features. Do not fine-tune — you will overfit.	The hard case. Train a classifier on features from an earlier layer, where features are still generic.
Large dataset	Fine-tune the whole network with confidence; you have enough data to avoid overfitting.	Fine-tune everything, or even train from scratch — but the pretrained initialization usually still helps.

Read the table as one sentence

More data buys you the right to change more weights. Less similarity forces you to reach further down the network for the parts that still apply. Every row and column of that table is one of those two dials.

The awkward quadrant — small and dissimilar — is the one people get wrong. The instinct is to use the top-layer features because they are the "smartest," but those are precisely the ones specialized to a task unlike yours. Cutting lower, where features are still edges and textures, gives you less refined but more relevant features. It feels like a downgrade and usually is not.

Try to recall

You have 800 satellite images of crop fields and an ImageNet-pretrained model. Which quadrant are you in, and what do you do?

Hint: Satellite imagery looks nothing like ImageNet photos of dogs and cars, and 800 is not many.

How much labeled data does this actually save you?

The honest headline number comes from NLP. ULMFiT, the method that made transfer learning standard in text classification, reported that with only 100 labeled examples it matched the performance of training from scratch on 100× more data, while reducing error by 18–24% on the majority of its benchmark datasets. That is the scale of the effect: not a few percent, but two orders of magnitude of labeling budget.

The shape of a transfer-learning data curve— interactive, drag & zoom

Loading chart…

Schematic, not measured: these are illustrative numbers chosen to show the qualitative shape you reliably see in practice, and the exact values depend entirely on the task. Three things in this shape are real and worth memorizing — from-scratch training is hopeless in the low-data regime on the left, frozen features win when data is scarce but plateau early because their capacity is capped, and full fine-tuning starts slightly worse than freezing yet overtakes it once there is enough data to constrain all those weights.

Read that caption before you quote this chart

The curves above are illustrative. The crossing point between the frozen and fine-tuned lines is the only thing you actually need from it, and where it falls for your problem is an empirical question — measure it, do not assume it. When someone shows you a transfer-learning plot with no axis for dataset size, they have hidden the most important variable.

Fine-tuning without wrecking what you borrowed

Fine-tuning has one characteristic failure mode, and it has a name.

is what happens when your fine-tuning is too aggressive: the model adapts to your 400 photos by destroying the features that made it worth downloading. You get a model that is excellent on your training set, mediocre on your test set, and no better than one trained from scratch.

Think of it like rewriting a chef's muscle memory:

Your afternoon of recipe training should teach the chef three dishes. It should not make them forget how to hold a knife. If your "training" is intense enough to overwrite a decade of technique, you have converted an experienced chef into a beginner who knows three recipes — which is worse than what you started with.

Four techniques prevent this, and every one of them is a variation on the same instruction: move the old weights gently and the new weights freely.

1. A much smaller learning rate for the body. The single most important rule. CS231n puts it directly: use a smaller learning rate for the ConvNet weights being fine-tuned than for the newly initialized classifier weights. A common starting point is 10× smaller for the body than for the head.

2. Warm up the head first. Freeze the body for one epoch and train only the head. This gets the random head to somewhere sensible so that when you unfreeze, the gradients arriving at the body are small and informative rather than large and destructive.

3. Gradual unfreezing. Rather than unfreezing everything at once, unfreeze from the top down — last block first, then the one below, and so on. ULMFiT introduced this alongside discriminative fine-tuning as a core part of its recipe.

4. Freeze the normalization statistics. With a small batch size, BatchNorm's running mean and variance get re-estimated from your tiny batches and become noise. Putting normalization layers in evaluation mode during fine-tuning is a standard and frequently decisive fix.

Python · runs in your browser

What this does: Builds a discriminative fine-tuning schedule — one learning rate per layer, each one ξ times smaller as you go down toward the input — and prints the table so you can see how aggressively the earliest layers are slowed. Change xi to 1.0 to recover a single global learning rate and see the whole idea collapse.

Here is the same idea in real PyTorch — the parameter-group pattern you will actually write:

Python · needs a GPU — run on Colab

import torch
from torchvision import models

model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)

# 1. Replace the 1000-class head with one sized for your task.
model.fc = torch.nn.Linear(model.fc.in_features, 3)

# 2. Two parameter groups: the body shuffles, the head sprints.
body = [p for n, p in model.named_parameters() if not n.startswith("fc.")]
opt = torch.optim.AdamW([
    {"params": body,             "lr": 1e-4},   # inherited weights: gentle
    {"params": model.fc.parameters(), "lr": 1e-3},   # fresh weights: free
], weight_decay=0.01)

# 3. Keep BatchNorm statistics frozen — decisive with small batches.
for m in model.modules():
    if isinstance(m, torch.nn.BatchNorm2d):
        m.eval()

# 4. From here it is the ordinary training loop from the Optimization lesson.

Which of these is NOT a standard defence against catastrophic forgetting during fine-tuning?

Parameter-efficient fine-tuning — LoRA and friends

Everything above assumed you can afford to store a full copy of the model per task. For a 25M-parameter ResNet that is fine. For a 175-billion-parameter language model, one fine-tuned copy per customer is not a strategy — it is a data-center invoice.

Do not edit the model. Bolt a small patch onto it.

Keep the entire pretrained model frozen and shared, and train a tiny extra piece alongside it that nudges its behavior. Every task gets its own tiny piece; they all share the one giant frozen model. Swapping tasks becomes swapping a few megabytes instead of a few hundred gigabytes.

Think of it like sticky notes in a reference book:

You need the encyclopedia annotated three different ways for three different readers. You do not print three encyclopedias. You keep one and add three small sets of sticky notes. The book is frozen; the notes are what you write, store, and swap.

The dominant version of this is . Its bet is that the change fine-tuning needs to make to a weight matrix, while the matrix itself is enormous, is simple enough to be captured by a very small one.

Why a low-rank patch is so much cheaper

Take one weight matrix inside a Transformer of size $4096 \times 4096$ . That is $16{,}777{,}216$ numbers — and a large model has hundreds of such matrices.

Full fine-tuning learns a correction for every one of those 16.7M entries. LoRA instead writes the correction as two skinny matrices multiplied together, with an inner size (the rank) of, say, $r = 8$ :

Matrix $B$ is $4096 \times 8$ → $32{,}768$ numbers.
Matrix $A$ is $8 \times 4096$ → $32{,}768$ numbers.
Their product $BA$ is $4096 \times 4096$ — the right shape to add to the original weights.

Total trainable: $65{,}536$ instead of $16{,}777{,}216$ . That is 256× fewer, for this matrix, at rank 8 — and you can go lower.

The catch, and it is a real one: not every possible correction can be written as a product of two rank-8 matrices. LoRA is a bet that the corrections fine-tuning actually needs are simple in this way. Empirically the bet pays off remarkably often.

The headline result from the paper: compared to fine-tuning GPT-3 175B with Adam, LoRA reduces the number of trainable parameters by 10,000× and the GPU memory requirement by 3×.

Python · runs in your browser

What this does: Computes LoRA's parameter savings across a range of ranks for one 4096×4096 weight matrix, showing how the trainable count collapses while the *shape* of the update stays identical. Change d, k, or the ranks list to see how the savings scale — the win grows with the size of the model, which is why LoRA matters most for the largest ones.

Try to recall

Why is LoRA's B matrix initialized to zeros rather than randomly?

Hint: What is the value of the product B·A at the very first training step?

When pretraining does not help

Transfer learning is close to free, which makes it tempting to treat as always correct. It is not, and knowing the failure modes is what separates using it from cargo-culting it.

is the sharpest failure: the borrowed features are not merely unhelpful but actively misleading, and the model spends its budget unlearning them. It shows up most when the source and target domains differ in their basic statistics — natural photographs versus audio spectrograms, or versus raw sensor traces.

The more nuanced finding is that even in the friendly case, pretraining may buy less than you think. He, Girshick and Dollár trained standard detection and segmentation models on COCO from random initialization and got results no worse than the ImageNet-pretrained versions — the only change needed being more training iterations so the random models could converge. Their conclusion was that ImageNet pretraining speeds up convergence early in training, but does not necessarily provide regularization or improve final target-task accuracy, and it held even when using only 10% of the training data.

What that result does and does not say

It does not say transfer learning is useless — it says the benefit is mostly speed, and that speed is worth a great deal when you have finite compute (which you do). It also studied a target dataset with 118,000 labeled images. If you have 400, the low-data left-hand side of the curve is where you live, and there the pretrained start is doing the heavy lifting rather than merely saving time.

The mistakes that bite everyone

Forgetting to replace the head. Loading a 1000-class checkpoint and training on 3 classes without swapping the final layer — a shape error if you are lucky, silent nonsense if you are not.
Preprocessing mismatch. The pretrained model expects the normalization it was trained with (its own channel means and standard deviations, its own input size). Feed it differently-scaled inputs and the borrowed features are being read in the wrong units.
One learning rate for everything. The most common cause of fine-tuning that underperforms plain feature extraction.
BatchNorm in train mode with a batch size of 8. Running statistics get re-estimated from noise. Freeze them.
Leaking the test set through the pretraining data. If your evaluation images were in the pretraining corpus, your number is fiction. Rarer than it sounds for custom data, very real for public benchmarks.
Not trying the linear probe first. It takes ten minutes and gives you the baseline that tells you whether the fancier thing helped.

Where you have already met this

Once you see the pattern, it turns out to be most of modern machine learning:

Every LLM assistant. The three-stage pipeline in How LLMs Are Trained — pretraining, then supervised fine-tuning, then preference tuning — is transfer learning twice over. The base model is the expensive general stage; SFT and RLHF are the cheap specific ones.
Every object detector. Models in Object Detection are conventionally built on a classification , so detection inherits from classification.
Foundation models generally. The phrase names exactly this economics: pretrain once at enormous cost, adapt cheaply many times.
Word and sentence embeddings. Reusing a learned vector space in a new model is the same move at the level of a single layer.

Explain it yourself

Explain to a friend why downloading a model trained on cat photos helps them classify 400 pictures of diseased leaves — using the chef or the Spanish-after-Italian picture, no formulas. Then explain when it would stop helping. If you cannot say clearly which layers transfer and which do not, that is the section to reread.

Recap — the key ideas

Transfer learning reuses a model pretrained on a big general task to solve your smaller specific one. Almost nobody trains from scratch.
It works because features run general to specific: early layers learn edges and textures that belong to no task in particular; late layers learn the source task's categories.
Mechanically it changes only the initialization — a warm start near a good valley, not a new algorithm. Every optimizer tool you know still applies.
Three strategies on one spectrum: feature extraction (freeze everything, train a head), fine-tuning (train the body too, gently), from scratch (last resort). Choose using the data-size × similarity quadrants.
The failure mode is catastrophic forgetting. The defences are all the same idea: small learning rate for the body, warm up the head, unfreeze gradually, freeze normalization statistics.
LoRA freezes the model entirely and learns a small low-rank patch $W = W_0 + BA$ — 10,000× fewer trainable parameters on GPT-3 175B, and foldable back into the original weights at inference.
Pretraining mostly buys convergence speed; on large target datasets, training from scratch can catch up given enough iterations. It buys far more than speed when your dataset is small.

Practice — and how to make it stick

Learn it the way that actually works

• Retrieval practice: before scrolling back, try to name the four quadrants and what each prescribes — pulling it from memory beats recognizing it on the page.
• Spaced repetition: mark this topic complete to add it to your Review queue, resurfacing right before you would forget it.
• Interleaving: mix these with problems from Regularization and Optimization — freezing is regularization, and fine-tuning is a learning-rate problem, so practising them together builds the connection instead of three isolated facts.

By hand: you have 300 labeled ultrasound images and an ImageNet-pretrained ResNet. Name your quadrant, your strategy, and the two hyperparameters you would set differently from training from scratch. Then argue the opposite case.
Baseline first: on any small image dataset, extract frozen features from a pretrained network and fit a plain logistic regression on them. Record the accuracy. This ten-minute number is the bar everything else must clear — and it clears it less often than people expect.
Find the crossing point: train the same task at 100, 500, 2,000 and 10,000 examples, with frozen features and with full fine-tuning. Plot both curves and locate where they cross. You have just measured, for your own problem, the one thing the schematic chart above could not tell you.
Break it deliberately: fine-tune once at a sensible body learning rate and once at 100× that. Watch catastrophic forgetting happen, and confirm the damaged model scores worse than the frozen-feature baseline.
Read like a scientist: skim the abstract of Rethinking ImageNet Pre-training and write one sentence on why its conclusion does not contradict this lesson. (Hint: dataset size.)

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Work out the transfer-learning budget for a real project. Run the starter to see the parameter counts for each strategy, then do the TODOs: (1) change N_LABELED to 400 and decide which strategy the examples-per-parameter ratio supports; (2) add a fourth strategy, LoRA at rank 8 applied to the 20 largest matrices in the body, and compute its trainable count — is it closer to feature extraction or to fine-tuning?

editor

TOTAL = 25_557_032        # ResNet-50 parameters
LAST_STAGE = 14_964_736   # parameters in the final residual stage
FEATURES, CLASSES = 2048, 3
N_LABELED = 20_000        # TODO (1): change this to 400 and re-read the table

head = FEATURES * CLASSES + CLASSES

strategies = {
    "feature extraction": head,
    "fine-tune last stage": head + LAST_STAGE,
    "full fine-tuning": head + TOTAL,
    # TODO (2): add "lora r=8" here.
    # A rank-r patch on a d x k matrix costs r * (d + k) parameters.
    # Assume 20 matrices of size 1024 x 1024 and r = 8, plus the new head.
}

print(f"{N_LABELED:,} labeled examples\n")
print(f"{'strategy':<22} {'trainable':>12}  {'examples/param':>15}")
for name, n in strategies.items():
    print(f"{name:<22} {n:>12,}  {N_LABELED / n:>15.4f}")

print("\nRule of thumb: well under 1 example per trainable parameter means you are")
print("relying entirely on the pretrained weights and your regularization to")
print("keep the model honest. Which rows clear that bar at N_LABELED = 400?")

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Next: see how these borrowed representations get combined and rebuilt in Attention & Transformers, or push on the regularization side of the same coin in Regularization.