Pretraining & Transfer Learning

What you will learn in this lesson:

• The core idea behind transfer learning: reusing representations rather than relearning them
• Why fine-tuning requires far less labeled data than training from scratch
• The freeze-then-fine-tune strategy for adapting a pretrained model safely
• Why this idea underlies essentially every modern large-scale deep learning system

Reusing What's Already Been Learned

Training a large network from scratch, randomly initialized weights and all, requires enough labeled data to teach it every level of representation: low-level edges and textures (for vision) or basic syntax and word relationships (for language), all the way up to task-specific decision boundaries. Transfer learning skips most of that: start from a model already trained on a large, general dataset, where it has already learned broadly useful representations, and adapt those representations to a new, often much smaller, task-specific dataset.

A network trained on millions of general images has already learned, in its early layers, to detect edges, textures, and simple shapes, features that are useful for almost any visual task, not just the one it was originally trained on. Reusing those early layers means a new task doesn't have to relearn "what an edge looks like" from a handful of task-specific examples.

Why Less Data Is Needed

This is the direct, practical payoff: since the general-purpose features are already learned, fine-tuning only has to adapt or specialize a comparatively small part of the network to the new task, rather than learning everything from raw pixels or tokens upward. A medical imaging classifier with only a few thousand labeled scans, far too few to train a deep CNN from scratch, can perform well by fine-tuning a model pretrained on millions of general images, because the bulk of low- and mid-level visual feature extraction doesn't need to be relearned for the new domain.

The Freeze-Then-Fine-Tune Strategy

A common, safe strategy for adapting a pretrained model to a small new dataset: freeze most of the early layers (hold their pretrained weights fixed, with zero gradient updates), replace or add a new output layer (or "head") suited to the new task, and train only that new head, plus optionally a few of the later layers, on the new data. This protects the broadly useful early representations from being overwritten by gradient updates computed from a small, potentially noisy dataset.

import numpy as np

class Layer:
    def __init__(self, W, b, frozen=False):
        self.W, self.b = W, b
        self.frozen = frozen

    def forward(self, x):
        return np.maximum(0, self.W @ x + self.b)

    def maybe_update(self, dW, db, lr):
        if self.frozen:
            return  # frozen layers ignore gradient updates entirely
        self.W -= lr * dW
        self.b -= lr * db

np.random.seed(0)
pretrained_layer1 = Layer(np.random.randn(8, 4) * 0.3, np.zeros(8), frozen=True)
pretrained_layer2 = Layer(np.random.randn(8, 8) * 0.3, np.zeros(8), frozen=True)
new_task_head = Layer(np.random.randn(3, 8) * 0.3, np.zeros(3), frozen=False)

# Simulated gradients computed for one training step (values don't matter here)
fake_dW, fake_db = np.ones_like(pretrained_layer1.W), np.ones_like(pretrained_layer1.b)

W_before = pretrained_layer1.W.copy()
pretrained_layer1.maybe_update(fake_dW, fake_db, lr=0.1)
print("Frozen layer's weights changed:", not np.allclose(W_before, pretrained_layer1.W))

W_before = new_task_head.W.copy()
new_task_head.maybe_update(np.ones_like(new_task_head.W), np.ones_like(new_task_head.b), lr=0.1)
print("Trainable head's weights changed:", not np.allclose(W_before, new_task_head.W))

In real frameworks, this freezing is usually implemented by simply excluding a layer's parameters from the optimizer, but the effect is identical: gradients can still be computed through the frozen layer (needed for the chain rule to reach the layers before it, if any are still trainable), but its own weights never change.

A common refinement: start with everything but the head frozen, train for a while, then gradually "unfreeze" some of the later pretrained layers and continue training with a small learning rate, letting them adapt slightly to the new task without the large, destabilizing updates a full learning rate might cause this late in a model's training.

Why Transfer Learning Underlies Modern Deep Learning

Almost every large, well-known deep learning system you'll encounter outside this course (large language models, modern image classifiers, speech systems) is built on exactly this two-stage pattern: an expensive pretraining phase on a large, general dataset, followed by a comparatively cheap fine-tuning phase on a smaller, task-specific dataset. Understanding transfer learning is understanding how nearly every practical deep learning system you'll actually use or build gets built, rather than every team training every model entirely from scratch.