Transfer Learning
Why training massive networks from scratch on small datasets fails, and how feature extraction and fine-tuning let you reuse pre-learned structure instead.
If you try to train a 100-million parameter model from scratch on just a few thousand labels, the network memorises the training set instead of generalising. You don't have enough data to constrain that many weights, so it overfits immediately.
The Primitive
A model trained on a massive corpus doesn't learn task-specific rules evenly. Its early layers figure out how to process basic structure—like edges in images or syntax in text. Only the final layers combine those structures to solve the pretraining objective.
Feature Extraction
Because those early features are general, you can reuse them. Feature extraction locks the entire pretrained body so its weights cannot change, throws away the original output head, and trains a small new classifier on top. This is fast, cheap, and very hard to overfit.
Fine-Tuning
If you have more data, feature extraction might leave performance on the table. Fine-tuning unfreezes some (or all) of the pretrained layers and trains them at a very low learning rate. This gently adapts the representations to your specific domain without destroying the structure they already learned.
Where It Breaks
Transfer learning assumes the source and target domains overlap. If you pretrain on English literature and try to fine-tune on radar returns, the early layers' knowledge of grammar is completely useless. This causes negative transfer, and you are forced to start from scratch.
The Quick Version
- Training huge models from scratch on small data guarantees overfitting.
- Early layers in pretrained models contain general structure you can reuse.
- Feature extraction freezes the network and only trains a new head.
- Fine-tuning unfreezes deeper layers at a low rate to adapt features.
- Transfer fails completely if your target domain shares no structure with the pretraining data.