Fine Tuning Adapts Vision Gently

Fine-tuning first trains a new head on frozen features, then unfreezes the top backbone block with a tiny learning rate so high-level features bend without erasing.

A frozen backbone trains a new head first, then the top block unfreezes with a tiny learning rate and accuracy climbs without erasing early features

Why Does This Exist?

A ResNet-50 trained on ImageNet has seen over a million photos and already knows edges, textures, and object parts. A dataset of 800 factory-defect photos cannot teach that from scratch: a deep network trained on 800 images memorizes them within a few passes and fails on the next batch off the line.

Transfer learning supplies the starting weights and feature extraction trains the new head first. Fine-tuning is the second half of that deal: after the head converges on frozen features, the top backbone block unfreezes with a learning rate around 10 to 100 times smaller than normal, so high-level features bend toward the new domain while generic edges stay intact.

Think of It Like This

Tailoring a rack suit

A rack suit already knows shoulders and lapels. A tailor takes in the waist and shortens the sleeves, small changes near the surface, and hands it back the same day. Nobody unpicks the shoulder seam, the most expensively right part, unless the fit truly demands it.

The frozen backbone is the rack suit and training the head is the waist adjustment, while unfreezing the top block at 1e-5 is letting out the cuffs slightly. Where it stops: a big learning rate unpicks the shoulders and destroys the part that was already right, which is catastrophic forgetting.

How It Actually Works

Phase 1 is a linear probe: freeze every backbone weight and train only the new head, which on ResNet-50 is one layer from 2048 features to the class count. With a few thousand head weights against a few hundred images it converges fast and barely overfits. Phase 2 unfreezes the top block, layer4 in a ResNet, and trains head plus block together with about 1e-3 on the head and 1e-5 on the old weights, while early layers stay frozen on their generic edges.

Worked example

Take 800 labeled images in 2 classes. Train the head alone for 5 to 10 epochs until validation plateaus near 88 percent. Then unfreeze layer4, set the backbone rate to 1e-5 against the head rate of 1e-3, a 100-times reduction, and train 5 more epochs. Accuracy typically rises from 88 percent to about 92 percent as top features reshape around defect textures. Unfreezing all 23 million weights on 800 images gives roughly $23{,}000{,}000 / 800 = 28{,}750$ weights per image, which memorizes instead of generalizing.

Code

The point of this snippet is the schedule arithmetic, not a training loop: pure Python recomputes the rate ratio, the accuracy lift, and the weights-per-image warning.

1print(1e-3 / 1e-5)2# -> 100.03
4print(92 - 88)5# -> 46
7print(round(23_000_000 / 800))8# -> 28750

The backbone learns 100 times slower than the head, the worked schedule gains 4 points from 88 to 92 percent, and full unfreezing leaves 28750 weights per image to memorize.

Watch Out For

A normal learning rate erases the pretraining

The symptom is validation that jumps then collapses below the 88 percent linear-probe baseline in the first epoch after unfreezing. Fix it by keeping backbone rates at 1e-5 or below and confirming the frozen-head baseline first, so forgetting reads apart from a bad head.

Unfreezing everything on a tiny dataset

The symptom is a widening train-validation gap right after unfreezing, with 28750 weights per image memorizing the 800 examples. Fix it by unfreezing one top block at a time and stopping when validation stops improving.

The Quick Version

Phase 1 trains only the new head on frozen features until about 88 percent on 800 images.
Phase 2 unfreezes the top block with 1e-5 against 1e-3 on the head, a 100-times reduction.
The worked schedule rises from 88 percent to 92 percent in 5 further epochs.
Early layers stay frozen because small data cannot improve generic edge detectors.
Full unfreezing gives 28750 weights per image and memorizes, while big rates cause forgetting.