Early CNN Architectures

From handwritten digits in LeNet-5 to GPU-scaled AlexNet and uniform 3x3 stacks in VGG, each milestone proved deeper layers extract richer features.

LeNet-5 reads digits with 60 thousand weights, AlexNet scales to ImageNet with ReLU and dropout, and VGG stacks only 3 by 3 filters for the same visual field at less cost

Why Does This Exist?

Before 2012, vision systems relied on hand-built descriptors such as SIFT and HOG paired with linear classifiers. These pipelines were brittle: shift the lighting, pose, or background clutter and the hand-tuned features discarded the semantic signal. Early convolutional networks proved visual hierarchies could be learned from raw pixels instead, but depth hit three walls at once: saturating activations that killed gradients, GPU memory that capped model size, and dense classifier heads whose weights exploded.

LeNet-5 from 1998, AlexNet from 2012, and VGGNet from 2014 each broke one wall. LeNet-5 set the template of alternating convolutions, subsampling, and dense layers on digits. AlexNet showed ReLU, dropout, augmentation, and GPU training could carry 8 layers across 1.2 million ImageNet images. VGGNet showed uniform stacks of small $3 \times 3$ filters beat large ones on both accuracy and cost. Everything after, including residual networks, builds on these three answers. The operations themselves live in CNN operations.

The one idea you need first

A convolution slides a small learned filter over the image and records matches, producing a feature map. Pooling shrinks that map by keeping one value per neighborhood, usually the maximum. A receptive field is the patch of the original image one deep unit actually sees. Stack more layers and each unit sees more; the question these architectures answered is how to stack without drowning in weights.

Think of It Like This

Telescopes, wide lenses, and micro-lens arrays

Learning to photograph bird species in the wild, LeNet-5 is a simple handheld spyglass: it cleanly resolves black digits on bank checks but cannot pick feathers out of dense branches. AlexNet is a heavy safari camera with oversized interchangeable lenses, an ultra-wide $11 \times 11$ lens to sweep the horizon, a $5 \times 5$ lens for mid-size objects, smaller ones for plumage. It captures 1,000 categories but swapping awkward lenses burns power and money.

VGGNet standardizes the kit on identical micro-lenses. Two small $3 \times 3$ filters in series see exactly what one $5 \times 5$ lens sees, weigh 28 percent less, and leave room for an extra color filter between them. The analogy stops at optimization: real photographers choose lenses per shot, while a network commits to one architecture and trains it for every image at once.

How It Actually Works

The structural evolution

LeNet-5 takes a $32 \times 32$ grayscale digit through two $5 \times 5$ convolutions with average pooling, then dense layers of 120 and 84 units to 10 outputs, about 60,000 weights total. AlexNet takes a $224 \times 224$ color image through an $11 \times 11$ convolution with stride 4, a $5 \times 5$ convolution, three $3 \times 3$ convolutions, and two dense layers of 4,096 units to 1,000 outputs, about 60 million weights. VGG-16 takes the same $224 \times 224$ input through thirteen $3 \times 3$ convolutions in five pooling stages, then the same dense pair, about 138 million weights.

Three AlexNet choices did the heavy lifting. The ReLU $f(x) = \max(0, x)$ keeps a gradient of 1 for every positive input, where $\tanh$ saturates and its gradient dies, which sped training roughly $6\times$ . Dropout with probability $p = 0.5$ on the two 4,096-unit layers stopped the giant dense head memorizing. Overlapping $3 \times 3$ max pooling with stride $S = 2$ cut spatial size without the blur of average pooling.

Why two small filters beat one large filter

Stack two convolutions with kernel size $K = 3$ and stride $S = 1$ . Receptive fields compose by the rule $R_l = R_{l-1} + (K_l - 1)$ at stride 1, so $R_1 = 3$ and $R_2 = 3 + 2 = 5$ : two $3 \times 3$ layers see a $5 \times 5$ field, and three see $7 \times 7$ . With $C$ channels in and out, one $5 \times 5$ layer costs $25C^2$ weights while two $3 \times 3$ layers cost $2 \times 9C^2 = 18C^2$ , a ratio of $18 / 25 = 0.72$ , which is 28 percent fewer weights plus a second ReLU the single layer never had.

Worked example

Take a VGG stage turning 64 channels at $112 \times 112$ into 128 channels with two $3 \times 3$ convolutions with bias, then $2 \times 2$ max pooling at stride 2. Padding of 1 keeps the spatial size at 112 through both convolutions.

Layer one holds $128 \times 64 \times 3 \times 3 = 73{,}728$ filter weights plus 128 biases, which is $73{,}856$ . Layer two holds $128 \times 128 \times 3 \times 3 = 147{,}456$ filter weights plus 128 biases, which is $147{,}584$ . Pooling holds nothing and halves 112 to 56. The stage total is $73{,}856 + 147{,}584 = 221{,}440$ weights, and a batch of 2 leaves as shape $2 \times 128 \times 56 \times 56$ , exactly what the snippet below prints.

Code

The snippet builds that VGG stage and checks the two numbers from the worked example: the output shape and the 221,440 weights.

1import torch2import torch.nn as nn3
4
5class VGGBlock(nn.Module):6    """Two 3x3 convolutions plus pooling: one VGG stage."""7
8    def __init__(self, in_channels: int, out_channels: int) -> None:9        super().__init__()10        self.block = nn.Sequential(11            nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),12            nn.ReLU(inplace=True),13            nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1),14            nn.ReLU(inplace=True),15            nn.MaxPool2d(kernel_size=2, stride=2),16        )17
18    def forward(self, x: torch.Tensor) -> torch.Tensor:19        return self.block(x)20
21
22x = torch.randn(2, 64, 112, 112)23stage = VGGBlock(in_channels=64, out_channels=128)24print(stage(x).shape)25# -> torch.Size([2, 128, 56, 56])26print(sum(p.numel() for p in stage.parameters()))27# -> 221440

Both prints match the hand computation above: the spatial map halves from 112 to 56 while channels double from 64 to 128, and the weight count lands on 221,440.

Watch Out For

Letting the dense head eat the whole memory budget

In AlexNet and VGG-16 the dense classifier holds most of the weights. VGG-16 pools to $512 \times 7 \times 7 = 25{,}088$ values, and the first dense layer alone needs $25{,}088 \times 4096 + 4096 \approx 102.7$ million weights. The symptom is out-of-memory crashes and overfitting on small datasets even though the convolutions look modest. Modern networks replace the giant head with global average pooling, collapsing each map to one value before a single small linear layer.

Copying the split-GPU halves as if they were architecture

AlexNet trained across two GPUs with half the filters on each and sparse cross-talk, a hardware workaround, not a design choice. The symptom in reimplementations is a two-stream network with crosstalk layers that gains nothing on one modern GPU while doubling the code paths. Treat the halves as one wider layer unless you are reproducing the 2012 setup exactly.

The Quick Version

LeNet-5 set the template of alternating convolutions, pooling, and dense layers on digits with about 60,000 weights.
AlexNet scaled that template to ImageNet with ReLU, dropout, augmentation, and GPU training at 60 million weights.
VGGNet proved stacks of $3 \times 3$ filters match larger filters with 28 percent fewer weights and more nonlinearity.
Two $3 \times 3$ layers see a $5 \times 5$ field; three see $7 \times 7$ .
The dense classifier head, not the convolutions, holds most of the weights in early networks.
The worked VGG stage turns 64 channels at 112 wide into 128 at 56 wide with 221,440 weights.