ConvNeXt Modernized Convolutions

ConvNeXt rebuilds ResNet with transformer-era training and large depthwise kernels, proving tuned convolutions still match vision transformers on accuracy and speed.

A 7x7 depthwise inverted block costs only 4,704 weights at width 96, lifting ConvNeXt Tiny to 82.1 percent top 1 past Swin

Why Does This Exist?

Vision transformers overtook CNNs around 2021, and the field read that as convolutions being obsolete. But the comparisons were unfair: transformers trained 300 epochs with AdamW, Mixup and heavy augmentation while ResNet baselines used 90-epoch SGD recipes from 2015. Nobody knew how much of the gap was architecture and how much was maintenance.

ConvNeXt (Liu et al., 2022) modernizes the ResNet-50 backbone step by step toward Swin Transformer accuracy while staying fully convolutional. Patchify stem, depth ratios, large kernels, inverted bottlenecks and transformer training close the gap. This page covers those changes. Residual foundations live in residual networks, separable large kernels in MobileNet, and the compound-scaled CNN baseline in EfficientNet.

The one idea you need first

A depthwise convolution gives each input channel its own spatial filter instead of mixing channels: a $7 \times 7$ depthwise layer at width $C$ costs $7 \times 7 \times C$ weights rather than $7 \times 7 \times C^2$ . That factor-of- $C$ saving is what makes large kernels affordable, and channel mixing is deferred to cheap $1 \times 1$ layers.

Think of It Like This

Restoring a classic car with modern parts

A 1990s sports car loses to modern rivals on old tires, old fuel and a worn suspension. Refit tires, fuel mapping, suspension and gearbox one part at a time and the classic chassis matches the newcomers, because the gap was maintenance rather than shape.

ResNet is the chassis. The 300-epoch recipe, AdamW, LayerNorm and $7 \times 7$ depthwise kernels are the refit. Where the analogy stops: some parts change the chassis itself, such as the patchify stem replacing the $7 \times 7$ stride-2 opener, and unlike a car, every change here was ablated so its individual contribution is measured.

How It Actually Works

Changes apply in order. The stem becomes a $4 \times 4$ stride-4 patchify layer. Stage compute ratio moves from $(3, 4, 6, 3)$ to $(3, 3, 9, 3)$ , matching Swin. Each block inverts: a $7 \times 7$ depthwise convolution first (only $7 \times 7 \times C$ weights, so $4{,}704$ at $C = 96$ ), then LayerNorm, then $1 \times 1$ expand $4\times$ with GELU, then $1 \times 1$ project back. Downsampling uses $2 \times 2$ stride-2 convolutions between stages.

Training carries half the gain: 300 epochs, AdamW, Mixup, CutMix, RandAugment and stochastic depth. ConvNeXt-Tiny reaches about 82.1% top-1 with 29 million parameters and 4.5 GFLOPs, against Swin-Tiny at 81.3% with 28 million and 4.5 GFLOPs. Same budget, convolution ahead.

Worked example

Cost the headline kernel directly: $7 \times 7 \times 96 = 49 \times 96 = 4{,}704$ weights. A standard $7 \times 7$ convolution at the same width would cost $7 \times 7 \times 96 \times 96 = 451{,}584$ — the depthwise form is $96\times$ cheaper, which is exactly why a kernel that was prohibitive in 2015 is trivial here. The accuracy margin is $82.1 - 81.3 = 0.8$ points on an equal compute budget.

Code

The two numbers the diagram is built from:

1depthwise_7x7_c96 = 7 * 7 * 962print(depthwise_7x7_c96)3# -> 47044
5swin_t, convnext_t = 81.3, 82.16print(round(convnext_t - swin_t, 1))7# -> 0.8

A $7 \times 7$ depthwise layer at width 96 costs fewer weights than a single $69 \times 69$ dense matrix row — large kernels are cheap once channels are not mixed.

Watch Out For

Using large kernels without the training recipe

A $7 \times 7$ depthwise kernel under a 90-epoch SGD schedule gains nothing and can lose accuracy. The symptom is a slower model scored as a failed idea, when the missing ingredient was the optimizer and schedule, not the kernel.

Adopt the 300-epoch AdamW recipe with Mixup and stochastic depth before judging the architecture. Ablate training and structure separately.

Keeping BatchNorm inside the depthwise block

BatchNorm on small per-device batches destabilizes the wide depthwise maps. The symptom is noisy validation curves that worsen as the job scales across nodes.

Use LayerNorm after the depthwise step as the paper does, especially on multi-node runs where per-device batch statistics are thinnest.

The Quick Version

ConvNeXt modernizes ResNet with a patchify stem, $(3, 3, 9, 3)$ stages and $7 \times 7$ depthwise inverted blocks.
A $7 \times 7$ depthwise layer at 96 channels costs only 4,704 weights, so large kernels stay cheap.
Transformer-era training supplies half the gain, so 90-epoch schedules underrate the model.
ConvNeXt-Tiny reaches about 82.1% top-1 against Swin-Tiny at 81.3% on equal budgets.
Use LayerNorm in the blocks and the full 300-epoch recipe before comparing.