ConvNeXt

The Problem

Following the introduction of the Vision Transformer (ViT) and Swin Transformer, the computer vision community largely concluded that Transformers were superior to Convolutional Neural Networks (CNNs). However, researchers suspected the performance gap was not solely due to the attention mechanism, but rather because Transformers benefited from advanced training techniques and modernized architectural micro-designs that standard CNNs (like ResNet, designed in 2015) lacked.

The Idea

The authors took a standard ResNet-50 and systematically modernized it step-by-step to see if they could reach the performance of a Swin Transformer. They adopted training recipes and architectural choices from Transformers (like patching, inverted bottlenecks, larger kernel sizes, and fewer activation functions). The resulting purely convolutional network, dubbed ConvNeXt, matched or outperformed Swin Transformers while remaining simpler and more efficient.

How It Works

The modernization steps from ResNet to ConvNeXt included:

1. Macro Design: Altering the stage compute ratio to match Swin-T, and changing the initial "stem" from a standard convolution to a non-overlapping "patchify" layer.
2. ResNeXtify: Replacing standard convolutions with depthwise convolutions (where filters operate on each channel independently), which mathematically resembles the weighted sum in self-attention.
3. Inverted Bottleneck: Flipping the standard ResNet block so the hidden dimension is expanded by 4x rather than compressed, mimicking the MLP layer in a Transformer block.
4. Large Kernel Sizes: Increasing the depthwise convolution kernel from 3x3 to 7x7 to increase the receptive field, similar to Swin's local windows.
5. Micro Design: Replacing ReLU with GELU, reducing the number of activation functions and normalization layers, and swapping Batch Normalization (BN) for Layer Normalization (LN).

Why It Mattered

ConvNeXt served as a powerful corrective to the "Transformers are all you need" hype in computer vision. It proved that the inductive biases of CNNs (translation invariance, local connections) were still highly valuable, and that standard convolutions could remain the backbone of state-of-the-art vision systems without the complexity of attention mechanisms.

What Came After

ConvNeXt established itself as a robust, highly efficient alternative to Swin Transformers for dense prediction tasks. A V2 was later released incorporating masked autoencoder training (a self-supervised learning technique originally designed for Transformers).