Paper breakdown

Vision Transformer (ViT)

The 2020 paper that proved Transformer architectures could replace CNNs for computer vision by treating image patches as a sequence of words.

Paper: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Authors: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby · 2020

Read the paper

ViT splits an image into fixed-size patches, flattens them, applies linear projections, and feeds them into a standard Transformer encoder just like text tokens.

The Problem

By 2020, Transformers dominated Natural Language Processing, but Convolutional Neural Networks (CNNs) like ResNet still ruled Computer Vision. Attempts to use attention in vision either combined it with CNNs or severely restricted it (e.g., local attention) because calculating global self-attention across every pixel in an image is computationally impossible. Researchers wanted a unified architecture for both text and vision.

The Idea

Google Brain proposed the Vision Transformer (ViT). Instead of processing pixels, they chopped the image into a grid of 16x16 patches. They treated each patch exactly like a "word" in a sentence. By flattening these patches and projecting them into vectors, they could feed them into a standard, unmodified NLP Transformer. With enough data, the Transformer learned to match or beat state-of-the-art CNNs without any image-specific inductive biases (like translation invariance).

How It Works

ViT processing steps:

Patch Extraction: An image (e.g., 224x224) is divided into a grid of non-overlapping patches (e.g., 16x16 pixels). This creates a sequence of 196 patches.
Linear Projection: Each patch is flattened into a 1D vector and passed through a trainable linear layer to create "patch embeddings" (analogous to word embeddings).
Positional Encoding: Because the Transformer has no sense of 2D geometry, learnable 1D positional embeddings are added to the patch embeddings so the model knows where each patch originated.
Transformer Encoder: The sequence (along with a special [CLASS] token) is fed through standard Transformer blocks. The final state of the [CLASS] token is used for image classification.

Why It Mattered

ViT demonstrated that you do not need convolutions to understand images if you have enough data. While CNNs perform better on small datasets (because their structure forces them to look for local patterns), ViT scales better on massive datasets (like JFT-300M). It unified the architecture of vision and language, paving the way for modern multimodal models.

What Came After

ViT became the backbone of almost every modern vision model, from CLIP and stable diffusion to large multimodal LLMs (like GPT-4V and Gemini). Subsequent research addressed ViT's data-hunger, leading to models like Swin Transformer (which reintroduced some hierarchical local structure) and self-supervised training methods like DINO.