Paper breakdown

CLIP

The 2021 OpenAI paper that aligned text and images in a shared embedding space, unlocking zero-shot classification and the modern generative image era.

Paper: Learning Transferable Visual Models From Natural Language Supervision

Authors: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever · 2021

Read the paper

CLIP is trained using a contrastive objective: it maximizes the cosine similarity between the correct image and text pairs on the diagonal, while minimizing it for all other pairs.

The Problem

Before CLIP, computer vision models were trained to predict a fixed set of predetermined object categories (like the 1,000 classes in ImageNet). This required massive, expensive, hand-labeled datasets. More importantly, if a model trained to recognize "dogs" and "cats" encountered a "hyena," it couldn't classify it because "hyena" wasn't in its fixed vocabulary. Adapting to a new task required gathering a new dataset and retraining the model head.

The Idea

OpenAI proposed CLIP (Contrastive Language-Image Pre-training). Instead of predicting a fixed class label, CLIP predicts which text snippet goes with which image. By scraping 400 million image-text pairs from the internet (removing the need for hand-labeling) and training a model to align them in a shared vector space, CLIP learns a vast, open-vocabulary understanding of visual concepts. At inference time, you can provide CLIP with the names of any classes, and it can perform "zero-shot" classification by checking which class text vector is closest to the image vector.

How It Works

CLIP consists of two encoders: an Image Encoder (like a ResNet or Vision Transformer) and a Text Encoder (a Transformer).

During training, a batch of $N$ image-text pairs is passed through the encoders to produce $N$ image vectors and $N$ text vectors. The model calculates the cosine similarity of every possible combination (an $N \times N$ matrix).

The objective is contrastive loss: maximize the similarity of the $N$ correct pairs (the diagonal of the matrix) and minimize the similarity of the $N^2 - N$ incorrect pairs (everything off the diagonal). This forces the model to learn a representation where "a photo of a dog" and a picture of a dog point to the exact same spot in mathematical space.

Why It Mattered

CLIP fundamentally changed computer vision. It proved that natural language was a vastly superior supervision signal compared to fixed one-hot labels, as it forced the model to learn relationships, attributes, and context. Its zero-shot capabilities matched fully supervised models on many datasets.

What Came After

CLIP became the bedrock of multimodal AI. When DALL-E 2, Midjourney, and Stable Diffusion needed a way to understand user text prompts and generate matching images, they all relied on CLIP embeddings (or variations of them) to guide the generation process. CLIP remains one of the most downloaded models on Hugging Face.