SigLIP

The Problem

CLIP trained using InfoNCE loss (softmax contrastive loss), which normalizes the similarity score of an image-text pair against all other negative pairs in the same batch. This means the loss function requires a global view of the entire batch across all GPUs. As models scaled up to batches of 32,000+, this global normalization created a massive communication bottleneck between GPUs, capping the efficiency of scaling.

The Idea

Google researchers proposed SigLIP. Instead of framing the problem as "which of these 32,000 texts matches this image?" (a multi-class classification problem solved with Softmax), they framed it as 32,000 independent yes/no questions: "does this specific text match this specific image?" (a binary classification problem solved with Sigmoid). This removed the need for global normalization across the batch.

How It Works

In standard CLIP, the denominator of the softmax function requires summing the exponentials of all pairs in the batch. GPUs had to constantly sync this sum.

SigLIP applies a standard binary cross-entropy (sigmoid) loss to every pair independently. A positive pair is pushed towards 1, and a negative pair is pushed towards 0.

Because the loss is calculated independently per pair, no global synchronization is needed. The authors implemented a chunked cross-entropy method where each GPU only needs to see a small chunk of the batch at a time, drastically reducing memory usage.

Why It Mattered

SigLIP solved the scaling bottleneck of multimodal pre-training. It achieved better zero-shot accuracy than CLIP while using far less memory, allowing researchers to scale to batch sizes previously impossible on standard hardware.

What Came After

SigLIP models quickly became the preferred vision encoder for newer large multimodal language models. Specifically, the open-source community largely abandoned OpenAI's original CLIP weights in favor of SigLIP weights for building systems like LLaVA-1.5 and PaliGemma.