SigLIP Sigmoid Pair Loss

Score every image-text pair on its own as match or not, instead of picking one winner per row. Small batches then train fine.

SigLIP scores all 16 pairs of a batch of 4 alone with sigmoid, so the mate loss of 0.13 and the non-mate loss of 0.31 never compete across a row.

Why Does This Exist?

Softmax over the batch ties every score together, so small batches give noisy targets and huge batches cost memory. Training a vision-language model then demands thousands of pairs per step just to keep the denominator stable. SigLIP removes the coupling: each of the $B^2$ pairs becomes its own yes-or-no lesson with label $+1$ for mates and $-1$ otherwise, and image retrieval training works at batch 256 instead of 16384.

This page covers the sigmoid loss only. Dual-encoder matching basics live under multimodal learning, and captioning use is image captioning.

The one idea you need first: the logit is $s = \gamma (v \cdot t) + b$ , where $v$ and $t$ are unit image and text vectors, $\gamma$ is a learned scale, and $b$ is a bias starting near $-10$ . The loss per pair is $\log(1 + \exp(-y s))$ with $y$ in $\{+1, -1\}$ . The bias starts very negative because almost every pair in the square does not match.

Think of It Like This

Stamping each ticket instead of picking one winner

A raffle judge usually picks one winner per row of tickets, so a good ticket can lose to a great neighbor. SigLIP instead stamps every ticket yes or no alone: a deserving ticket passes on its own merits, and a strong row never suppresses its neighbors.

Softmax is the one-winner rule, sigmoid is the stamp, and the bias of $-10$ is strict starting ink that assumes no until evidence says yes. Where it stops: raffle tickets are independent, while image pairs still share encoders, so lessons leak across pairs through the weights.

How It Actually Works

Encode images $v_i$ and texts $t_j$ to unit vectors, where $i$ and $j$ run over the batch and $B$ is the batch size. Score all $B^2$ pairs with the shared scale and bias, then pay binary logistic loss on each pair alone. Mates with $y = +1$ pull their score up, non-mates with $y = -1$ push theirs down, and nothing is normalized across $j$ . In plain words, $B^2$ independent lessons per batch replace $2B$ softmax picks, and the bias absorbs the base rate that nearly every pair is a non-match.

Worked example

Take a mate pair with logit $s = 2.0$ and $y = +1$ . Its loss is $\log(1 + e^{-2}) = \log(1.1353) \approx 0.1269$ . Take a non-mate with $s = -1.0$ and $y = -1$ : its loss is $\log(1 + e^{-1}) = \log(1.3679) \approx 0.3133$ . A batch of 4 holds 4 mates and 12 non-mates, so the mean is $(4 \times 0.1269 + 12 \times 0.3133) / 16 = 4.2668 / 16 \approx 0.2667$ . Raising mate scores and lowering the rest drops each term alone, with no row competition deciding who learns.

Code

1import math2
3
4def siglip_loss(s, y):5    return math.log(1.0 + math.exp(-y * s))6
7
8mate = siglip_loss(2.0, 1)9neg = siglip_loss(-1.0, -1)10print(round(mate, 4), round(neg, 4), round((4 * mate + 12 * neg) / 16, 4))11# -> (0.1269, 0.3133, 0.2667)

Each pair pays its own logistic loss, and the batch mean mixes 4 mates with 12 non-mates.

Watch Out For

Zero bias that floods early training

Starting $b = 0$ scores most negatives as matches and the gradients explode on step one. The symptom is a loss spike before any learning happens. The standard fix is initializing the bias near $-10$ with the log-scale near $\log 10$ , so negatives start quiet and mates earn their scores.

Softmax habits that re-couple the pairs

Normalizing each row with a softmax before the loss reintroduces batch competition and kills the small-batch benefit. The symptom is training that degrades exactly like the old loss at batch 256. Keep the raw per-pair sigmoids and let every pair learn alone.

The Quick Version

SigLIP pays binary sigmoid loss on every image-text pair independently.
Mates carry plus one and all other pairs carry minus one.
No batch softmax means small batches train stably.
Bias near minus ten absorbs the non-match base rate.
Retrieval quality matches softmax runs at a fraction of the memory.