SimCLR Contrastive Pretraining

Show the net two random crops of one photo and teach it they match, while crops from other photos do not. Matching without labels builds strong features.

SimCLR pulls two crops of one photo together against seven negatives, and the worked anchor pays loss 0.95 at temperature 0.5.

Why Does This Exist?

Labeling millions of photos costs too much, yet unlabeled photos are free. SimCLR turns each photo into its own lesson: two views of photo $i$ must agree more than views of different photos do. A siamese network supplies the shared encoder and contrastive loss supplies the push-apart rule. Together they learn features that later classify with few labels.

This page covers the SimCLR recipe only: strong crops, the projection head, and NT-Xent over a large batch.

The one idea you need first: an embedding $z$ is a short vector that stands for one view. Cosine similarity $\mathrm{sim}(u, v) = u \cdot v / (\lVert u \rVert \lVert v \rVert)$ says how alike two embeddings are, from $-1$ to $1$ .

Think of It Like This

Two blurred photos of one dog in a lineup

You hold two blurry snapshots of one dog and face a lineup of many dogs. Your job is to point at the pair that came from the same animal. Ears, spots, and posture survive the blur, so you match them and ignore the rest.

SimCLR is that game played millions of times. Crops are the blur, the lineup is the batch, and the prize is a feature reader that knows dogs from cats before it ever sees a label. Where it stops: a real lineup holds one match, while SimCLR treats every other photo as a non-match even when two dogs look alike.

How It Actually Works

From each photo make two random views with cropping, color jitter, and blur. Encode both with a shared backbone $f$ plus a small projection head $g$ to get $z_i$ and $z_j$ , where $f$ is the deep encoder kept for later, $g$ is the throwaway head used only for training, and $z$ is the projected embedding. For a batch of $N$ photos there are $2N$ views. The NT-Xent loss for pair $(i, j)$ is

L_{i,j} = -\log \frac{\exp(\mathrm{sim}(z_i,z_j)/\tau)}{\sum_{k \ne i} \exp(\mathrm{sim}(z_i,z_k)/\tau)}

In plain words, $\tau$ is the temperature that sharpens scores, the top term rewards the true mate, and the bottom sum runs over all $2N - 1$ other views. Small $\tau = 0.5$ makes hard negatives count more.

Worked example

Say $N = 4$ , so 8 views. For one anchor let the mate score 0.92 and seven negatives score $0.10, 0.20, 0.05, 0.30, 0.12, 0.18, 0.25$ . With $\tau = 0.5$ the mate term is $\exp(1.84) = 6.30$ . The negative terms are $\exp(0.2) = 1.22$ , $\exp(0.4) = 1.49$ , $\exp(0.1) = 1.11$ , $\exp(0.6) = 1.82$ , $\exp(0.24) = 1.27$ , $\exp(0.36) = 1.43$ , $\exp(0.5) = 1.65$ , summing to 9.99. The denominator is $6.30 + 9.99 = 16.29$ . The loss is $-\log(6.30 / 16.29) = -\log(0.387) \approx 0.95$ . Only a higher mate similarity or weaker negatives lowers it.

Code

1import math2
3mate, tau = 0.92, 0.54negs = [0.10, 0.20, 0.05, 0.30, 0.12, 0.18, 0.25]5num = math.exp(mate / tau)6den = num + sum(math.exp(n / tau) for n in negs)7print(round(-math.log(num / den), 4))8# -> 0.9506

The mate term of 6.30 beats the negative total of 9.99 only weakly, so the loss stays near 0.95.

Watch Out For

Weak crops that make the task trivial

Small crops with no color change let the net match by average brightness. Accuracy looks fine yet transfer flops. The symptom is high pair accuracy with poor image classification later. Use strong random crops plus color jitter and Gaussian blur together.

Judging the projection head features as final

SimCLR trains $g(z)$ for the loss but downstream uses $f$ before $g$ . Reading off $g$ throws away about two accuracy points. Always freeze $f$ , drop $g$ , and probe or fine-tune from $h = f(x)$ .

The Quick Version

SimCLR learns from two augmented views of one photo treated as a positive pair.
NT-Xent with temperature 0.5 contrasts the mate against all other views in the batch.
Strong crops and a nonlinear projection head carry most of the gain.
Large batches supply the negatives, so small batches need memory tricks.
Use the backbone output, not the projection output, for later tasks.