Image Retrieval Visual Search

Image retrieval answers queries with pictures, ranking a gallery by embedding distance so the closest match to your photo or sketch floats to the top.

A query photo ranks the gallery by embedding distance and the nearest neighbour wins: d 0.71 beats 2.24, 3.61 and 5.39

Why Does This Exist?

Shoppers photograph a chair and want that chair, tourists photograph a landmark and want its name, and moderators photograph a meme and want its origin. Keywords fail all three because the query is visual. Image retrieval exists to search by pixels: embed the query, rank the gallery by distance, return the neighbours. Metric learning trains the space; hard-negative mining keeps training honest.

Think of It Like This

A librarian who remembers every cover

A librarian who has memorized every book cover takes your torn dust jacket, walks the stacks, and returns with the closest matches armful-first. She compares wholes before details and knows which shelf sections to skip. Retrieval systems shelve gallery embeddings in index structures (inverted files, graph search) that skip the irrelevant millions the same way. The analogy stops at the memory: hers is visual and fuzzy, while the index is geometric and exact to the descriptor.

How It Actually Works

Backbones produce regional descriptors pooled into one compact global vector per image (GeM pooling, NetVLAD aggregation), trained with margin losses so same-object views cluster. At query time the vector searches an approximate nearest-neighbour index, then geometric verification re-ranks the shortlist by matching local features. Recall at K and mean average precision measure whether the true match lands in the returned handful.

Worked example

Place the query at $q = (2, 2)$ and four gallery vectors at $g_1 = (2.5, 2.5)$ , $g_2 = (4, 3)$ , $g_3 = (0, 5)$ , $g_4 = (7, 0)$ . Euclidean distances: $d_1 = \sqrt{0.25 + 0.25} = \sqrt{0.5} = 0.71$ , $d_2 = \sqrt{4 + 1} = \sqrt{5} = 2.24$ , $d_3 = \sqrt{4 + 9} = \sqrt{13} = 3.61$ , $d_4 = \sqrt{25 + 4} = \sqrt{29} = 5.39$ . Ranking nearest-first gives $g_1, g_2, g_3, g_4$ , so with the true match at rank 1, recall at 10 scores 1.0. At web scale the index pulls 100 such candidates from 2 million products in milliseconds, then geometric verification re-ranks the shortlist.

Code

The point of this snippet is the ranking above, recomputed with only the standard library: four Euclidean distances from the query, sorted nearest-first.

1import math2
3q = (2.0, 2.0)4gallery = [(2.5, 2.5), (4.0, 3.0), (0.0, 5.0), (7.0, 0.0)]5
6ranked = sorted(7    (math.dist(q, g), i) for i, g in enumerate(gallery, start=1)8)9for d, i in ranked:10    print(i, round(d, 2))11# -> 1 0.7112# -> 2 2.2413# -> 3 3.6114# -> 4 5.39

Gallery entry 1 wins at 0.71, matching the diagram, and the order never changes no matter how many distractors join the index.

Watch Out For

Landmark models that memorize tourist angles

Training on landmark photos clusters the ten classic viewpoints and fails on night shots and close-ups. The symptom is perfect queries of postcard views with zero recall on user angles. Fix it by mining hard positives (same place, strange view) as aggressively as hard negatives.

Unnormalized vectors let magnitude outvote content

Comparing raw dot products across unnormalized descriptors ranks bright, busy images above quiet ones regardless of subject. The symptom is lifestyle shots swamping exact product matches at the top. Fix: L2-normalize every descriptor at write time and use one distance everywhere, so ranking measures direction, not energy.

The Quick Version

Retrieval embeds the query and ranks the gallery by distance, no classifier involved.
Global descriptors handle scale; local verification re-ranks the shortlist.
Approximate indexes make million-image search interactive.
Recall at K and mAP measure shortlist quality from different angles.
Mine hard positives for viewpoint robustness, not just hard negatives.