DeepSORT Appearance Tracking

DeepSORT remembers what each person looks like, so tracks survive the short occlusions where pure motion matching hands out new identities.

A reappearing detection overlaps its prediction at IoU 0.11 below the motion gate, but its embedding sits at cosine distance 0.06 inside the 0.2 appearance gate, so the track reclaims its identity

Why Does This Exist?

SORT identifies targets by position alone: predict each box forward with a Kalman filter, match by overlap, mint a new identity for anything unmatched. That works until someone steps behind a pillar. The prediction drifts while they are hidden, the overlap comes back tiny, and SORT hands a returning person a brand-new ID. Every occlusion becomes an identity switch.

People, unlike blips, have persistent looks: jacket color, shape, texture. DeepSORT gives each track a visual memory, a person re-identification embedding fused with motion in the matching cost, so occlusions that break SORT become routine re-findings. The full assignment setting is multi-object tracking; the motion-only baseline is classical object tracking; the embedding craft comes from re-identification.

Think of It Like This

A bouncer who remembers jackets

SORT is a bouncer who checks ticket stubs, meaning positions, only. When the lights dip and stubs smudge, re-entry becomes guesswork. DeepSORT trains the bouncer to also remember jackets: a smudged stub plus the right red jacket re-admits the right person even after a blackout.

The memory is a compact embedding compared by cosine distance, and it ages as jackets change, which is why each track banks recent sightings instead of one old photo. The analogy stops at the gallery: the bouncer keeps vectors, not pictures, and similar jackets still fool it.

How It Actually Works

A small convolutional network pre-trained for person re-identification embeds every detection into a normalized feature vector. Each track banks its recent embeddings rather than trusting a single snapshot. Association runs in cascaded stages: recently seen tracks match first using motion gating plus appearance distance, then older coasting tracks get their chance, which prioritizes the living over the lost. A new detection joins the track whose bank it resembles most, provided the cosine distance clears the appearance gate (commonly $0.2$ ) and the motion gate does not veto it.

A worked re-finding

A track coasts behind a pillar with its Kalman prediction drifting. The reappearing detection overlaps the prediction at IoU $0.11$ , far below the $0.5$ motion gate, so geometry alone would mint a new ID. But the detection embedding matches the track bank at cosine distance $0.06$ , comfortably inside the $0.2$ appearance gate. Appearance overrules geometry, the track reclaims its identity, and the switch counter stays still where SORT would have incremented it.

Code

The check below is the heart of the re-finding: cosine distance between a banked track embedding and a new detection, tested against the appearance gate.

1import math2
3
4def cosine_distance(a: list[float], b: list[float]) -> float:5    dot = sum(x * y for x, y in zip(a, b))6    na = math.sqrt(sum(x * x for x in a))7    nb = math.sqrt(sum(x * x for x in b))8    return 1 - dot / (na * nb)9
10
11banked = [1.0, 0.4, 0.2, 0.1]12detected = [0.75, 0.6, 0.3, 0.25]13d = cosine_distance(banked, detected)14print(round(d, 3))  # -> 0.05715print(d < 0.2)  # -> True

Distance $0.057$ sits inside the $0.2$ gate, so the match is accepted even though the IoU of $0.11$ failed the motion gate. Move the gate to $0.05$ and this true match dies; move it to $0.5$ and strangers start matching.

Watch Out For

Embedding drift across camera changes

Re-identification embeddings trained on one camera network misjudge under new lighting and angles. Symptom: confident wrong matches after a camera swap, with distances that look healthy. Fix it by recalibrating the appearance gate per deployment and fine-tuning the embedding network on local crops.

Matching uniforms swapping identities

When two people dress alike, appearance stops discriminating and the tracker swaps their IDs with full confidence. Symptom: clean tracks through occlusions, but the names come out crossed after people in similar clothes pass each other. Fix it by tightening the appearance gate and giving the motion term more weight in the matching cost for uniformed settings.

The Quick Version

DeepSORT extends SORT with one re-identification embedding per detection, banked per track over recent sightings.
Cascaded matching serves young tracks before old coasting ones, so the living match before the lost.
Appearance rescues matches where drifted motion prediction fails, as when IoU $0.11$ misses the $0.5$ gate but cosine distance $0.06$ clears the $0.2$ gate.
Each track banks recent embeddings instead of trusting a single snapshot.
Embedding gates must be recalibrated for new cameras, lighting, and lookalike crowds.