ByteTrack Low-Score Matching

ByteTrack keeps the low-confidence boxes other trackers trash, matching strong ones first and rescuing occluded targets with the leftovers.

ByteTrack matches high-score boxes first, then rescues an unmatched track by pairing it with a 0.35-score box overlapping its prediction at IoU 0.55

Why Does This Exist?

Every detector emits shaky low-score boxes under occlusion, and every classic tracker deletes them before association starts. Those deleted boxes are exactly the occluded people worth keeping: a $0.3$ -score box overlapping a coasting track's prediction is evidence, not garbage. ByteTrack exists on that single observation. Its two-stage matching cut identity switches dramatically with no appearance model at all, just motion plus careful bookkeeping.

The one idea you need first. Multi-object tracking keeps one identity per person across frames by predicting each track forward and matching predictions to new detections by overlap. If tracks, predictions, and identity switches are unfamiliar, read that page first.

This page covers the two-stage association: the split, the two matches, and births and deaths. Appearance-based re-identification across cameras is a different mechanism on a different page.

Think of It Like This

A lost-and-found with two shelves

A lost-and-found puts crisp labelled items on the front shelf and crumpled unknowns on the back shelf. Owners first claim from the front; only the still-missing browse the back, where a bent umbrella is obviously theirs. Other trackers burn the back shelf nightly. ByteTrack keeps it one round: high-score boxes match first, then unmatched tracks shop the low-score leftovers.

The analogy stops at the geometry: claims are settled by box overlap with motion predictions, not by owners recognizing their property.

How It Actually Works

Detections split at a score threshold, often $0.6$ , into high and low sets. Stage one matches high-score boxes to tracks with Hungarian assignment on IoU distance, exactly like the SORT baseline. Stage two matches the leftover unmatched tracks against low-score boxes, recovering the occluded: a $0.35$ box that overlaps a coasting prediction clears the second-stage gate and updates the track instead of letting it die. Unmatched tracks coast on prediction briefly before dying; unmatched high-score boxes seed births. Low-score boxes can never start tracks, which is what keeps clutter from minting identities. No embedding network runs anywhere, so the whole method inherits the detector's speed.

Worked example

A track coasts behind a kiosk while the detector emits a $0.35$ box overlapping the Kalman prediction at IoU $0.55$ . Stage one ignores the box because $0.35$ sits below the $0.6$ split, leaving the track unmatched. Stage two pairs them since $0.55$ clears the second-stage gate: the track updates instead of dying, and the identity-switch counter never moves. A threshold-only tracker deletes the box, kills the track two frames later, and mints a fresh ID on reappearance, scoring one switch plus one fragmentation.

Code

The point of this snippet is the gate arithmetic: overlap decides the rescue, and the score decides which stage runs.

1def iou_xywh(a: tuple[float, float, float, float], b: tuple[float, float, float, float]) -> float:2    """IoU of two boxes in (x, y, w, h) format."""3    ax2, ay2 = a[0] + a[2], a[1] + a[3]4    bx2, by2 = b[0] + b[2], b[1] + b[3]5    inter: float = max(0.0, min(ax2, bx2) - max(a[0], b[0])) * max(6        0.0, min(ay2, by2) - max(a[1], b[1])7    )8    union: float = a[2] * a[3] + b[2] * b[3] - inter9    return inter / union10
11
12predicted: tuple[float, float, float, float] = (100.0, 100.0, 50.0, 80.0)13detected: tuple[float, float, float, float] = (105.0, 102.0, 50.0, 80.0)14score: float = 0.3515overlap: float = iou_xywh(predicted, detected)16print(round(overlap, 4))  # -> 0.781717print(score < 0.6 and overlap > 0.5)  # -> True

The overlap prints $0.78$ , above the $0.5$ gate, while the score $0.35$ sits below the $0.6$ split: stage one skips it, stage two rescues the track.

Watch Out For

Low-score clutter in static crowds

In dense static crowds, background boxes pile up at low scores and stage two glues tracks to clutter. The symptom is drifting boxes in crowds despite clean tracking in open scenes. Raise the low-score floor with scene density and shorten the coast window where clutter is thick.

The Quick Version

ByteTrack splits detections into high and low score at a threshold such as $0.6$ , then matches in two stages.
Stage one matches high-score boxes to tracks; stage two pairs leftover tracks with low-score boxes.
Low-score boxes rescue occluded tracks instead of dying as noise, cutting identity switches.
No appearance model runs anywhere: motion plus careful bookkeeping is the whole method.
Births come from high scores only and deaths wait out a coast window.
Tune both thresholds per scene density, or crowds glue tracks to clutter.