Background Subtraction

Background subtraction keeps a running portrait of the empty scene and flags any pixel that stops matching it, turning a fixed camera into a motion sensor.

A background pixel of 100 barely moves to 104 when a 180-bright intruder arrives, so the difference of 76 clears the threshold of 25 and becomes foreground

Why Does This Exist?

A security camera watches an empty hallway 99 percent of the night. Running a full detector on every frame wastes power rediscovering emptiness, and optical flow measures motion without saying what moved. What you actually want is a binary answer per pixel: did something new arrive?

Background subtraction maintains a model of the scene at rest and compares each frame against it. Pixels that disagree enough become foreground, and connected foreground blobs become detections for classical object tracking. It is the cheapest detector that exists when the camera does not move, which is why it still runs inside traffic counters and wildlife cameras.

The one idea you need first. A running average blends the old estimate with each new frame: $B \leftarrow (1 - \alpha) B + \alpha I$ , with a small learning rate like $\alpha = 0.05$ . The model absorbs slow change such as drifting daylight and ignores sudden change such as a person, which is exactly the split a motion sensor needs.

Think of It Like This

A museum guard's mental photo

A guard memorizes the gallery at opening: painting here, bench there, empty floor. All day she compares the live room against that memory. A visitor standing before a painting pops out instantly, not because she analyzed the visitor but because the visitor matches nothing in her memory.

Each part maps. The memorized gallery is the background model, the comparison is the per-pixel difference, her judgment about what counts as different enough is the threshold, and the afternoon light shifting through the windows is the slow drift the running average absorbs. A moved bench fools her exactly once, until her memory absorbs the new arrangement.

Where it stops: the guard understands what she sees while the model only counts disagreeing pixels. She knows a shadow is not a visitor; the model needs extra machinery to learn the same.

How It Actually Works

1. Model the background per pixel

The simplest model is the running average $B \leftarrow (1 - \alpha) B + \alpha I$ , where $B$ is the background estimate, $I$ the current frame, and $\alpha$ the learning rate. Worked: background pixel $100$ , new frame pixel $180$ , $\alpha = 0.05$ . Update: $B = 0.95 \times 100 + 0.05 \times 180 = 95 + 9 = 104$ . Difference $|180 - 104| = 76$ , far above a threshold like $T = 25$ , so the pixel is foreground. The model barely moved, which is the point: background adapts slowly, intruders spike instantly.

2. Threshold into a mask

$|I - B| > T$ marks foreground. One global $T$ is fragile under shadows and auto-exposure, so real systems keep per-pixel variance: a Mixture of Gaussians holds 3 to 5 modes per pixel to survive waving trees and flickering screens, plus shadow suppression that discounts brightness-only changes.

3. Clean the mask

Raw masks are speckled. Morphological operations open the mask to delete dots, close it to fill holes inside objects, then connected components turn blobs into boxes, and a minimum-area filter drops the remaining flicker.

Code

The point of this snippet is the split personality of the update: the background barely moves while the mask fires instantly, on the exact numbers from the worked example.

1import numpy as np2
3bg: np.ndarray = np.full((2, 2), 100.0)4frame: np.ndarray = np.array([[100.0, 180.0], [100.0, 102.0]])5alpha: float = 0.056thresh: float = 25.07
8bg = (1 - alpha) * bg + alpha * frame  # background adapts slowly9mask: np.ndarray = np.abs(frame - bg) > thresh  # intruders spike instantly10print(bg[0, 1], mask[0, 1])  # -> 104.0 True11print(bg[1, 1], mask[1, 1])  # -> 100.1 False

The 180 pixel drags its background estimate only to 104.0 yet trips the mask, while the 102 pixel is absorbed quietly to 100.1. That asymmetry is the whole mechanism.

Watch Out For

Stationary objects melt into the background

A suitcase left on the platform is foreground today and furniture next week, because the running average absorbs it. The absorption speed is $\alpha$ : fast learning forgets stopped objects quickly. The symptom is abandoned-object alarms that clear themselves if nobody moves the bag.

Fix it with a second slow model or an explicit stationary-object test rather than a tuned $\alpha$ . One rate cannot serve both motion detection and abandonment detection.

Sudden light changes are fake motion

Someone flips a light switch and every pixel disagrees at once: full-frame false foreground. Per-pixel Gaussians recover over seconds, but the alarm already fired. The symptom is bursts of detections with no object in them, correlated with time of day or auto-exposure steps.

Fix it by gating global-change frames on the fraction of pixels that flipped, and by normalizing illumination before differencing where the camera allows it.

The Quick Version

Background subtraction models the empty scene and thresholds per-pixel disagreement into a foreground mask.
The worked pixel moves its estimate from 100 to 104 on a 180 intruder, and the difference of 76 clears the threshold of 25.
Mixture of Gaussians per pixel survives waving trees and flicker that kill single-value models.
Morphology plus connected components turns speckled masks into detection boxes for tracking.
Fixed cameras only: stopped objects absorb at speed alpha and light switches cause full-frame false alarms.