GrabCut Cuts Objects Out From One Box

Draw one loose box around the object and GrabCut learns what foreground and background look like, then refines the border itself by repeating graph cut with color models.

One loose box seeds color models that retrain and re-cut in a loop until the border settles.

Why Does This Exist?

Graph cut finds the optimal binary labeling but demands per-pixel evidence: somebody must say which pixels are foreground and which are background. Painting exact seeds is slow, expert work. GrabCut (Rother, Kolmogorov, and Blake, 2004) asks for one loose rectangle instead and manufactures the rest itself, which made it the default photo cutout tool in editors for a decade.

The prerequisite is the graph-cut machinery: t-links, n-links, and min-cut optimality. This page covers the color models, the tri-map, and the iteration loop. Fully automatic multi-class labeling belongs to semantic segmentation instead, and prompt-driven cutouts belong to Segment Anything.

Think of It Like This

A bouncer with an improving description

A bouncer gets one rough instruction, "everyone inside the rope line might belong, everyone outside definitely does not," plus a description of the guest of honor that sharpens every round. Each round the bouncer re-sorts the crowd, studies the two groups afresh, and redraws the line. After a few rounds the line sits exactly on the guest list.

Where it stops: when the guest wears the same uniform as the crowd, no description separates them. Camouflage in vision terms defeats color evidence entirely, which is why GrabCut accepts corrective strokes as new sure labels.

How It Actually Works

The tri-map from one box

Everything outside the user box is marked sure background. Everything inside starts as unknown, tentatively foreground. That three-way labeling (sure background, probable foreground, probable background after the first cut) is the tri-map, and it is the only human input the algorithm ever sees unless corrections arrive later.

Color models both sides learn

Foreground and background colors are each modeled as a Gaussian mixture with five components in RGB space. Unknown pixels get assigned to their most likely component, the mixtures refit to their assigned pixels, and a graph cut re-labels every pixel using the fresh mixtures as its data term. Assign, learn, cut, repeat, usually about five rounds, until labels stop changing. The smoothness term keeps borders clean while the learned colors pull interiors, and a final border-matting pass estimates partial alpha along the edge for hair-soft transitions.

Worked example

Take a $100 \times 100$ photo with a $60 \times 60$ user box. Total pixels: $100 \cdot 100 = 10{,}000$ . Unknown inside the box: $60 \cdot 60 = 3{,}600$ . Sure background outside: $10{,}000 - 3{,}600 = 6{,}400$ . A typical run settles with roughly $2{,}100$ of the unknown labeled foreground, leaving $1{,}500$ reclaimed as background, the mixtures having learned skin and shirt tones on one side and grass tones on the other. The exact split depends on the image; the budget arithmetic does not.

One assign step by hand: suppose the foreground mixture currently centers on brightness $190$ and the background on $30$ . Unknown pixel $175$ sits $|175 - 190| = 15$ from foreground but $|175 - 30| = 145$ from background, so it joins foreground. Unknown pixel $60$ sits $|60 - 30| = 30$ from background against $|60 - 190| = 130$ from foreground, so it joins background. Refitting then moves each mixture toward its new members, and the next cut uses those sharper models.

Code

Pure Python, no imports: the tri-map budget from the worked example plus one nearest-model assignment round on toy brightness values.

1total = 100 * 1002unknown = 60 * 603sure_bg = total - unknown4print(total, unknown, sure_bg)5# -> 10000 3600 64006
7fg_model, bg_model = 190, 308unknowns = [175, 60, 200, 25]9labels = ["FG" if abs(p - fg_model) < abs(p - bg_model) else "BG" for p in unknowns]10print(labels)11# -> ['FG', 'BG', 'FG', 'BG']12
13settled_fg, settled_bg = 2100, 3600 - 210014print(settled_fg, settled_bg)15# -> 2100 1500

The box manufactures $6{,}400$ sure-background seeds from nothing, the toy round sorts $175$ and $200$ to foreground with $60$ and $25$ to background, and the fixture settles at $2{,}100$ foreground against $1{,}500$ reclaimed background.

Watch Out For

Camouflage bleeding across the border

The cut leaks into background regions whose colors match the subject, like a green shirt against foliage, and more iterations make it worse as the mixtures absorb the leak. Color is the only evidence GrabCut learns. Fix it with corrective strokes (sure-foreground inside the subject, sure-background in the leak) and rerun in mask-correction mode.

Losing hair, wires, and thin straps

Fine structures vanish because the smoothness term charges per boundary pixel and thin shapes are nearly all boundary. The energy genuinely prices them out. Fix it by over-protecting thin parts with foreground strokes, or accept the limit and finish sub-pixel strands with border matting or manual alpha work.

The Quick Version

GrabCut needs one bounding box: outside is sure background ( $6{,}400$ pixels in the fixture), inside starts unknown ( $3{,}600$ ).
Five-component Gaussian mixtures learn both sides' colors, then graph cut re-labels with those models as evidence.
Assign, refit, and cut iterate about five times until the border stops moving, settling near $2{,}100$ foreground here.
Border matting finishes the edge with partial alpha for soft transitions like hair.
Camouflaged colors and hair-thin structures are the two classic failures; corrective strokes fix the first.