SAM Cuts Masks From Any Prompt

SAM encodes each image once, then turns points, boxes, or text into masks on demand, so one model segments new objects with no retraining.

SAM pays for one heavy image encoding, then each point or box prompt decodes into three scored masks so ambiguity resolves by predicted quality.

Why Does This Exist?

Every new mask task used to train a new model on a small labeled set: one for cells, one for products, one for satellite roofs, each starved of labels. Hand-labeling a billion masks at that pace would take years, and fixed-class models still fail the first object outside their label list. Semantic segmentation names every pixel but needs retraining per vocabulary; what was missing was a model that segments whatever you point at.

SAM, the Segment Anything Model (2023), splits the job into a heavy image encoding done once and a light prompt decoding done per request. Points, boxes, or text pick the target, and the same weights answer all of them, including objects never seen in training. This page covers the encode-once design, the prompt decoder, and the three-mask ambiguity rule; counting separate objects is instance segmentation, naming them in words is open-vocabulary detection, and the general prompting pattern is promptable segmentation.

Think of It Like This

One printed photo, many tracing overlays

You print one sharp photo a single time, then answer each request by laying a thin tracing sheet over it: circle the dog, box the car, outline the window. Printing is slow and happens once; each tracing is fast and disposable.

The ViT image encoder is the printing step and the prompt decoder is the tracing step. The analogy stops at blur: no tracing sheet can sharpen a blurry print, and likewise weak image features cap every mask a prompt can pull out of them.

How It Actually Works

Encode once with the heavy backbone

A ViT-H image encoder takes a $1024 \times 1024$ input down to a $64 \times 64 \times 256$ feature grid, a stride of 16 in each direction. This pass costs seconds and runs exactly once per image; its output is cached and reused for every later prompt on that image. A separate prompt encoder turns clicks, boxes, and coarse masks into embeddings in the same space, with foreground and background clicks getting distinct learned tokens.

Decode lightly per prompt

A lightweight two-way transformer attends the prompt embeddings to the cached image features and back, then upsamples to full-resolution masks. Because one click is genuinely ambiguous, a click on a wheel could mean the tire, the wheel, or the car, the decoder always outputs 3 masks, whole, part, and subpart, each with a predicted IoU quality score. Keeping the top-scoring mask resolves the ambiguity without asking the user a follow-up question.

Worked example

Say a foreground click lands at normalized position $(0.52, 0.48)$ on a dog. The decoder returns three masks with pixel areas $12{,}400$ , $8{,}200$ , and $3{,}100$ and predicted IoU scores $0.91$ , $0.84$ , and $0.62$ . The $12{,}400$ -pixel mask at $0.91$ is the whole dog, $8{,}200$ is the torso, $3{,}100$ is the head: you keep the first and the ambiguity never reaches the user. The encoding ran once; each additional prompt on the same photo pays only the millisecond-scale decode, so twenty prompts amortize the one heavy pass.

Watch Out For

Prompting thin structures with one foreground point

One click on a wire, a pole, or a cell boundary usually returns the blob around it instead of the thin thing itself. The symptom is a coarse mask with a decent score that still misses the target shape. Add a background click beside the structure or draw a tight box so the decoder has something to push against.

Expecting class names out of a class-free model

SAM returns masks, never labels: it separates the dog from the couch but cannot tell you which is which. The symptom is a pipeline that treats mask order as identity and silently misnames objects. Pair SAM with a classifier or an open-vocabulary detector when the output needs words, and keep SAM in charge of boundaries.

The Quick Version

SAM encodes a $1024 \times 1024$ image once to $64 \times 64 \times 256$ features and reuses them per prompt.
Points, boxes, or text decode through a light transformer in milliseconds each.
Every prompt yields 3 masks plus predicted IoU scores; the top score resolves whole-versus-part ambiguity.
A data engine of 11M photos and 1.1B masks powers transfer to unseen objects.
Thin targets need background clicks or tight boxes, and labels need a separate classifier.