Weakly Supervised Learning

Weakly supervised learning draws boxes and masks from image-level tags using heatmaps and multi-instance rules instead of hand-drawn outlines.

A 64 by 64 heatmap thresholds at 0.5 to keep 1200 hot pixels, and the tight box around the largest hot cluster localizes the tagged object.

Why Does This Exist?

Drawing boxes around every dog in a million images costs months of annotator time. Image-level tags saying dog or no dog cost almost nothing and already exist in search logs and alt text. The gap between cheap tags and expensive boxes blocks datasets that could be ten times larger.

Weakly supervised learning bridges it with localization from classification. A classifier trained on tags alone still has to look at the right pixels to decide, and class activation maps expose where it looked. Multi-instance rules then turn bag-level tags into pixel decisions. Programmatic labeling with hand-written rules rather than heatmaps lives in weak supervision, and box craft with full labels lives in object detection. This page covers heatmap boxes only.

Think of It Like This

Finding a hot pan by hovering hands

You cannot see which burner is on, but hovering hands feel heat rising strongest over one spot. You mark that hot zone and reach for the handle beside it.

The classifier is the hovering hand, the heatmap is felt warmth, and the threshold picks the hot zone. The analogy stops at precision: hands find heat, not object edges, and heatmaps hug discriminative parts like faces while missing full bodies.

How It Actually Works

Train a classifier on image tags with global average pooling before the final layer, which preserves spatial maps instead of flattening them away. A class activation map, CAM, weights each feature plane by its class weight, $M_c = \sum_k w_k^c F_k$ , where $F_k$ is feature plane $k$ and $w_k^c$ is its weight for class $c$ , then upsamples $M_c$ to image size. Thresholding at $0.5$ keeps the hot pixels, connected components clean the speckle, and the tight box around the largest component becomes the pseudo-box. Multi-instance learning frames it formally: an image is a bag, regions are instances, and a positive bag holds at least one positive instance.

Worked example

Heatmap $64 \times 64 = 4{,}096$ pixels for class dog. Threshold $0.5$ keeps $1{,}200$ hot pixels clustered over the torso and head. The largest connected component holds $640$ of those pixels and spans columns $18$ to $46$ and rows $20$ to $44$ , giving box $[18, 20, 46, 44]$ in heatmap cells. The box area is $29 \times 25 = 725$ cells, comfortably holding its $640$ hot pixels. Scaling by $8$ to a $512$ image gives $[144, 160, 368, 352]$ . One tag became one box with no hand drawing.

Code

Threshold plus tight box on an $8 \times 8$ toy heatmap with a hot block at rows $2$ to $4$ , columns $3$ to $6$ :

1W = 82vals = [0.1] * (W * W)3for r in range(2, 5):4    for c in range(3, 7):5        vals[r * W + c] = 0.96
7hot = [i for i, x in enumerate(vals) if x >= 0.5]8xs = [i % W for i in hot]9ys = [i // W for i in hot]10print(len(hot), (min(xs), min(ys), max(xs), max(ys)))11# -> 12 (3, 2, 6, 4)

$12$ hot cells with tight box $(3, 2, 6, 4)$ , exactly the planted block. The same gate-and-box routine scales to the $64 \times 64$ map in the worked example.

Watch Out For

Boxes that hug faces instead of bodies

Classifiers discriminate by parts: dog faces over torsos, bird heads over wings. The symptom is tight boxes around heads that score well on classification but fail detection IoU on a labeled probe. Expand coverage with erasing or multi-map fusion during training, and judge boxes by IoU on a small fully labeled set, never by classification accuracy alone.

Thresholding speckle into confetti boxes

A single global threshold on a noisy heatmap keeps scattered hot pixels far from the object, and boxing every component sprays false positives. The symptom is many tiny high-confidence boxes around background texture. Keep only the largest connected component, or merge components closer than a few cells, before drawing the box.

The Quick Version

Weak supervision turns image tags into boxes and masks via heatmaps and bag rules.
CAM weights feature planes by class weights, $M_c = \sum_k w_k^c F_k$ , and upsamples to full resolution.
Worked numbers: $1{,}200$ of $4{,}096$ pixels kept at $0.5$ , largest cluster boxed as $[144, 160, 368, 352]$ .
Largest-component cleanup removes speckle before boxing.
Expect part-focus on faces over bodies, and verify against true boxes on a small probe set.