OpenPose Real-Time Skeletons

Why Does This Exist?

Multi-person pose has a grouping problem: finding all elbows in a crowd photo is easy, but knowing whose elbow belongs to whose wrist is hard. Top-down systems crop each person first and run pose per crop, so their cost grows with headcount and they miss people the person-detector drops. A busy plaza frame with twenty people needs twenty network passes.

OpenPose (Cao and colleagues, 2017) made the bottom-up answer practical. One network pass predicts every joint location plus, for each limb, a Part Affinity Field (PAF): a dense 2D arrow map pointing along that limb. Linking becomes a scoring problem over arrows, and crowds assemble in milliseconds after the single pass. The general pose setup lives in pose estimation; lifting these flat skeletons into depth lives in 3D human pose estimation. This page covers heatmaps, affinity fields and greedy assembly.

The one idea you need first. A joint heatmap is an image-sized score sheet with one bright peak per visible instance of that joint. Peaks give candidates; they never say which elbow pairs with which wrist, which is exactly the gap the arrow fields fill.

Think of It Like This

How It Actually Works

Heatmaps for joints, arrow fields for limbs

A convolutional backbone feeds refinement stages that output two things. First, one heatmap $H_j$ per joint type $j$ (the original build predicts 19 sheets covering 18 joints plus background; COCO builds use its own keypoint set), where each pixel holds the confidence that joint $j$ sits there. Candidate joints come from non-maximum suppression over each heatmap. Second, one Part Affinity Field $L_c$ per limb type $c$, where each pixel holds a 2D unit vector pointing from the limb's start joint toward its end joint (19 limbs times 2 direction numbers gives 38 channels in the original build).

Scoring a candidate limb

Take candidate joints $E$ (elbow) and $W$ (wrist) and the unit vector $u$ pointing from $E$ to $W$. Sample $P$ evenly spaced points $p_1$ through $p_P$ along segment $EW$ and average how well the field agrees with the segment:

$E(E, W) = \frac{1}{P} \sum_{k=1}^{P} L_c(p_k) \cdot u$

where $\cdot$ is the dot product, so each sample scores $1$ when the painted arrow points exactly along the segment and near $0$ when it points across it. Each limb type is then matched greedily: sort all candidate pairs by $E$, link the best, remove its joints, and repeat. Low-total-score people are pruned at the end.

Worked assembly numbers

Elbow candidates $E_1, E_2$ and wrist candidates $W_1, W_2$, with $P = 10$ samples each. Segment $E_1W_1$ points along $u = (1, 0)$ and every field sample reads $(0.9, 0.4359)$, a unit arrow whose dot with $u$ is $0.9$; the sum over 10 samples is $9.0$, so $E = 0.90$. Rival segment $E_1W_2$ samples read $(0.2, 0.9798)$, each dotting to $0.2$, so $E = 0.20$. Segment $E_2W_2$ samples read $(0.85, 0.5268)$, each dotting to $0.85$, so $E = 0.85$. Greedy order links $E_1W_1$ at $0.90$ first, removes $E_1$ and $W_1$, then links $E_2W_2$ at $0.85$. The $0.20$ cross-connection never survives, which is the arrows doing the grouping work.

Code

1pairs = [(("E1", "W1"), 0.9), (("E1", "W2"), 0.2), (("E2", "W2"), 0.85)]2print(sorted(pairs, key=lambda p: p[1], reverse=True)[0][0])3# -> ('E1', 'W1')

Watch Out For

The Quick Version

• OpenPose predicts joint heatmaps plus limb-direction arrow fields in a single pass.
• Each candidate limb scores the average arrow alignment along its segment.
• Greedy matching links the best pair first, then removes those joints and repeats.
• Bottom-up cost stays flat as headcount grows, unlike per-person top-down crops.
• Tangled crowds are the accuracy ceiling, where arrows overlap and links swap.

What to Read Next

• Pose Estimation
• 3D Human Pose Lifting