3D Human Pose Lifting

3D human pose lifts flat 2D skeletons into depth-aware joint positions, resolving the missing camera-axis coordinate with motion or body-shape priors.

A flat 2D wrist gains a 300 mm depth axis through lifting, and a 10 mm error on each axis becomes 17.3 mm of joint error.

Why Does This Exist?

A 2D skeleton flattens a punch flying toward the camera into a short jab: the depth axis is gone, and with it every measurement animators, coaches and robots need. 3D pose recovers a position $(x, y, z)$ per joint in camera or root-relative space. The catch is that monocular lifting is ambiguous, since many 3D poses project to one flat skeleton, so methods add evidence: temporal motion over neighbouring frames (VideoPose3D, Pavllo et al., 2019), body-shape priors that force plausible limbs (SMPL mesh recovery, Kanazawa et al., 2018), or multi-view geometry. It builds directly on 2D pose estimation and bottom-up systems like OpenPose.

This page covers the lifting setup, the MPJPE error with a hand-computed joint example, and why single-frame depth stays a guess.

Think of It Like This

Rebuilding a statue from its shadow

A lamp casts a dancer's shadow on a wall, and many arm positions throw the same silhouette. Watching the shadow move over several seconds rules out statues whose elbows would need to teleport, and knowing human bone lengths rules out rubber arms. 3D lifting is shadow plus motion plus anatomy: the 2D skeleton is the shadow, the video history bans teleporting joints, and the body model bans impossible limbs.

The analogy stops at a single still frame. One frozen shadow keeps infinitely many statue candidates, so monocular single-image depth stays a guess no prior fully fixes.

How It Actually Works

The lifting setup

Let $J$ be the joint count (often 17). The input is a flat skeleton $x$ in $\mathbb{R}^{2J}$ and the output is a 3D skeleton $X$ in $\mathbb{R}^{3J}$ , usually root-relative so translation is factored out and only pose shape remains. Two-stage lifting trusts a 2D detector, then regresses depth with temporal convolutions over around 243 frames: motion parallax disambiguates what one frame cannot. Mesh methods instead regress SMPL pose plus shape parameters and read joints off the posed body model. Direct heads classify each joint into depth bins. When only one camera and one frame exist, monocular depth cues are all the lifter has, and error bars grow.

The error metric

Accuracy is MPJPE, the mean per-joint position error in millimetres. With true joints $X_j$ and predictions $\hat{X}_j$ :

$\text{MPJPE} = \frac{1}{J}\sum_{j=1}^{J} \lVert X_j - \hat{X}_j \rVert_2$

PA-MPJPE first aligns prediction to truth with a Procrustes fit (rotation, translation, scale), so it scores pose shape while forgiving global misalignment. Around 50 mm raw MPJPE is a strong monocular score on Human3.6M; aligned numbers read much lower and must never be compared against raw ones.

Worked joint example

A wrist sits at 2D $(100, 200)$ with the root at $(120, 210)$ and an estimated depth of $+300$ mm toward the camera. Lifting outputs root-relative $(x, y, z) = (-20, -10, 300)$ in millimetre units. Suppose each axis errs by $10$ mm. The joint error is $\sqrt{10^2 + 10^2 + 10^2} = \sqrt{300} \approx 17.3$ mm. Averaged over all $J = 17$ joints, this wrist contributes its $17.3$ mm to the MPJPE sum, so one slightly-off limb moves the headline number by about a millimetre.

Code

1import math2
3# MPJPE contribution of one joint with 10 mm error per axis.4print(round(math.sqrt(10 ** 2 + 10 ** 2 + 10 ** 2), 1))5# -> 17.3

Watch Out For

Trusting monocular depth on single frames

Single-image $z$ is underdetermined: many depths project to the same pixels. The symptom is plausible-looking skeletons with wildly wrong reach, discovered only when the animation foot slides through the floor. Use video lifting or multi-view fusion for anything measured, and treat single-frame depth as visualization only.

Comparing MPJPE across alignment protocols

PA-MPJPE after Procrustes alignment reads far lower than raw MPJPE on the same outputs. The symptom is method A looking better only because of friendlier alignment. Compare identical protocols, and report both numbers whenever space allows.

The Quick Version

3D pose adds the depth axis that flat projection destroys, outputting $(x, y, z)$ per joint.
Temporal lifting, SMPL shape priors and multi-view geometry tame the ambiguity.
MPJPE averages joint distances in millimetres; PA-MPJPE aligns first.
A 10 mm error on each axis gives $\sqrt{300} \approx 17.3$ mm of joint error.
Single-frame monocular depth is a guess; video turns it into a measurement.