PointNet and PointNet++ on Raw Points

Points arrive shuffled with no grid. PointNet reads each point alone then max-pools so order never matters; PointNet++ adds local neighborhoods.

Per-point MLPs max-pool into one global vector while a second path groups local neighborhoods for fine curves.

Why Does This Exist?

Point clouds are sets: shuffling $1024$ points must not change the answer, which rules out plain sequence networks. PointNet is the order-free baseline that point cloud processing builds on: shared per-point MLPs plus one symmetric pool, with a small learned alignment up front. PointNet++ fixes its blindness to local curves by grouping neighborhoods hierarchically before pooling. This page covers both: the baseline mechanism, the hierarchy upgrade, and the boundary where each stops working.

Think of It Like This

Judging two Lego builds dumped in different orders

Dump two identical Lego builds on tables in different orders and judge by counting brick types, not by assembly sequence. Then zoom into the wheels and windows locally before judging the whole castle, because the global count cannot see small curved parts.

Max-pooling is the counting and set abstraction is the zoom. The analogy stops at rotation: brick counts ignore pose entirely, while points need alignment, which PointNet handles with a small learned spatial transformer before any features are read.

How It Actually Works

PointNet maps each point $p_i$ through a shared $h(p_i)$ into a wide feature such as $1024$ channels, then $g = \max_i h(p_i)$ pools over the set, where max ignores order so shuffling inputs cannot change $g$ . A mini spatial transformer predicts a $3 \times 3$ alignment matrix from the cloud itself and applies it to the raw points, giving rough robustness to rotation. PointNet++ adds set abstraction layers: sample spread-out centroids, gather ball neighborhoods around each, run a mini-PointNet inside every ball, and repeat with fewer centroids and wider balls, so curves resolve at multiple scales before the final global pool.

Worked numbers

Four points $(0,0,0)$ , $(1,0,0)$ , $(0,1,0)$ , $(0,0,1)$ through $h(p) = [x + y, z]$ give $[0,0]$ , $[1,0]$ , $[1,0]$ , $[0,1]$ . Max-pooling takes $\max(0,1,1,0) = 1$ and $\max(0,0,0,1) = 1$ , so $g = [1,1]$ regardless of input order. On the ModelNet40 shape benchmark with $1024$ points per shape, reference accuracy moves from about $89$ percent for PointNet to about $91$ percent for PointNet++.

Code

1def pointnet_pool(feats):2    width = len(feats[0])3    return [max(f[d] for f in feats) for d in range(width)]  # symmetric: order-free4
5
6print(pointnet_pool([[0, 0], [1, 0], [1, 0], [0, 1]]))7# -> [1, 1]8print(pointnet_pool([[0, 1], [0, 0], [1, 0], [1, 0]]))9# -> [1, 1]

Watch Out For

Expecting fine local detail from vanilla PointNet

Global pooling keeps one maximum per channel for the whole shape, so small handles and thin legs vanish from the representation. The symptom is strong classification accuracy paired with blobby part segmentations. Use PointNet++ grouping, or a local operator, whenever parts and curves matter.

Trusting the spatial transformer for large rotations

The mini alignment network cleans up small pose jitter, not arbitrary orientation. The symptom is a model trained on upright shapes collapsing on sideways test objects. Augment training with the full rotation range the deployment can meet instead of assuming the transformer covers it.

The Quick Version

PointNet reads raw points with shared MLPs plus max-pooling for order freedom.
Shuffled inputs give identical global vectors by construction.
A learned $3 \times 3$ transform adds rough rotation robustness.
PointNet++ groups neighborhoods hierarchically for local geometry.
Reference ModelNet40 accuracy rises from about $89$ to about $91$ percent with hierarchy.
Use Plus or local operators when parts and curves matter.