Face Landmarks Pin 68 Points for Alignment

Face landmark models pin a fixed set of points to the eyes, brows, nose and jaw, turning a loose face box into measured geometry that later steps can warp, compare and decorate in one shared coordinate frame.

Sixty-eight landmark points trace the jaw, brows, nose, eyes and mouth so the face box becomes aligned geometry, and six lid points read an eye-aspect ratio of 0.30 against a 0.20 blink line

Why Does This Exist?

A face box says where the face is but not where the eyes point or the mouth opens. An AR filter pasted onto an unaligned box slides off the lips, and a recognizer fed tilted faces must relearn the same identity at every angle. Landmarks fix this by converting the box into geometry: the 68-point convention used by the 300-W benchmark assigns 17 points to the jaw, 10 to the brows, 9 to the nose, 12 to the eyes and 20 to the mouth, because $17 + 10 + 9 + 12 + 20 = 68$ , so every face arrives in the same coordinate frame.

One idea first. A landmark is a predicted $(x, y)$ coordinate pinned to an anatomical fixture such as an eye corner or the nose tip. Error is measured relative to face size rather than raw pixels, since five pixels of drift means nothing on a billboard and everything on a thumbnail. The surrounding pipeline, detection first and recognition after, is covered in face detection and recognition, and the body-keypoint sibling of this task is pose estimation.

Think of It Like This

Tent pegs before raising canvas

Pitching a tent starts with pegging corners and guylines, not draping canvas over grass. You stake the fixed points first, then stretch the canvas over them, and every panel lands where it should because the pegs agree.

Landmarks are the pegs: eye corners and jaw points stake the face shape, then recognition embeddings and AR makeup stretch over them. A shifted peg wrinkles the whole tent, which is why alignment error upstream becomes identity error downstream. Where it stops: pegs mark the surface, not the skull, so extreme expressions move the mouth pegs while identity stays put. Geometry aligns the face; it does not define the person.

How It Actually Works

Classical methods refine points iteratively. The dlib Kazemi-Sullivan regressor from 2014 starts from the mean face shape inside the detection box and applies a cascade of learned updates, each one correcting the residual error of the last. Modern heads skip the cascade and regress all $68 \times 2 = 136$ coordinates directly from backbone features, or predict one heatmap per point and read each peak as a coordinate. Heatmaps cost more memory but localize small fixtures such as eye corners more precisely.

Models are compared with normalized mean error, NME: the mean Euclidean distance between predicted and true points, divided by the inter-ocular distance (the gap between the outer eye corners). Dividing by eye distance makes scores comparable across face sizes. An NME near 3 percent is a solid operating point on 300-W, while anything above 8 percent is visibly off. Sparse 5-point sets (two eye centers, nose tip, two mouth corners) are enough to warp a face to canonical eye positions before recognition, while dense 468-point meshes anchor AR makeup and expression tracking.

Worked example

Put the outer eye corners at $(100, 100)$ and $(150, 102)$ . The inter-ocular distance is $\sqrt{50^2 + 2^2} = \sqrt{2504} \approx 50.04$ , effectively 50. Suppose the predicted nose tip misses truth by $(3, 4)$ pixels: the error is $\sqrt{9 + 16} = 5$ , so NME $= 5 / 50.04 \approx 10.0$ percent, poor. After refinement the miss shrinks to $(1, 1)$ : the error is $\sqrt{2} \approx 1.41$ , so NME $\approx 1.41 / 50.04 \approx 2.8$ percent, good.

The same six lid points drive blink detection through the eye-aspect ratio, EAR: the sum of the two vertical lid distances divided by twice the horizontal eye width. With vertical gaps of 6 and 6 and a width of 20, EAR $= (6 + 6) / (2 \times 20) = 12 / 40 = 0.30$ , an open eye. When EAR sits below about 0.20 for several consecutive frames, the eye counts as closed.

Code

The snippet below recomputes both worked numbers with only the standard library: NME for a nose-tip prediction and EAR for one eye.

1import math2
3
4def nme(pred: tuple, true: tuple, iod: float) -> float:5    """Normalized mean error as a percentage."""6    return math.dist(pred, true) / iod * 1007
8
9def ear(v1: float, v2: float, h: float) -> float:10    """Eye-aspect ratio from two vertical gaps and one width."""11    return (v1 + v2) / (2 * h)12
13
14print(round(nme((103, 104), (100, 100), 50.04), 1))  # -> 10.015print(round(nme((101, 101), (100, 100), 50.04), 1))  # -> 2.816print(round(ear(6, 6, 20), 2))                       # -> 0.3

The first prediction scores 10.0 percent NME, the refined one 2.8 percent, and the eye reads 0.3, open. In production the threshold logic wraps the EAR line: count consecutive frames below 0.20 before declaring a blink, so a single noisy frame cannot trigger it.

Watch Out For

Profile poses breaking 68-point models

A model trained mostly on frontal faces hallucinates the occluded cheek points past about 45 degrees of yaw. The symptom is the jawline sliding off the visible face while the visible eye points stay correct, which looks like a tracking glitch but is really a training-distribution gap. Fix it by routing steep-yaw faces to a pose-specific model or a 3D-aware mesh head that reasons about self-occlusion, instead of trusting frontal points everywhere.

Expression shift poisoning recognition

A wide smile moves the twenty mouth points several pixels while identity does not change. The symptom is expression-sensitive mismatches: the same person verifies on a neutral face and fails while laughing. Fix it by aligning on the stable eye-plus-nose anchors and by preferring expression-robust embeddings downstream, so geometry normalizes pose without smuggling mood into identity.

The Quick Version

Sixty-eight points trace the jaw, brows, nose, eyes and mouth by dataset convention.
NME divides mean point error by eye-corner distance so scores compare across face sizes.
A 5-pixel miss on a 50.04-pixel eye gap scores 10.0 percent NME, while a 1.41-pixel miss scores 2.8 percent.
Eye-aspect ratio turns six lid points into blink detection with a threshold near 0.20.
Profile yaw past 45 degrees and heavy expressions need specialized handling, not default models.