Critical Points and Saddle Points

Flat in every direction is only a candidate. Curvature votes next: all up is a valley, all down is a hill, and a split vote is a mountain pass.

The hyperbolic contours of x squared minus y squared cross at the origin, rising to 1 along x and falling to minus 1 along y, which is the signature of a saddle

Why Does This Exist?

Training hunts minima by walking downhill until the slope vanishes. But vanishing slope reports flatness, not arrival at the bottom. The walk can halt on a valley floor, a hilltop, or a mountain pass, and in high dimensions the pass is by far the most common halt. Treating every flat point as a minimum turns plateaus into false convergences and wastes the run.

This page sorts flat points with curvature in every direction at once, read from the Hessian matrix. It is the multivariable sequel to the single-variable sorting of peaks and valleys. How noisy steps escape saddles belongs to optimization; this page teaches the identification.

The one idea you need first

A critical point, also called a stationary point, is where all partial derivatives vanish at once, so the gradient is the zero vector. That is necessary for any interior peak or valley and sufficient for nothing. Flatness proposes candidates. Curvature disposes them.

Think of It Like This

A mountain pass at night

Stand on a pass between two peaks with a valley ahead and behind. Under your boots the ground feels level in every direction, exactly as a summit or a valley floor would feel. The level feeling cannot tell the three apart.

What distinguishes them is the curvature vote. Step north and south and you climb toward the peaks. Step east and west and you descend into the valleys. Up in some directions and down in others is the pass. All up is a bowl and all down is a dome. Second derivatives cast those votes, one per direction, and the Hessian collects the ballots.

The analogy stops at dimensionality. A pass has one up axis and one down axis. A network has millions of axes, and with that many votes a split is a near certainty, which is why high-dimensional training meets passes far more often than traps.

How It Actually Works

At a critical point the local shape is governed by the Hessian matrix $H$ of second partial derivatives. Its eigenvalues are the curvatures along its principal directions:

All eigenvalues positive: the surface bowls upward in every direction. Local minimum.
All eigenvalues negative: it domes downward everywhere. Local maximum.
Mixed signs: up in some directions, down in others. Saddle point.
Some eigenvalue zero: the test is inconclusive and higher-order terms decide. In one variable, $x^4$ is flat and positive at the origin while $x^3$ is flat and crossing, yet both have zero second derivative there.

In plain words: the gradient going quiet tells you to stop walking, and the Hessian tells you where you stopped. A minimum needs every vote positive while a saddle needs just one negative direction, so with $n$ directions the chance of all $n$ votes coming up positive shrinks fast. In large models essentially every critical point is a saddle, and plateaus are passes being crossed slowly.

Worked example

Take $f(x, y) = x^2 - y^2$ at the origin.

The gradient is $[2x, -2y] = [0, 0]$ , so the origin is a critical point. The Hessian is:

H = \begin{bmatrix} 2 & 0 \\ 0 & -2 \end{bmatrix}

Eigenvalues $2$ and $-2$ read straight off the diagonal. Mixed signs, so this is a saddle. Walk it to feel the vote: along the $x$ axis $f(1, 0) = 1 - 0 = 1$ , uphill. Along the $y$ axis $f(0, 1) = 0 - 1 = -1$ , downhill. Level ground at the origin, rising one way and falling the other, exactly the mountain pass in the diagram.

Contrast $g(x, y) = x^2 + 3y^2$ . Its gradient $[2x, 6y]$ also vanishes at the origin, but its Hessian is diagonal with entries $2$ and $6$ , both positive: a genuine minimum with $g(1, 0) = 1$ and $g(0, 1) = 3$ . Same flatness, opposite verdict. Flatness proposes, curvature disposes.

Watch Out For

Declaring victory at a small gradient norm

A gradient norm near zero with a loss far above expectations is usually a saddle, not a minimum, and stopping there locks in a bad model. The signature is a stalled loss with a small but non-collapsing gradient.

Check curvature cheaply with Hessian-vector products before accepting the point. Prefer momentum plus minibatch noise, which follow negative-curvature directions out, over shrinking the learning rate, which only settles deeper into the pass.

Trusting the test when an eigenvalue is zero

Zero eigenvalues mark flat directions the quadratic approximation cannot see, and the point can be a minimum, a maximum, or a saddle regardless of the other signs. Ridges in overparameterized models produce exact zeros routinely.

When zeros appear, the verdict needs higher-order information or a walk along the flat direction, not a louder restatement of the Hessian. Probe the flat direction directly before classifying.

The Quick Version

A critical point has all partial derivatives zero at once, which is necessary for peaks and valleys and sufficient for nothing.
Hessian eigenvalues vote: all positive is a minimum, all negative a maximum, mixed a saddle, zero inconclusive.
For x squared minus y squared the votes are 2 and minus 2, a saddle you can walk: 1 uphill along x, minus 1 downhill along y.
In high dimensions split votes dominate, so training stalls at saddles far more often than at bad minima.
A small gradient with a bad loss means check curvature, not the learning rate.