Lagrange Multipliers

The best point on an equality boundary is where your goal's contours just touch it. The multiplier measures how much loosening the boundary would be worth.

The circle x squared plus y squared equals 4.5 just touches the line x plus y equals 3 at (1.5, 1.5), where the two gradients line up and the multiplier reads 3

Why Does This Exist?

Unconstrained optimization is the easy case, and it is rarely the real one. Budgets are fixed, margins must hold, probabilities must sum to one, and weight norms get capped. The moment a rule you cannot break enters, setting the gradient to zero stops working, because the unconstrained peak usually sits outside the allowed region.

Lagrange multipliers handle equality constraints with one geometric insight: at the best allowed point, you cannot improve without leaving the region, which means the goal's contours run tangent to the boundary there. That tangency becomes a system of equations you can solve.

This page covers equalities only: the tangency, the Lagrangian, the worked fixture, the shadow price. Inequalities need the slackness switch in KKT conditions, certified step by step in optimality conditions. The pricing view with primal dual gaps is Lagrangian duality, and the full story with the SVM dual and the kernel trick is constrained optimisation and duality. The gradients-alignment picture underneath is pure gradients geometry.

The one idea you need first

The gradient points in the steepest-ascent direction, perpendicular to the level curves. If the best feasible point were not a tangency, the goal's contour would cut across the constraint, and walking along the constraint in the cutting direction would improve the objective while staying feasible. So at an optimum the contour cannot cut; it must touch, and touching curves share parallel gradients.

Think of It Like This

Splitting a fixed bonus pool between two teams

Two teams share one fixed bonus pool, and joint output grows with each team's share but with diminishing returns. Any split that leaves money undistributed is wasteful, so the winner uses the whole pool. Now consider shifting one dollar from team A to team B: the trade gains output exactly when B's marginal return on that dollar beats A's.

You stop when no shift helps, when the marginal returns are equal. Geometrically the output contours just touch the fixed-pool line. And the multiplier answers the follow-up question: what is one more dollar of pool worth? That price interpretation survives into economics, where multipliers are called shadow prices, and into ML, where a multiplier on a fairness constraint prices fairness in accuracy.

The analogy stops at smoothness. It assumes returns diminish smoothly so equal margins mean a global best. Real objectives kink and flatten, and then tangency proposes candidates without crowning one, which is the first pitfall below.

How It Actually Works

To maximize or minimize $f(x)$ subject to $g(x) = 0$ , build the Lagrangian:

\mathcal{L}(x, \lambda) = f(x) - \lambda\, g(x)

In plain words: the objective minus one penalty term per constraint, where each multiplier $\lambda$ is an unknown price for violating its constraint. Then solve the stationary system: $\nabla_x \mathcal{L} = 0$ alongside $g(x) = 0$ . The first block says $\nabla f = \lambda \nabla g$ , the gradients are parallel, which is the tangency. The second block says the point is feasible. With $n$ variables and $m$ constraints that is $n + m$ equations for $n + m$ unknowns.

Read the multiplier after solving: $\lambda$ is the rate at which the optimal value improves per unit of loosened constraint. If $\lambda = 3$ , one more unit of budget buys about 3 units of objective. That reading is what makes the answer interpretable rather than mechanical.

Worked example

Minimize $f(x, y) = x^2 + y^2$ subject to $x + y = 3$ .

The Lagrangian is $\mathcal{L} = x^2 + y^2 - \lambda(x + y - 3)$ . Stationarity gives $2x - \lambda = 0$ and $2y - \lambda = 0$ , so $x = y = \lambda / 2$ . Feasibility gives $\lambda / 2 + \lambda / 2 = 3$ , so $\lambda = 3$ and the candidate is $(1.5, 1.5)$ with value $1.5^2 + 1.5^2 = 4.5$ .

Confirm it is the minimum, not a maximum or saddle: the objective is convex and the constraint is linear, so any stationary point of the Lagrangian is a global minimum. Concretely, neighbours on the line such as $(1, 2)$ give $1 + 4 = 5 > 4.5$ , and far points grow without bound. So $(1.5, 1.5)$ wins. And the multiplier prices the constraint: raising the sum to $4$ gives $(2, 2)$ with value $8$ , an increase of $3.5$ for one unit, close to the predicted $3$ and exact in the limit, since $f^{\star} = c^2/2$ gives $df^{\star}/dc = c = 3$ at $c = 3$ .

Code

The point is the check: stationarity, feasibility, and the shadow-price confirmation, numerically.

1import numpy as np2
3
4def lagrangian(v: np.ndarray, lam: float) -> float:5    x, y = v  # objective x^2 + y^2 minus price times violation6    return float(x**2 + y**2 - lam * (x + y - 3))7
8
9opt = np.array([1.5, 1.5])10print(round(lagrangian(opt, 3.0), 3))  # -> 4.511grad = np.array([2 * opt[0] - 3.0, 2 * opt[1] - 3.0])  # stationarity residual12print(np.round(grad, 6).tolist())  # -> [0.0, 0.0]13print(round(float(opt.sum()) - 3, 6))  # -> 0.0  feasible14print(round((2.0**2 + 2.0**2) - 4.5, 3))  # -> 3.5  about lam * 1

Residual zero, constraint met, value 4.5, and relaxing the sum to 4 gains 3.5 against the priced 3. Tangency with the receipt.

Watch Out For

Solving the system and skipping the verification

Stationarity finds candidates, and constraints can hide minima, maxima, and saddles among them. The method never labels which is which. On this page convexity did the labelling: a convex objective with a linear constraint makes the candidate the global minimum, checked against the neighbour $(1, 2)$ at value 5. Without convexity there is no such shortcut.

Always classify the candidate: check second-order conditions on the bordered Hessian, verify convexity of the problem, or evaluate every candidate plus the feasible region's endpoints or boundary. An unverified candidate is a guess with algebra.

Applying equality machinery to inequalities

The tangency picture assumes the optimum sits on the constraint. With $x + y \le 3$ the unconstrained peak of a concave objective may sit strictly inside, where the constraint is irrelevant and forcing tangency invents a wrong answer. Equality constraints use this page; inequalities need KKT conditions, which add the complementary-slackness switch that turns idle constraints off.

The Quick Version

Equality constraints are solved by tangency: at the optimum the objective contours touch the feasible region.
The Lagrangian subtracts one priced penalty per constraint, and stationarity plus feasibility give the candidates.
The multiplier is the shadow price: optimal value gained per unit of loosened constraint.
Minimizing $x^2 + y^2$ on a sum of 3 picks $(1.5, 1.5)$ with value 4.5 and multiplier 3.
Convexity plus a linear constraint certifies the candidate; otherwise classify every candidate before trusting it.