Mean-Field Approximation

Pretend every hidden variable decides on its own, ignoring the others, so a tangled joint posterior becomes a product of simple pieces you can update one at a time.

A coupled two-variable joint table is replaced by two independent factors, and one coordinate update moves the first factor to 0.71.

Why Does This Exist?

Posteriors couple everything to everything. In a mixture model the assignment of point 7 depends on the means, which depend on all the other assignments. Exact inference sums over every joint configuration, and that sum grows as one term per combination: 2 hidden bits need 4 terms while 100 bits need about $10^{30}$ terms. Nobody sums that.

Expectation-maximization avoids the sum when the exact conditional over the hidden variables is available in closed form. Often it is not. Mean-field variational inference is the fallback with a guarantee: pick the closest distribution inside a tractable family, where closeness is KL divergence, and climb a computable lower bound instead of the true evidence. This page teaches the family and the coordinate climb. The bound itself is background in Bayesian inference.

The one idea you need first

For any distribution $q$ over hidden variables $z$ , the log evidence splits as $\log p(x) = \text{ELBO}(q) + \text{KL}(q \parallel p(z \mid x))$ . In plain words, the ELBO is computable without the true posterior, and pushing it up pushes $q$ toward the posterior. Mean-field is the choice of which $q$ family to push inside.

Think of It Like This

A seating chart decided by independent ballots

A wedding planner must seat 100 guests with tangled friendships. The exact solution considers every seating at once. The mean-field planner does something humbler: each guest independently picks a table distribution, ignoring what any specific other guest does, and responds only to the average behaviour of the rest.

Round by round, guest 1 updates against the current averages of guests 2 through 100, then guest 2 updates, and so on. Nobody ever reasons about a joint seating. The plan converges to tables that are good on average, but it can never represent the statement that Alice and Bob must sit together or both leave. That statement is about correlation, and independent ballots cannot say it.

Where the analogy stops: real mean-field factors are distributions with parameters, and each update is an exact expectation under the other factors, not a vote.

How It Actually Works

Write the hidden variables as $z = (z_1, \ldots, z_M)$ and restrict $q$ to factorize:

q(z) = \prod_{i=1}^{M} q_i(z_i)

In plain words, each hidden piece gets its own distribution and the joint is their product. No correlations survive: under $q$ , learning $z_1$ tells nothing about $z_2$ . That is the whole approximation, and everything good and bad about mean-field flows from it.

Holding all factors except $q_j$ fixed, the ELBO is maximized by:

q_j(z_j) \propto \exp\!\left( \mathbb{E}_{q_{-j}}[\log p(x, z)] \right)

Here $q_{-j}$ means all factors except $j$ . In plain words, set factor $j$ from the average log-joint, averaging over everyone else's current beliefs. Cycling $j = 1, \ldots, M$ is coordinate ascent variational inference, called CAVI. Each step raises the ELBO, so the procedure converges to a local optimum of the bound, exactly as EM climbs to a local optimum of the likelihood.

Worked example

Take two binary variables with joint table $p(0,0) = 0.10$ , $p(0,1) = 0.20$ , $p(1,0) = 0.30$ , and $p(1,1) = 0.40$ . Use $q(z_1, z_2) = \text{Bernoulli}(\mu_1) \text{Bernoulli}(\mu_2)$ starting from $\mu_1 = \mu_2 = 0.50$ .

Fix $\mu_2 = 0.50$ and update factor 1. The optimal log-odds is the averaged log ratio:

\log\frac{\mu_1}{1 - \mu_1} = \mu_2 \log\frac{0.40}{0.20} + (1 - \mu_2)\log\frac{0.30}{0.10}

In plain words, each term compares the two table entries that differ only in $z_1$ , weighted by the current belief about $z_2$ . Substituting $\mu_2 = 0.50$ gives $0.50 \times 0.693 + 0.50 \times 1.099 = 0.896$ , because $\log 2 \approx 0.693$ and $\log 3 \approx 1.099$ . The sigmoid then gives $\mu_1 = 1/(1 + e^{-0.896}) \approx 0.71$ .

One coordinate step moved the first factor from $0.50$ to $0.71$ , leaning toward 1 because the table mass favours $z_1 = 1$ . Continuing to alternate settles near $(0.73, 0.63)$ . The true marginals are $p(z_1 = 1) = 0.30 + 0.40 = 0.70$ and $p(z_2 = 1) = 0.20 + 0.40 = 0.60$ : close on means, while the true correlation between the variables is silently dropped.

Watch Out For

Trusting the reported uncertainty

Mean-field posteriors are systematically overconfident. Because $q$ cannot represent correlation, it picks one mode and reports that mode's width as the whole uncertainty. A mean-field logistic regression can report 99 percent confidence on weights whose true posterior spans a wide correlated ridge.

The symptom is that decisions needing calibrated uncertainty fail while point predictions look fine, which shows up in active learning, bandits, and safety checks. The fix is to read only the means from mean-field and get uncertainty elsewhere: MCMC, a full-rank Gaussian, or at least a structured factorization that keeps the important correlations.

Strong coupling stalls or flips the coordinate updates

When variables are tightly coupled, updating one at a time crawls: each factor's optimum barely moves given the others, so CAVI takes tiny steps along a narrow valley. With symmetric multimodality, meaning two equally good explanations, the initialization decides the mode and the updates never visit the other.

The symptom is that different random starts land on qualitatively different factorizations with similar ELBO values. The fix is to run several starts and keep the best bound, consider damping the updates, and when two variables are nearly locked together, merge them into one joint factor instead of factorizing across the lock.

The Quick Version

Mean-field replaces $p(z \mid x)$ with a product $q(z) = \prod_i q_i(z_i)$ , deleting all correlations by construction.
Each factor updates from the averaged log-joint under the other factors, and cycling them climbs the ELBO to a local optimum.
On the worked table, one update moves $\mu_1$ from $0.50$ to $0.71$ through log-odds $0.896$ , near the true marginal $0.70$ .
Means are usually decent while variances and correlations are not, so expect overconfidence.
Strongly coupled variables need joint factors, multiple starts, or a richer family.