Conditional Expectation

Conditional expectation re-takes the average after learning something. A die averages 3.5, but averages 4 once you know the roll was even.

A die averages 3.5 overall, but inside the known cell of even rolls the re-averaged value E[X | even] equals 4

Why Does This Exist?

Models rarely predict in the dark. They predict after seeing features: the expected price given the neighborhood, the expected click given the history. Plain expectation $\mathbb{E}[X]$ averages over everything; conditional expectation $\mathbb{E}[X \mid \mathcal{G}]$ averages after information $\mathcal{G}$ arrives.

Without it there is no regression theory, no Bayes update, and no filtering. The conditional expectation is the least-squares best predictor given what you know, and every "expected value given features" in ML is one. It extends expectation, variance and covariance to the case where knowledge has arrived.

The one idea you need first

Expectation is a weighted average: for a fair die, $\mathbb{E}[X] = (1+2+3+4+5+6)/6 = 3.5$ . Conditional expectation repeats that average on a smaller set, reweighted to sum to 1 inside it.

Think of It Like This

Re-grading after the class splits

A teacher averages all exam scores: 72. Then the class splits into two rooms and she re-averages inside each room: 68 in one, 78 in the other. Nobody retook the exam; the information changed.

Conditional expectation is the room average. It is still a random object before you know your room, and a plain number after. The tower rule says averaging the room averages, weighted by room size, returns the class average. Where the analogy stops: rooms are fixed, while information in a running system refines continuously as new data streams in.

How It Actually Works

Given information $\mathcal{G}$ (an event, a variable, or a collection of events), $\mathbb{E}[X \mid \mathcal{G}]$ is the variable built from $\mathcal{G}$ alone that matches $X$ on every question $\mathcal{G}$ can ask: $\mathbb{E}[\mathbb{E}[X \mid \mathcal{G}] \cdot 1_A] = \mathbb{E}[X \cdot 1_A]$ for every event $A$ in $\mathcal{G}$ . In plain words: it matches $X$ on everything you can distinguish, and it uses nothing you cannot.

Properties you actually use

Tower: $\mathbb{E}[\mathbb{E}[X \mid \mathcal{G}] \mid \mathcal{H}] = \mathbb{E}[X \mid \mathcal{H}]$ when $\mathcal{H} \subseteq \mathcal{G}$ . Coarsen in steps or all at once.
Taking out what is known: if $Z$ is already determined by $\mathcal{G}$ , $\mathbb{E}[ZX \mid \mathcal{G}] = Z\,\mathbb{E}[X \mid \mathcal{G}]$ .
Best predictor: $\mathbb{E}[X \mid \mathcal{G}]$ minimizes $\mathbb{E}[(X - f)^2]$ over all $f$ built from $\mathcal{G}$ . Regression targets it.

Worked example

Fair die, $X$ the face. Plain mean $3.5$ . Learn the event $E = \{2, 4, 6\}$ ("even"). Re-average inside: $\mathbb{E}[X \mid E] = (2 + 4 + 6)/3 = 12/3 = 4$ . Learn instead the exact face $X = 5$ and the conditional collapses to $5$ . Tower check: $P(E) = 1/2$ , $\mathbb{E}[X \mid E^c] = (1+3+5)/3 = 3$ , and $\tfrac{1}{2} \cdot 4 + \tfrac{1}{2} \cdot 3 = 3.5$ . The cells recombine to the whole.

Code

Exact fractions, no sampling noise: the even cell, the odd cell, and the tower recombination.

1from fractions import Fraction2
3even = [Fraction(2), Fraction(4), Fraction(6)]4odd = [Fraction(1), Fraction(3), Fraction(5)]5mean_even = sum(even) / len(even)6mean_odd = sum(odd) / len(odd)7print(mean_even)  # -> 48print(mean_odd)  # -> 39print((mean_even + mean_odd) / 2)  # -> 7/2

The cells give 4 and 3, and weighting them equally returns 7/2, which is 3.5, the plain mean from the worked example.

Watch Out For

Treating E[X | Y] as a number before Y is known

$\mathbb{E}[X \mid Y]$ is a function of $Y$ , hence random. People plug in $\mathbb{E}[X \mid Y = y]$ for one $y$ and treat the whole object as that number. Then the tower rule looks false and downstream variance math breaks.

Keep the two apart: $\mathbb{E}[X \mid Y]$ varies with $Y$ ; $\mathbb{E}[X \mid Y = y]$ is its value at one point. Take the outer expectation to get back to a number.

Conditioning on zero-probability points with the ratio formula

$P(A \mid B) = P(A \cap B)/P(B)$ needs $P(B) > 0$ . With continuous features, $P(X = x) = 0$ and the ratio is $0/0$ . Filtering code that subsets to exact floats hits this silently.

Condition on intervals or use densities and the measure-theoretic definition. If the subset has fewer than a dozen rows, widen the window before trusting the average.

The Quick Version

Conditional expectation is the average after information arrives: same variable, smaller world.
A fair die averages 3.5 overall and 4 given even, since $(2+4+6)/3 = 4$ .
The tower rule recombines cell averages into the whole average.
Known factors pull out; the conditional expectation is the least-squares best predictor.
Before the information is known it is random; after, it is a number.
Never condition on exact continuous values with the elementary ratio formula.