Importance Sampling: Ordinary vs Weighted

Both correct for sampling from the wrong policy with probability ratios. Ordinary divides by the episode count and can explode, while weighted divides by the total ratio weight and stays inside the observed payoffs.

On three episodes with weights 0.10, 0.40, and 14.50, ordinary sampling spikes to 4.87 while weighted sampling lands at 0.97

Why Does This Exist?

Off-policy prediction lets you score a target policy $\pi$ using episodes collected by a different behavior policy $b$ . Each episode's return $G_i$ gets reweighted by its likelihood ratio $\rho_i$ , the product of $\pi(a \mid s) / b(a \mid s)$ along the trajectory. The question is what you divide by.

Ordinary importance sampling divides by the episode count $n$ . That keeps the estimate unbiased, but one rare episode with a giant ratio can drag the whole average outside anything the environment ever paid. Weighted importance sampling divides by the sum of the ratios instead. That adds a small finite-sample bias, but the estimate becomes a weighted average of observed returns, so it can never leave their range. In practice the bias fades fast and the stability wins, which is why weighted is the default for Monte Carlo control off policy.

Think of It Like This

Judging a crowd from a coffee shop survey

You want the average height at a basketball tournament but you only surveyed the coffee shop next door, where players rarely go. To correct the sampling bias you give each person a multiplier: regulars get a small one, players get a huge one.

Ordinary sampling multiplies every height by its multiplier and divides by the headcount. One 7-foot player with a multiplier of 50 adds 350 feet to the total, and dividing by 10 people surveyed says the average attendee is over 30 feet tall. The denominator knows nothing about the multipliers, so one outlier writes the headline.

Weighted sampling divides by the sum of the multipliers instead. That same 50 appears upstairs and downstairs, so the player counts as one large share of a true average. The answer must land between the shortest and tallest person you met.

Where the analogy stops: survey multipliers are fixed constants, while trajectory ratios multiply per-step chances along a whole episode and can span many orders of magnitude in one training run.

How It Actually Works

From $n$ episodes under $b$ , with coverage ( $\pi(a \mid s) > 0$ implies $b(a \mid s) > 0$ ), write $\rho_i$ for the full-trajectory ratio of episode $i$ and $G_i$ for its return.

Ordinary:

V^{\mathrm{ord}}(s) = \frac{1}{n} \sum_{i=1}^{n} \rho_i G_i

Its expectation under $b$ equals $v_{\pi}(s)$ , so bias is zero at any $n$ . But its variance carries $\mathbb{E}_b[\rho_i^2 G_i^2]$ , and $\rho_i$ is a product of per-step ratios, so the second moment grows exponentially with horizon. When that moment is infinite, averages never settle no matter how many episodes you collect.

Weighted (self-normalized):

V^{\mathrm{wis}}(s) = \frac{\sum_{i=1}^{n} \rho_i G_i}{\sum_{i=1}^{n} \rho_i}

With weights $w_i = \rho_i / \sum_k \rho_k$ summing to 1, this is a convex combination of observed returns, so it always sits between $\min G_i$ and $\max G_i$ . At $n = 1$ it collapses to $G_1$ , whose expectation is the behavior value $v_b$ , hence the finite-sample bias. As $n$ grows the numerator converges to $v_{\pi}$ and the denominator to 1, so the bias washes out: the estimator is consistent.

Worked example

Three episodes evaluated at $S_0$ , all returns in $[0.0, 1.0]$ :

Episode 1: $G_1 = 1.0$ , $\rho_1 = 0.10$ , product $0.10$
Episode 2: $G_2 = 0.0$ , $\rho_2 = 0.40$ , product $0.00$
Episode 3: $G_3 = 1.0$ , $\rho_3 = 14.50$ , product $14.50$

Weighted sum: $0.10 + 0.00 + 14.50 = 14.60$ .

Ordinary divides by $n = 3$ :

V^{\mathrm{ord}} = 14.60 / 3 \approx 4.87

Every observed return was at most $1.0$ , yet the estimate says $4.87$ . One rare trajectory wrote the headline.

Weighted divides by the ratio sum $0.10 + 0.40 + 14.50 = 15.00$ :

V^{\mathrm{wis}} = 14.60 / 15.00 \approx 0.97

Inside the payoff range and close to the true $v_{\pi} = 1.0$ . Same evidence, different denominator, different story.

Code

Direct recomputation of the fixture, no sampling involved.

1returns = [1.0, 0.0, 1.0]2ratios = [0.10, 0.40, 14.50]3
4weighted_sum = sum(r * g for r, g in zip(ratios, returns))5ois = weighted_sum / len(ratios)6wis = weighted_sum / sum(ratios)7
8print(f"weighted sum = {weighted_sum:.2f}")9print(f"ordinary = {ois:.2f}")10print(f"weighted = {wis:.2f}")11
12assert abs(weighted_sum - 14.60) < 1e-913assert abs(ois - 4.87) < 0.0114assert abs(wis - 0.97) < 0.0115assert min(returns) <= wis <= max(returns)16print("checks passed")

1weighted sum = 14.602ordinary = 4.873weighted = 0.974checks passed

Watch Out For

Picking the unbiased estimator on principle

Classical training says unbiased beats biased, so teams reach for ordinary sampling. Off policy, that principle misfires: the variance of ordinary sampling can be infinite when $b$ undercovers $\pi$ even slightly, and then the mean squared error is dominated by variance no matter how clean the bias looks. The symptom is estimates that swing wildly between runs and occasionally land outside any feasible return.

Fix it by defaulting to the weighted estimator and comparing mean squared error, not bias alone. The weighted bias shrinks like $1/n$ while the variance stays bounded by construction.

Forgetting the coverage assumption

Both estimators need $b(a \mid s) > 0$ wherever $\pi(a \mid s) > 0$ . If the behavior policy never tries an action the target policy takes, the ratio divides by zero and no denominator trick can recover the missing evidence. The symptom is ratios of exactly zero masking whole branches of the target policy.

Fix it by auditing action support before estimating: confirm every target action appears under $b$ , and if not, widen the behavior policy or restrict the claim to the covered region.

The Quick Version

Ordinary sampling averages ratio-weighted returns over the episode count. It is unbiased at any sample size but its variance grows exponentially with horizon and can be infinite.
Weighted sampling divides by the sum of the ratios, which makes it a weighted average that always stays between the smallest and largest observed return.
Weighted sampling is biased for small samples and consistent as samples grow, since the denominator converges to 1 and the bias fades.
On weights 0.10, 0.40, and 14.50 with returns in [0.0, 1.0], ordinary spikes to 4.87 while weighted lands at 0.97.
Both need coverage: the behavior policy must try every action the target policy takes, or the missing branch stays invisible.