Off-Policy Prediction via Importance Sampling

Judge a policy you never run by watching a different policy act, then reweighting each episode by how much more or less likely your policy would have acted the same way.

Step ratios 1.60, 0.50 and 2.00 multiply to 1.60, correcting one episode return of 8.0 to 12.80 for a policy that never acted

Why Does This Exist?

Learning and acting pull in opposite directions. To find a good policy you must try doubtful actions, but to score a policy you must play it straight. On-policy methods force one policy to do both jobs, so every update throws away yesterday's episodes and every evaluation is polluted by exploration noise.

Off-policy prediction splits the jobs. The behavior policy $b$ touches the environment: it explores, plays safe, or just keeps the lights on. The target policy $\pi$ never acts at all; it is the candidate you want scored. Raw episode returns belong to $b$ , so averaging them scores the wrong policy. Importance sampling is the correction that turns $b$ 's episodes into an unbiased score for $\pi$ .

This page covers prediction: scoring one fixed target policy from behavior data. The variance-reduced sibling is ordinary vs weighted importance sampling, and the on-policy counterpart that needs no correction is on-policy first-visit MC control.

The one idea you need first

An expectation under one distribution can be estimated with samples from another, as long as every outcome possible under the first is possible under the second. That support condition is the coverage assumption: $\pi(a \mid s) > 0$ implies $b(a \mid s) > 0$ . Each sample gets multiplied by the probability ratio $\pi / b$ , which upweights underrepresented outcomes and downweights overrepresented ones. Everything below is that trick applied step by step along a trajectory.

Think of It Like This

The campus poll that forecasts a whole city

A polling crew wants to forecast a city vote but only manages to interview students on one campus. Among students, 80 percent back the transit plan and 20 percent back highways. Census data says the city as a whole splits 40 percent transit, 60 percent highways. Averaging raw student answers would call the election wrong by a mile.

So each answer gets a correction weight. A transit answer counts $0.40 / 0.80 = 0.50$ , half a vote, because students overrepresent that camp. A highway answer counts $0.60 / 0.20 = 3.00$ , triple, because students underrepresent it. The weighted average now estimates the city, even though no non-student was ever interviewed.

Where it stops: a poll corrects one answer with one ratio. An episode is a chain of decisions, so RL multiplies a ratio per step, and that product can swing wildly in long episodes in a way polls never do.

How It Actually Works

Write $\pi(a \mid s)$ for the target policy's action probability and $b(a \mid s)$ for the behavior policy's. For a trajectory from step $t$ to termination $T$ , the probability of that exact action sequence under each policy differs only in the action probabilities, because the environment dynamics $p(s' \mid s, a)$ appear in both and cancel:

\rho_{t:T-1} = \prod_{k=t}^{T-1} \frac{\pi(A_k \mid S_k)}{b(A_k \mid S_k)}

That product is the importance sampling ratio. Multiply the episode return $G_t$ by it, and the behavior expectation equals the target value exactly: $\mathbb{E}_b[\rho_{t:T-1} G_t \mid S_t = s] = v_\pi(s)$ . Average over $N$ episodes and you have ordinary importance sampling:

V(s) = \frac{1}{N} \sum_{i=1}^{N} \rho^{(i)} G_t^{(i)}

Unbiased, model-free, and built only from action probabilities you already know.

Worked example

One 3-step episode with discount $\gamma = 1.0$ and rewards 2, 1, 5, so the raw return is $G_0 = 2 + 1 + 5 = 8.0$ . The policies disagree at every step:

Step 0: $\pi = 0.80$ , $b = 0.50$ , ratio $0.80 / 0.50 = 1.60$
Step 1: $\pi = 0.40$ , $b = 0.80$ , ratio $0.40 / 0.80 = 0.50$
Step 2: $\pi = 0.50$ , $b = 0.25$ , ratio $0.50 / 0.25 = 2.00$

The cumulative ratio is $\rho = 1.60 \times 0.50 \times 2.00 = 1.60$ , and the corrected vote of this episode is $1.60 \times 8.0 = 12.80$ .

A second episode from the same start has return $G = 4.0$ with cumulative ratio $0.75$ , giving $0.75 \times 4.0 = 3.00$ . The ordinary importance sampling estimate across both episodes is $(12.80 + 3.00) / 2 = 7.90$ , an unbiased read on a policy that never touched the environment.

Code

This snippet computes the corrected return for both episodes above:

1def corrected_return(rewards: list[float], ratios: list[float]) -> tuple[float, float, float]:2    """Raw return, cumulative ratio rho, and rho-weighted return."""3    rho = 1.04    for ratio in ratios:5        rho = rho * ratio6    total = 0.07    for reward in rewards:8        total = total + reward9    weighted = rho * total10    return round(total, 2), round(rho, 2), round(weighted, 2)11
12
13print(corrected_return([2.0, 1.0, 5.0], [1.60, 0.50, 2.00]))  # -> (8.0, 1.6, 12.8)14print(corrected_return([1.0, 3.0], [0.75, 1.00]))  # -> (4.0, 0.75, 3.0)

Both outputs match the hand computation, and their mean $(12.8 + 3.0) / 2 = 7.9$ is the value estimate.

Watch Out For

Products of ratios explode on long episodes

Ordinary importance sampling is unbiased, but its variance grows exponentially with horizon. Each extra step multiplies another ratio, so one episode that happens to match $\pi$ everywhere can carry a ratio in the hundreds while typical episodes sit near zero. The average then jerks around with every new batch, and short experiments look conclusive right before they reverse.

Watch the distribution of $\rho$ itself, not just the estimate. If a handful of episodes dominate the sum, switch to weighted importance sampling, per-decision ratios that stop the product at each reward, or a one-step TD method that never forms the long product at all.

A zero in the behavior policy silently voids the guarantee

If $b(a \mid s) = 0$ anywhere $\pi(a \mid s) > 0$ , coverage is broken: that action never appears in the data, the ratio divides by zero, and the estimate is biased with no warning from the math. Deterministic loggers are the usual culprit, since a policy that always picks one action assigns zero to everything else.

Keep the behavior policy soft, with every action playable in every visited state, and assert $b > 0$ before dividing. A cheap guard is to log $\min b(A_t \mid S_t)$ per batch and investigate any batch that touches zero.

The Quick Version

Off-policy prediction scores a target policy $\pi$ using episodes gathered by a separate behavior policy $b$ .
Coverage requires $b(a \mid s) > 0$ wherever $\pi(a \mid s) > 0$ , or the correction divides by zero.
The trajectory ratio $\rho = \prod \pi / b$ cancels environment dynamics, so the method is fully model-free.
Weighting returns by $\rho$ is unbiased: episode ratios 1.60, 0.50 and 2.00 give $\rho = 1.60$ and turn 8.0 into 12.80.
Long products mean wild variance, so check the spread of $\rho$ and prefer weighted or per-decision variants on long horizons.