Emphatic-TD Methods in RL

Off-policy updates visit the wrong states too often, so some corrections need extra weight. Emphatic TD adds that weight with a followon trace.

Followon trace grows from 1.0 to 2.8 after an off-policy step with ratio 2.0, boosting the correction on the downstream state

Why Does This Exist?

Off-policy learning with function approximation can diverge. You follow one behavior policy to collect data but evaluate a different target policy, while bootstrapping from your own guesses. That trio is the deadly triad, and on tasks like Baird's counterexample plain TD weights grow without bound.

The importance ratio $\rho_t = \pi(A_t \mid S_t) / b(A_t \mid S_t)$ fixes which action was picked, where $\pi$ is the target policy and $b$ is the behavior policy. It does not fix which states you visit. States the behavior policy loves still dominate the updates even when the target policy rarely visits them.

Emphatic TD fixes the state weighting with one extra scalar per step. It keeps a single weight vector and one step size, unlike gradient TD methods which need a second weight vector and two timescales.

The one idea you need first

Interest $I_t$ says how much you care about state $S_t$ . Use $I_t = 1$ everywhere for plain evaluation, or 1 on states you care about and 0 elsewhere for selective prediction. Everything else builds on that choice.

Think of It Like This

A teacher grading borrowed homework

You grade homework from a night class (behavior policy) but the grades count for the day class (target policy). Some day-class topics appear constantly in the night pile, others almost never.

If you grade every sheet equally, the overrepresented topic steers the whole curve. So you attach a slip to each sheet saying how much day-class work depended on it. Sheets that many day-class answers build on get a heavy slip, loners get a light one. The average then tracks the day class, not the night pile.

Where it stops: a teacher can reweight after seeing the pile. Emphatic TD builds the weight forward step by step, before seeing the future.

How It Actually Works

The followon trace $F_t$ carries past interest forward. Start with $F_0 = I_0$ . Then:

F_t = \gamma \rho_{t-1} F_{t-1} + I_t

Here $\gamma$ discounts the future and $\rho_{t-1}$ is the importance ratio of the step that arrived here. Emphasis $M_t$ turns followon into an update weight. For one-step ETD with $\lambda = 0$ , emphasis equals followon:

M_t = F_t

The weight update with features $\mathbf{x}(S_t)$ , error $\delta_t$ , and step size $\alpha$ is:

\mathbf{w}_{t+1} = \mathbf{w}_t + \alpha M_t \rho_t \delta_t \mathbf{x}(S_t)

In expectation this reweights states toward the target visitation instead of the behavior visitation, which restores a positive definite system matrix and a stable fixed point.

Worked example

Use $\gamma = 0.9$ , $\alpha = 0.05$ , interest $1.0$ , features $\mathbf{x}(s_1) = [1, 0]$ and $\mathbf{x}(s_2) = [0, 2]$ , weights $\mathbf{w}_0 = [1.0, 1.0]$ , so values are $1.0$ and $2.0$ . Rewards are 0.

Step 0 goes $s_1$ to $s_2$ . Followon and emphasis start at $F_0 = 1.0$ and $M_0 = 1.0$ . Target picks the action for sure while behavior picks it half the time, so $\rho_0 = 2.0$ . The error is $\delta_0 = 0 + 0.9 \times 2.0 - 1.0 = 0.8$ . The update adds $0.05 \times 1.0 \times 2.0 \times 0.8 = 0.08$ on the first weight, giving $\mathbf{w}_1 = [1.08, 1.00]$ .

Step 1 stays in $s_2$ . Followon inherits the last step:

F_1 = 0.9 \times 2.0 \times 1.0 + 1.0 = 2.8, \quad M_1 = 2.8

Both policies agree here so $\rho_1 = 1.0$ . The value is still $2.0$ , so $\delta_1 = 0 + 0.9 \times 2.0 - 2.0 = -0.2$ . The update adds $0.05 \times 2.8 \times 1.0 \times (-0.2) \times [0, 2] = [0, -0.056]$ , giving $\mathbf{w}_2 = [1.08, 0.944]$ .

The downstream correction got boosted because $s_1$ had bootstrapped from $s_2$ through an off-policy step. That boost is the whole method.

Code

Followon is one line. Emphasis equals followon for one-step ETD, and the effective step grows with it.

1def followon(f_prev: float, rho_prev: float, gamma: float, interest: float) -> float:2    return gamma * rho_prev * f_prev + interest  # inherited plus fresh interest3
4
5f0: float = 1.06f1: float = followon(f0, 2.0, 0.9, 1.0)7print(f"{f0:.1f} {f1:.1f} {0.05 * f1 * 1.0 * -0.2 * 2.0:.3f}")8# -> 1.0 2.8 -0.056

The triple is $F_0$ , $F_1$ , and the second-coordinate change that moves 1.000 to 0.944. It matches the hand steps.

Watch Out For

Watching emphasis spike on long off-policy chains

Followon multiplies past ratios, so two or three large $\rho$ values in a row push $M_t$ into the tens or hundreds. The effective step $\alpha M_t$ then jumps far past a safe range and weights lurch.

Damp it with $\lambda$ above 0, start $\alpha$ smaller than your on-policy value, and clip $M_t$ in code that runs on logged data. If you see occasional huge updates amid calm ones, this is the cause.

Leaving interest at one everywhere without thinking

Uniform interest is fine for full evaluation, but it lets high-traffic behavior states dominate followon. When you only care about start states or a subtask, uniform interest wastes stability budget where it does not matter.

Set $I_t$ to 1 on states you actually predict and 0 elsewhere. Followon then channels emphasis to their ancestors instead of spreading it thin.

The Quick Version

Emphatic TD reweights each update by emphasis $M_t$ built from followon $F_t$ .
Followon adds fresh interest to discounted, ratio-weighted past interest.
One-step ETD sets $M_t = F_t$ and keeps one weight vector with one step size.
The reweighting restores a contracting update under off-policy sampling.
The price is variance: long chains of large ratios spike $M_t$ , so damp or clip it.