Backward View of TD(lambda)

Why Does This Exist?

The forward view defines the $\lambda$ return as a mix of all future $n$-step returns. Clean theory, but you must see the future to compute it. In continuing tasks or live control you cannot pause for the episode to finish.

The backward view makes the same idea causal. Keep an eligibility trace $z(s)$ per state, a fading note of how recently you were there. On each step compute one scalar surprise and broadcast it to every state weighted by its trace. Credit flows the moment the transition lands, with $O(|\mathcal{S}|)$ memory and no trajectory store.

This page covers the online mechanism. See eligibility traces for trace variants, forward view TD lambda for the lookahead definition, and their equivalence for when the totals match.

You need temporal difference learning first: $R + \gamma V(s') - V(s)$ as prediction error.

Think of It Like This

How It Actually Works

Keep values $V(s)$ and traces $z(s)$, both zeroed at episode start. With discount $\gamma$ and decay $\lambda$, accumulating traces update as:

Here $1(s = S_t)$ is one for the visited state and zero else. Then compute the one step error for $S_t \to S_{t+1}$ with reward $R$:

Then broadcast with step size $\alpha$:

Fresh states with $z \approx 1$ take the full hit. Old states take a decayed share. Unvisited states with $z = 0$ take nothing.

Worked example

Chain $A \to B \to C \to \text{end}$ with $R = 0.0, 1.0, 5.0$. Set $\gamma = 0.9$, $\lambda = 0.8$, so decay is $0.72$, and $\alpha = 0.1$. Start $V = [1.00, 2.00, 3.00]$, traces zero.

Step $A \to B$: traces $[1.00, 0.00, 0.00]$. Error $\delta_0 = 0.0 + 0.9 \times 2.00 - 1.00 = 0.80$. Then $V(A) = 1.00 + 0.1 \times 0.80 \times 1.00 = 1.08$.

Step $B \to C$: traces $[0.72, 1.00, 0.00]$. Error $\delta_1 = 1.0 + 0.9 \times 3.00 - 2.00 = 1.70$. Then $V(A) = 1.08 + 0.1 \times 1.70 \times 0.72 = 1.2024$ and $V(B) = 2.00 + 0.1 \times 1.70 \times 1.00 = 2.17$. Note A learns while the agent stands at B.

Step $C \to \text{end}$: traces $[0.5184, 0.72, 1.00]$ since $0.72 \times 0.72 = 0.5184$. Error $\delta_2 = 5.0 - 3.00 = 2.00$. Then $V(A) = 1.2024 + 0.1 \times 2.00 \times 0.5184 = 1.3061$, $V(B) = 2.17 + 0.1 \times 2.00 \times 0.72 = 2.3140$, $V(C) = 3.00 + 0.2 = 3.20$.

One $2.00$ surprise updates all three states at once.

Code

Same three steps, no hidden state.

1from typing import Dict, List2
3
4def td_broadcast(5    values: Dict[str, float],6    traces: Dict[str, float],7    gamma: float,8    lam: float,9    alpha: float,10    state: str,11    reward: float,12    v_next: float,13) -> float:14    """Decay traces, bump the visit, broadcast one TD error."""15    decay = gamma * lam16    for s in traces:17        traces[s] = decay * traces[s] + (1.0 if s == state else 0.0)18    delta = reward + gamma * v_next - values[state]19    for s in values:20        values[s] += alpha * delta * traces[s]21    return delta22
23
24states: List[str] = ["A", "B", "C"]25v: Dict[str, float] = {"A": 1.0, "B": 2.0, "C": 3.0}26z: Dict[str, float] = {"A": 0.0, "B": 0.0, "C": 0.0}27
28print(f"{td_broadcast(v, z, 0.9, 0.8, 0.1, 'A', 0.0, v['B']):.4f}")  # -> 0.800029print(f"{td_broadcast(v, z, 0.9, 0.8, 0.1, 'B', 1.0, v['C']):.4f}")  # -> 1.700030print(f"{td_broadcast(v, z, 0.9, 0.8, 0.1, 'C', 5.0, 0.0):.4f}")  # -> 2.000031print(f"{v['A']:.4f} {v['B']:.4f} {v['C']:.4f}")  # -> 1.3061 2.3140 3.2000

Deltas and finals match the hand run: $0.80$, $1.70$, $2.00$, ending $[1.3061, 2.3140, 3.20]$.

Watch Out For

The Quick Version

• Traces $z$ are fading visit memory with per step factor $\gamma \lambda$, here $0.72$.
• Each step broadcasts one scalar $\delta = R + \gamma V(s') - V(s)$ to all states via $\alpha \delta z$.
• The run gives deltas $0.80$, $1.70$, $2.00$ and finals $1.3061$, $2.3140$, $3.20$.
• Decay every step and reset each episode or credit leaks across time.
• Offline totals match the forward $\lambda$ return under small steps.

What to Read Next

• Eligibility Traces
• Forward View of TD(lambda)
• Equivalence of Forward and Backward Views