Forward View of TD(lambda)

Instead of fixing one lookahead, mix every lookahead at once: mostly the one-step return, a little less of the two-step, fading geometrically out to the full episode. One knob slides from TD to Monte Carlo.

Weights 0.30, 0.21 and 0.49 blend the n-step returns 4.70, 6.95 and 6.14 into one lambda target of 5.88, sitting between TD at 4.70 and Monte Carlo at 6.14

Why Does This Exist?

One-step TD and Monte Carlo sit at opposite ends of a real dilemma. Looking one transition ahead gives low-variance targets and updates after every step, but bootstraps on young, biased guesses. Waiting for the full episode removes that bias and replaces it with the compounded noise of the whole trajectory. Multi-step TD compromises with a fixed horizon $n$ , covered at n-step TD prediction, yet any single $n$ is arbitrary: short horizons win in some states, long ones in others.

The forward view refuses to choose. It defines the $\lambda$ -return $G_t^\lambda$ : a weighted average of all $n$ -step returns with geometrically fading weights $(1 - \lambda)\lambda^{n-1}$ , $\lambda$ in $[0, 1]$ . Short low-variance horizons get the most weight, long low-bias horizons get progressively less, and the weights always sum to one. The geometric shape is load-bearing, not decorative: its memoryless property is exactly what lets the online backward view with eligibility traces reproduce this target causally, step by step.

The one idea you need first.

An $n$ -step return $G_{t:t+n}$ adds up $n$ real rewards then bootstraps: $G_{t:t+n} = R_{t+1} + \gamma R_{t+2} + \dots + \gamma^{n-1}R_{t+n} + \gamma^n V_t(S_{t+n})$ , where $\gamma$ in $[0, 1]$ is the discount factor and $V_t$ is the current value guess. At $n = 1$ this is the TD target; at the episode end it is the Monte Carlo return. Everything below mixes these objects.

Think of It Like This

A concert hall blending direct sound with echoes

A violin note reaches you twice: first straight from the strings, sharp and exact, then as early reflections off the near walls, then as a long fading tail from the far balconies. You never hear the dry pluck alone or the wash of echoes alone; your ear blends the crisp attack with the decaying tail into one warm percept.

Point for point: the direct sound is the 1-step return with weight $1 - \lambda$ , the early reflections are the 2-step and 3-step returns with weights $(1 - \lambda)\lambda$ and $(1 - \lambda)\lambda^2$ , the tail is the distant horizons and the Monte Carlo remainder, and the blend you perceive is $G_t^\lambda$ . Where it stops: the hall's decay is fixed by architecture, while $\lambda$ is a knob you turn deliberately. And real echoes arrive after the note, whereas the forward view's echoes are future rewards that have not happened yet at time $t$ , which is why this view is theory, not an online algorithm.

How It Actually Works

The lambda return

In continuing tasks the definition runs over every horizon:

$G_t^\lambda = (1 - \lambda) \sum_{n=1}^{\infty} \lambda^{n-1} G_{t:t+n}$

The weights form a valid distribution because $(1 - \lambda)\sum_{k=0}^{\infty}\lambda^k = 1$ . In an episode ending at $T$ , every $n$ -step return past $T - t$ equals the full return $G_t$ , so the infinite tail collapses into one remainder weight $\lambda^{T-t-1}$ :

$G_t^\lambda = (1 - \lambda) \sum_{n=1}^{T-t-1} \lambda^{n-1} G_{t:t+n} + \lambda^{T-t-1} G_t$

The finite weights sum to $1 - \lambda^{T-t-1}$ and the tail supplies the rest, so the total is exactly $1.0$ . The value update aims at this composite target with step size $\alpha$ in $(0, 1]$ : $V(S_t) \leftarrow V(S_t) + \alpha[G_t^\lambda - V(S_t)]$ .

The two endpoints fall out for free

At $\lambda = 0$ every weight except the first vanishes and $G_t^0 = G_{t:t+1}$ , pure one-step TD. At $\lambda = 1$ the finite weights vanish and the tail weight is $1$ , so $G_t^1 = G_t$ , pure Monte Carlo. Between them the target slides smoothly from low-variance bootstrapping to unbiased full rollouts, which is why $\lambda$ reads as a bias-variance knob.

Worked example

Three transitions: $S_0$ with reward $2.0$ , $S_1$ with reward $1.0$ , $S_2$ with reward $4.0$ , then termination. Discount $\gamma = 0.9$ , blend $\lambda = 0.7$ , step size $\alpha = 0.1$ . Current guesses $V(S_0) = 1.0$ , $V(S_1) = 3.0$ , $V(S_2) = 5.0$ , terminal $0.0$ . Target at $t = 0$ , where $T - t = 3$ .

The three $n$ -step returns from $t = 0$ :

$G_{0:1} = 2.0 + 0.9 \times 3.0 = 4.7000$
$G_{0:2} = 2.0 + 0.9 \times 1.0 + 0.81 \times 5.0 = 2.90 + 4.05 = 6.9500$
$G_{0:3} = G_0 = 2.0 + 0.90 + 0.81 \times 4.0 = 2.90 + 3.24 = 6.1400$

The weights: $w_1 = 1 - 0.7 = 0.30$ , $w_2 = 0.30 \times 0.7 = 0.21$ , tail $w_3 = 0.7^2 = 0.49$ , summing to $1.00$ . The blend: $0.30 \times 4.70 + 0.21 \times 6.95 + 0.49 \times 6.14 = 1.4100 + 1.4595 + 3.0086 = 5.8781$ . Against the endpoints, TD alone aims at $4.70$ and Monte Carlo at $6.14$ ; the $0.7$ blend lands at $5.88$ between them. The update moves $V(S_0)$ from $1.0$ to $1.0 + 0.1 \times (5.8781 - 1.0) = 1.4878$ .

Code

The $n$ -step returns, the blended target, the full $\lambda$ spectrum, and the offline update, using only the standard library.

1from typing import Dict, List2
3
4def n_step(rewards: List[float], start: int, n: int, gamma: float, boot: float) -> float:5    """n-step return from index start: n real rewards, then bootstrap value."""6    total = 0.07    discount = 1.08    for k in range(start, min(start + n, len(rewards))):9        total += discount * rewards[k]10        discount *= gamma11    if start + n < len(rewards):12        total += discount * boot13    return total14
15
16def lambda_return(rewards: List[float], start: int, gamma: float, lam: float, values: List[float]) -> float:17    """Episodic forward lambda return at index start."""18    remaining = len(rewards) - start19    targets = [n_step(rewards, start, n, gamma, values[start + n]) for n in range(1, remaining + 1)]20    blended = sum((1.0 - lam) * lam ** (n - 1) * targets[n - 1] for n in range(1, remaining))21    return blended + lam ** (remaining - 1) * targets[-1]22
23
24rewards = [2.0, 1.0, 4.0]25values = [1.0, 3.0, 5.0, 0.0]  # last entry is the terminal value V = 0.026gamma, lam, alpha = 0.9, 0.7, 0.127print(f"{n_step(rewards, 0, 1, gamma, values[1]):.4f}")  # -> 4.700028print(f"{n_step(rewards, 0, 2, gamma, values[2]):.4f}")  # -> 6.950029print(f"{n_step(rewards, 0, 3, gamma, 0.0):.4f}")  # -> 6.140030print(f"{lambda_return(rewards, 0, gamma, lam, values):.4f}")  # -> 5.878131for test in [0.0, 0.5, 0.7, 1.0]:32    print(f"{test:.1f} {lambda_return(rewards, 0, gamma, test, values):.4f}")  # -> 0.0 4.7000, 0.5 5.6225, 0.7 5.8781, 1.0 6.140033print(f"{values[0] + alpha * (lambda_return(rewards, 0, gamma, lam, values) - values[0]):.4f}")  # -> 1.4878

Every figure matches the hand trace, including the $5.6225$ midpoint of the spectrum at $\lambda = 0.5$ .

Watch Out For

Shipping the forward view as an online algorithm

The definition peeks at rewards that have not arrived yet: at step $t$ the target needs $R_{t+1}$ through $R_T$ . Code that computes it directly cannot update until the episode ends, grows unbounded buffers on continuing tasks, and reintroduces Monte Carlo latency into what was supposed to be online learning.

The symptom is an agent that learns in end-of-episode jerks or an implementation that never finalizes a target on non-terminating streams. The fix is the division of labor the theory intends: keep the forward view as the specification, and run the backward view with eligibility traces for the causal updates.

Parking lambda at 1 on long noisy episodes

With $\lambda = 1$ the target is the full Monte Carlo return, so every bit of trajectory noise lands in the update. On episodes hundreds of steps long with stochastic transitions, targets swing so widely that learning stalls despite being unbiased.

The symptom is high-variance updates and a value curve that thrashes while short-horizon runs on the same task settle. The fix is an intermediate $\lambda$ around $0.7$ to $0.9$ : keep most of the low bias while the geometric tail damps the far noise. Reserve exactly $1$ for short deterministic episodes where the variance has nowhere to hide.

The Quick Version

The $\lambda$ -return blends every $n$ -step return with geometric weights $(1 - \lambda)\lambda^{n-1}$ plus a tail weight $\lambda^{T-t-1}$ on the full return, always summing to one.
At $\lambda = 0$ it is one-step TD and at $\lambda = 1$ it is Monte Carlo; between them it trades bootstrap bias against rollout variance.
In the worked episode the weights $0.30$ , $0.21$ , $0.49$ blend $4.70$ , $6.95$ , $6.14$ into $5.88$ , moving $V(S_0)$ from $1.00$ to $1.49$ .
The forward view needs future rewards, so it cannot run online; it is the specification that eligibility traces implement causally.
On long noisy episodes prefer an intermediate lambda near $0.7$ to $0.9$ over exactly $1$ .