Equivalence of Forward and Backward Views

Looking ahead to future rewards and looking back with a memory trace sound different, but offline they add up to the same total. The proof swaps two sums.

Forward lambda return 6.06 and backward trace updates both give 0.506 for S0 and 0.200 for S1

Why Does This Exist?

The forward view defines the ideal target: mix all future $n$ -step returns into one lambda return $G_t^\lambda$ . You cannot compute it online because it needs rewards you have not seen yet. The backward view is the runnable method: keep eligibility traces and broadcast each one-step error backward as it arrives.

Are they solving the same problem or just rhyming? The equivalence says the totals match exactly when you freeze values during the episode and add updates at the end. That license matters: it lets you implement the cheap backward pass while claiming the forward target.

Think of It Like This

Splitting a dinner bill two ways

Four friends share dishes over two hours. One person totals the whole receipt at the end and divides it by what each person ordered. Another writes down each new dish as it arrives and splits that dish on the spot across whoever is still at the table.

At midnight both methods charge each person the same total. One waited for the full picture, the other kept a running tab. The running tab only matches because nothing on the receipt changed mid-meal.

Where it stops: restaurant bills do not discount later dishes. TD weights later errors by $(\gamma \lambda)^{k-t}$ , so timing changes the split.

How It Actually Works

Write the one-step error with reward $R_{k+1}$ , discount $\gamma$ , and values $V$ :

\delta_k = R_{k+1} + \gamma V(S_{k+1}) - V(S_k)

An $n$ -step return minus the current value telescopes into errors:

G_t^{(1)} - V(S_t) = \delta_t, \quad G_t^{(2)} - V(S_t) = \delta_t + \gamma \delta_{t+1}

Mixing all horizons with weights $(1 - \lambda)\lambda^{n-1}$ groups each error under one exponential:

G_t^\lambda - V(S_t) = \sum_{k=t}^{T-1} (\gamma \lambda)^{k-t} \delta_k

In words: the forward error equals discounted future one-step errors. Summing that over every visit and swapping the two sums regroups the weights into accumulating traces $z_k(s)$ , so the forward total equals $\sum_k \alpha \delta_k z_k(s)$ . That sum is exactly the backward total. Same number, two routes.

Worked example

Walk $S_0$ to $S_1$ to termination with $R_1 = 3.0$ and $R_2 = 4.0$ . Start $V(S_0) = 1.0$ and $V(S_1) = 2.0$ , with $\gamma = 0.9$ , $\lambda = 0.7$ , decay $0.63$ , and $\alpha = 0.1$ .

Forward: $G_0^{(1)} = 3.0 + 0.9 \times 2.0 = 4.800$ and $G_0^{(2)} = 3.0 + 0.9 \times 4.0 = 6.600$ . The lambda return is $0.3 \times 4.800 + 0.7 \times 6.600 = 6.060$ . Updates are $0.1 \times (6.060 - 1.0) = 0.506$ for $S_0$ and $0.1 \times (4.0 - 2.0) = 0.200$ for $S_1$ .

Backward: errors are $\delta_0 = 3.0 + 0.9 \times 2.0 - 1.0 = 3.8$ and $\delta_1 = 4.0 - 2.0 = 2.0$ . Traces visit $S_0$ then $S_1$ , giving $z_0 = [1.00, 0.00]$ and $z_1 = [0.63, 1.00]$ . Totals are $0.1 \times 3.8 \times 1.00 + 0.1 \times 2.0 \times 0.63 = 0.506$ for $S_0$ and $0.200$ for $S_1$ . Identical to three decimals.

Code

Compute both totals from frozen start values and compare. They agree to machine precision.

1def both_views() -> tuple:2    gamma, lam, alpha = 0.9, 0.7, 0.1  # shared decay 0.633    gl = 0.3 * 4.8 + 0.7 * 6.6  # lambda return from 4.8 and 6.64    fwd0 = alpha * (gl - 1.0)  # forward S05    bwd0 = alpha * 3.8 * 1.0 + alpha * 2.0 * 0.63  # backward S06    return round(fwd0, 3), round(bwd0, 3), round(alpha * 2.0, 3)7
8
9print(both_views())10# -> (0.506, 0.506, 0.2)

The triple is forward $S_0$ , backward $S_0$ , and the shared $S_1$ update. The first two match.

Watch Out For

Expecting online TD to match the forward total

The proof freezes values while accumulating. Standard online TD updates values mid-episode, so later errors use newer values and the telescoping breaks. With a large step size the drift is visible within a few episodes.

If you need the forward number while learning online, use true online TD with Dutch traces, or keep alpha small so drift stays minor. Do not benchmark online updates against offline totals.

Swapping in replacing traces and keeping the proof

The sum swap produces accumulating traces, which add on revisits. Replacing traces reset to 1.0 and behave better in loops, but they change the weights and the exact identity no longer holds.

Use accumulating traces when you claim exact offline equivalence. Use replacing traces as a practical upgrade and describe it as an approximation, not the same theorem.

The Quick Version

The forward view defines the lambda return from future rewards, which is ideal but not causal.
The backward view broadcasts each one-step error through eligibility traces, which runs online.
Multi-step errors telescope into sums of one-step errors weighted by gamma times lambda.
Swapping the double sum turns future weights into past traces, so offline totals match.
Online updates drift unless you use true online TD, and replacing traces break the exact match.