Semi-Gradient TD(0) in RL

Learn a value guess after every step by aiming at your own next guess, while refusing to chase the target as it moves.

One semi-gradient TD(0) step turns reward 2.0 and next guess 1.25 into target 3.125, error 0.625, and weights [0.5625, 1.125]

Why Does This Exist?

Monte Carlo prediction waits for the episode to end, then learns from the full return that actually happened. That target is honest but slow: useless for continuing tasks that never end, and noisy enough that learning crawls.

Temporal difference learning refuses to wait. After a single transition $(S_t, R_{t+1}, S_{t+1})$ it builds a bootstrapped target from its own next guess:

$U_t = R_{t+1} + \gamma \hat{v}(S_{t+1}, \mathbf{w}_t)$

Here $\gamma$ in $[0, 1]$ discounts the future and $\mathbf{w}$ is the shared weight vector. The catch is visible in the formula: the target contains the very weights being learned. This page covers the standard fix, the semi-gradient update, and one computed step. The action-value version is semi-gradient SARSA, the patient alternative is gradient Monte Carlo, and the mechanism underneath is temporal difference learning.

The one idea you need first

With linear features, the value guess is a dot product $\hat{v}(s) = \mathbf{w}^T \mathbf{x}(s)$ , and its gradient with respect to $\mathbf{w}$ is the feature vector $\mathbf{x}(s)$ . That single identity turns the whole update into one vector push. Full grounding lives in value function approximation.

Think of It Like This

Two hikers shouting distance guesses down the trail

Two hikers walk a trail in fog. The one behind shouts, how far to the hut? The one ahead shouts back their own guess, plus the steps already walked. The behind hiker adjusts toward that answer without arguing about how the ahead hiker guessed.

Both guesses come from the same guidebook, so the target moves as the guidebook gets edited. That is bootstrapping: learning from your own moving estimate instead of waiting to reach the hut. It is fast, and it only works if you treat the ahead guess as fixed for this one adjustment.

Where the analogy stops: hikers can eventually reach the hut and check. In continuing tasks the agent never does, so the fixed point it settles at, not the truth, is the real object of study.

How It Actually Works

The squared error against the bootstrapped target is $\frac{1}{2}[U_t - \hat{v}(S_t, \mathbf{w})]^2$ . A true gradient would differentiate through $U_t$ too, since $U_t$ holds $\mathbf{w}$ . The semi-gradient method freezes the target instead, $\partial U_t / \partial \mathbf{w} = 0$ , and steps only along the current guess:

$\mathbf{w}_{t+1} = \mathbf{w}_t + \alpha \, [R_{t+1} + \gamma \hat{v}(S_{t+1}, \mathbf{w}_t) - \hat{v}(S_t, \mathbf{w}_t)] \, \nabla_{\mathbf{w}} \hat{v}(S_t, \mathbf{w}_t)$

Here $\alpha > 0$ is the step size. For linear features this simplifies to $\mathbf{w}_{t+1} = \mathbf{w}_t + \alpha \, \delta_t \, \mathbf{x}(S_t)$ with TD error $\delta_t$ . On-policy with linear features this converges to a fixed point whose error stays within $1/(1 - \gamma)$ of the best linear fit (Tsitsiklis and Van Roy, 1997). Off-policy it can diverge, which is the subject of the deadly triad.

Worked example

Two features, $\gamma = 0.9$ , $\alpha = 0.1$ , weights $\mathbf{w} = [0.5, 1.0]$ . Current features $[1.0, 2.0]$ , next features $[0.5, 1.0]$ , reward $R = 2.0$ .

Current guess: $0.5 \times 1.0 + 1.0 \times 2.0 = 2.5000$ . Next guess: $0.5 \times 0.5 + 1.0 \times 1.0 = 1.2500$ .

Target: $2.0 + 0.9 \times 1.2500 = 3.1250$ . Error: $3.1250 - 2.5000 = 0.6250$ .

Update: $[0.5, 1.0] + 0.1 \times 0.625 \times [1.0, 2.0] = [0.5625, 1.1250]$ . New current guess: $0.5625 + 2.2500 = 2.8125$ , moved $0.3125$ toward the target, exactly $\alpha \, \delta \, \lVert \mathbf{x} \rVert^2 = 0.0625 \times 5$ .

Code

The same step in plain Python:

1w = [0.5, 1.0]2x_now = [1.0, 2.0]3x_next = [0.5, 1.0]4reward, gamma, alpha = 2.0, 0.9, 0.15
6v_now = sum(a * b for a, b in zip(w, x_now))7v_next = sum(a * b for a, b in zip(w, x_next))8target = reward + gamma * v_next9delta = target - v_now10w = [a + alpha * delta * b for a, b in zip(w, x_now)]11
12print(f"v_now {v_now:.4f}")      # -> v_now 2.500013print(f"v_next {v_next:.4f}")    # -> v_next 1.250014print(f"target {target:.4f}")    # -> target 3.125015print(f"delta {delta:.4f}")      # -> delta 0.625016print(f"w [{w[0]:.4f}, {w[1]:.4f}]")  # -> w [0.5625, 1.1250]

Watch Out For

Treating the semi-gradient as a true gradient

The update looks like gradient descent, so teams monitor a loss curve and expect it to fall monotonically. It will not. There is no fixed loss here because every step rewrites the target, so the curve wobbles even in healthy runs. The symptom is a debugging chase for a bug that does not exist.

Fix it by tracking the TD error magnitude and the weight norm instead of a loss, and by keeping data on-policy while you validate the setup.

Going off-policy without a correction diverges

Collect data under one policy while evaluating another, keep bootstrapping, keep the features, and weights can blow up instead of settling. The symptom is a weight norm that climbs round after round toward overflow. This is one leg of the deadly triad.

Fix it by staying on-policy for plain TD, or by switching to a gradient TD method built for the off-policy case.

The Quick Version

Semi-gradient TD(0) learns after every step from $R_{t+1} + \gamma \hat{v}(S_{t+1})$ , its own next guess, instead of waiting for the episode to end.
It freezes the target during differentiation, stepping only along the current features with $\mathbf{w} + \alpha \, \delta \, \mathbf{x}(S_t)$ .
In the worked step, reward 2.0 and next guess 1.25 give target 3.125, error 0.625, and weights [0.5625, 1.125].
On-policy linear TD converges near the best fit, but the same update off-policy can diverge.
It is not gradient descent on a fixed loss, so monitor TD error and weight norms, not a loss curve.