Target Networks in Deep RL

A Q-network that learns from targets it generates itself chases its own tail, so deep Q-learning grades against a frozen copy that only syncs up every few thousand steps.

With one network the TD target for the first transition drifts from 2.90 to 4.12 in two steps, while a frozen target copy holds it exactly at 2.90

Why Does This Exist?

Supervised regression has fixed labels, so gradient descent descends a fixed landscape. Deep Q-learning replaced the label with a bootstrapped target built from the same weights being trained:

y_t = R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a'; \theta)

Every gradient step moves $\theta$ , which moves the next target, which changes what the following step descends. Because one update shifts predictions across many states at once, this feedback can compound into runaway overestimation. It is one leg of the deadly triad and the reason naive deep Q-learning diverges on problems where tables converge.

The fix, introduced with deep Q-networks, is a second copy of the weights $\theta^-$ that generates targets while staying frozen, turning each stretch of training into ordinary regression against fixed labels. This page covers the frozen-target mechanism and the two sync schedules. The network it stabilizes is taught in deep Q-networks, and the paired trick of shuffling past transitions is experience replay.

Think of It Like This

Grading exams against an answer key that students keep editing

A class takes weekly quizzes, and the teacher grades each quiz against an answer key. In the sane version the key is locked in a drawer all week: students learn, the key stays put, scores mean something.

In the single-network version the answer key is a shared document the students can edit, and every study session quietly rewrites a few answers to match what the class already believes. Next week the class studies toward the edited key, edits it again, and scores climb while nobody learns anything. The grades drift upward on their own momentum.

A target network is the locked drawer. The class studies against a frozen key for a month, and only then does the teacher publish a new edition based on what the class has actually mastered. Scores still rise, but now they track learning instead of self-flattery.

Where it stops: students conspire, but weights do not intend anything. The drift is pure mechanics, generalization leaking each update into the target, not anyone gaming the test.

How It Actually Works

Two identical networks run side by side. The online network $Q(s, a; \theta)$ is updated every step and picks actions. The target network $Q(s, a; \theta^-)$ never receives gradients; it only produces the regression target:

y_t = R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a'; \theta^-)

With $\theta^-$ fixed, the loss is a plain supervised objective in $\theta$ , and its gradient never flows into the target branch.

The two sync schedules

Hard updates freeze $\theta^-$ for $C$ steps, typically $1{,}000$ to $10{,}000$ , then copy $\theta^- \leftarrow \theta$ wholesale. Targets are perfectly stationary between copies at the cost of a small shock at each sync. Soft updates, also called Polyak averaging, blend a little of the online weights in after every step: $\theta^- \leftarrow \tau\theta + (1 - \tau)\theta^-$ with $\tau$ around $0.005$ . There is no shock, but the target always trails slightly behind. Hard copies are the DQN default; soft blends are standard in continuous control with DDPG, TD3, and SAC.

Worked example

Linear model $Q = \theta x$ with all features $1.0$ , $\gamma = 0.9$ , $\alpha = 0.5$ , starting $\theta_0 = 1.0$ . Two transitions: $(s_0 \to s_1, R = 2.0)$ then $(s_1 \to s_2, R = 1.0)$ .

Single network, step 0: prediction $1.0$ , live target $2.0 + 0.9 \times 1.0 = 2.90$ , error $1.90$ , so $\theta_1 = 1.0 + 0.5 \times 1.90 = 1.95$ . Step 1: prediction $1.95$ , live target $1.0 + 0.9 \times 1.95 = 2.755$ , error $0.805$ , so $\theta_2 = 1.95 + 0.5 \times 0.805 = 2.3525$ . Re-evaluate the first transition now: $2.0 + 0.9 \times 2.3525 = 4.1173$ . The target drifted $+1.2173$ , over 40 percent, in two steps.

Frozen target copy $\theta^- = 1.0$ , step 0: same target $2.90$ , same $\theta_1 = 1.95$ , target weights unmoved. Step 1: prediction $1.95$ but frozen target $1.0 + 0.9 \times 1.0 = 1.90$ , error $-0.05$ , so $\theta_2 = 1.95 - 0.025 = 1.925$ . Re-evaluate the first transition: still $2.90$ . Drift $0.0000$ across the whole frozen window; the copy syncs only when the schedule says so.

Code

1def case_single_network() -> tuple[float, float]:2    """Two updates, target rebuilt from live weights each time."""3    theta: float = 1.04    theta += 0.5 * ((2.0 + 0.9 * theta) - theta)5    theta += 0.5 * ((1.0 + 0.9 * theta) - theta)6    drifted_target = 2.0 + 0.9 * theta7    return theta, drifted_target - 2.908
9
10def case_frozen_target() -> tuple[float, float]:11    """Two updates against a frozen copy fixed at 1.0."""12    theta: float = 1.013    frozen: float = 1.014    theta += 0.5 * ((2.0 + 0.9 * frozen) - theta)15    theta += 0.5 * ((1.0 + 0.9 * frozen) - theta)16    return theta, (2.0 + 0.9 * frozen) - 2.9017
18
19live_theta, live_drift = case_single_network()20frozen_theta, frozen_drift = case_frozen_target()21print(f"{live_theta:.4f} {live_drift:.4f}")  # -> 2.3525 1.217322print(f"{frozen_theta:.4f} {frozen_drift:.4f}")  # -> 1.9250 0.0000

Watch Out For

Syncing so often the frozen copy thaws

If the hard-copy interval $C$ is tiny, tens of steps, the target moves almost every update and the feedback loop returns wearing a disguise. The symptom is periodic loss spikes spaced exactly $C$ steps apart with no lasting downward trend: each sync yanks the rug, the online net re-fits, repeat. People usually blame the learning rate because the spikes look like instability from oversized steps.

The fix is to scale $C$ with task horizon and batch turnover, starting near $1{,}000$ gradient steps for small tasks, and to confirm by checking that loss spikes at syncs shrink as training proceeds. If spikes dominate the curve, raise $C$ or switch to soft Polyak blending, which has no sync shocks at all.

Freezing so long the agent studies an obsolete exam

The mirror failure: $C$ in the hundreds of thousands, or $\tau$ near zero, pins the target to ancient values. The online network overfits to stale horizons, TD errors stop informing, and wall-clock time burns with nothing learned. The symptom is a loss that flatlines early while evaluation scores refuse to move, which looks exactly like a capacity problem and invites uselessly bigger networks.

The fix is to watch the gap between online and target predictions on a held-out batch of transitions. If the gap grows while scores stall, the target is stale: shorten $C$ or raise $\tau$ toward $0.005$ until the gap tracks learning instead of leading it.

The Quick Version

Bootstrapped targets built from live weights move with every update, which can compound into runaway overestimation.
A target network is a frozen copy that produces regression targets while receiving no gradients.
Hard updates copy weights wholesale every few thousand steps; Polyak averaging blends a fraction in after each step.
In the worked example the live target drifts 2.90 to 4.12 in two steps while the frozen target stays at 2.90.
Too-frequent syncing resurrects the feedback loop; too-rare syncing strands learning on stale targets.
Target networks plus experience replay are the two stabilizers that make deep Q-learning work.