Double DQN (DDQN)

Let one network point at the best action and a second, slower network say what it is worth, so lucky noise cannot vote for itself twice.

The online net picks action a1 while the target net scores it at minus 0.20, giving 0.82 instead of the coupled 1.54

Why Does This Exist?

Deep Q-networks learn from noisy estimates. Each update samples a minibatch from experience replay, backpropagates through a network, and bootstraps from its own predictions. Write the estimate as truth plus noise:

Q(s, a; \theta) = q_{\star}(s, a) + \epsilon(s, a)

Here $q_{\star}$ is the true value, $\theta$ holds the online weights, and $\epsilon$ is zero mean noise. The standard target takes a max over noisy values:

y^{\text{DQN}} = r + \gamma \max_{a} Q(s^{\prime}, a; \theta^{-})

Here $r$ is the reward, $\gamma$ the discount, $s^{\prime}$ the next state, and $\theta^{-}$ the target network weights. The max keeps whichever noise draw is largest, so its expectation sits above the true max. That gap is maximization bias, and the twin tabular version is covered in maximization bias and double Q-learning. Over thousands of steps the inflation compounds and policies wobble.

Double DQN (van Hasselt, Guez, and Silver, 2015) splits the job across the two networks you already keep. The online net picks, the target net scores:

y^{\text{DDQN}} = r + \gamma Q(s^{\prime}, \arg\max_{a} Q(s^{\prime}, a; \theta); \theta^{-})

No third network, no extra memory. This page shows why the split removes the bias and works the arithmetic on one transition.

Think of It Like This

A scout and a valuer at an auction

You send two people to an auction. The scout walks the floor fast and points at the lot that looks most promising. The valuer arrives later, inspects only that lot, and names a price from a separate notebook.

If one person does both jobs, trouble follows. They fall in love with a lot because their first glance was generous, then they quote that same generous glance as the price. Every ledger entry drifts upward.

Split the roles and the failure breaks. The scout can still point at a dud, but the valuer has no reason to share that exact misreading, so the price stays honest. The analogy stops at the sync: in Double DQN the valuer copies the scout's notebook every few thousand steps, so the two opinions stay related but never identical at decision time.

How It Actually Works

Name the parts. Selection asks which action looks best. Evaluation asks what that action is worth. Standard DQN lets $\theta^{-}$ answer both. Double DQN lets $\theta$ select and $\theta^{-}$ evaluate.

Concretely, for next state $s^{\prime}$ :

Select with the online net: $a^{\star} = \arg\max_{a} Q(s^{\prime}, a; \theta)$ .
Evaluate with the target net: $Q(s^{\prime}, a^{\star}; \theta^{-})$ .
Train $\theta$ toward $y^{\text{DDQN}}$ with the usual squared error, then copy $\theta$ into $\theta^{-}$ every $C$ steps or blend slowly toward it.

The three designs compare like this:

Standard DQN selects with $\theta^{-}$ and evaluates with $\theta^{-}$ , so coupled noise inflates the target.
Tabular Double Q-learning keeps two tables and flips a coin each step for who selects and who evaluates, which is unbiased but needs symmetric updates.
Double DQN always selects with $\theta$ and evaluates with $\theta^{-}$ , reusing the target net, which removes the systematic lift at zero extra cost.

Worked example

Both actions are truly worth $0.0$ at $s^{\prime}$ , so any spread is pure noise. Reward $r = 1.0$ and $\gamma = 0.9$ , giving a true target of $1.00$ . Suppose the nets output:

Online $\theta$ : $Q = [+0.50, -0.30]$ for $[a_1, a_2]$ .
Target $\theta^{-}$ : $Q = [-0.20, +0.60]$ for $[a_1, a_2]$ .

Standard DQN selects and scores with $\theta^{-}$ . It picks $a_2$ at $+0.60$ and builds $1.0 + 0.9 \times 0.60 = 1.54$ , which overshoots truth by $0.54$ .

Double DQN selects with $\theta$ . It picks $a_1$ at $+0.50$ , then asks $\theta^{-}$ what $a_1$ is worth: $-0.20$ . The target becomes $1.0 + 0.9 \times (-0.20) = 1.0 - 0.18 = 0.82$ . One draw lands low, the next may land high, but the average across draws sits near $1.00$ instead of floating above it. That centering is the whole trick, and it stacks well with prioritized experience replay.

Code

This snippet rebuilds both targets from the worked numbers. No training loop, just the two selection and evaluation paths.

1def dqn_target(reward: float, gamma: float, q_target: list[float]) -> float:2    best = max(range(len(q_target)), key=lambda i: q_target[i])3    return reward + gamma * q_target[best]4
5
6def ddqn_target(reward: float, gamma: float, q_online: list[float], q_target: list[float]) -> float:7    best = max(range(len(q_online)), key=lambda i: q_online[i])8    return reward + gamma * q_target[best]9
10
11def main() -> None:12    r, gamma = 1.0, 0.913    q_online = [0.50, -0.30]14    q_target = [-0.20, 0.60]15
16    y_dqn = dqn_target(r, gamma, q_target)17    y_ddqn = ddqn_target(r, gamma, q_online, q_target)18
19    print(f"DQN target: {y_dqn:.2f}")  # -> DQN target: 1.5420    print(f"DDQN target: {y_ddqn:.2f}")  # -> DDQN target: 0.8221    print(f"truth: 1.00, DQN error: {y_dqn - 1.0:+.2f}")  # -> truth: 1.00, DQN error: +0.5422    print(f"truth: 1.00, DDQN error: {y_ddqn - 1.0:+.2f}")  # -> truth: 1.00, DDQN error: -0.1823
24
25if __name__ == "__main__":26    main()

Watch Out For

Building a third network for Double DQN

Tabular Double Q-learning uses two symmetric estimators, so teams new to the deep version create two online nets plus a target and wire an alternating update. That doubles memory and compute for no gain.

You already own both roles. Freeze the target copy, let the online net pick every action, let the target score it, and sync on schedule. If your code constructs more than two Q-nets for DDQN, delete one.

The Quick Version

Noisy estimates plus a max give systematic overestimation, since the max keeps the luckiest noise draw.
Standard DQN selects and evaluates with the target net $\theta^{-}$ , which locks the bias in.
Double DQN selects with the online net $\theta$ and evaluates with $\theta^{-}$ , at zero extra networks.
On the worked transition with truth $1.00$ , coupled selection gives $1.54$ while decoupled selection gives $0.82$ .
Across draws the decoupled mean sits near truth instead of above it, which steadies training.