Double Q-Learning

One table picks the winner, a second table prices it. Splitting pick and price stops noise from looking like value.

One table picks action a1 at 0.40 while the second table prices it at minus 0.15, so the update uses minus 0.15 not 0.40

Why Does This Exist?

Q-learning backs up the best next value:

Y = r + \gamma \max_a Q(s^\prime, a)

When estimates wobble, max keeps the luckiest wobble. Expected max sits above max of means:

\mathbb{E}[\max_a Q(s, a)] \ge \max_a \mathbb{E}[Q(s, a)]

That gap is maximization bias. With many noisy actions the agent chases phantom highs, values drift up, and learning slows. Deep Q-networks inherit the same flaw through the same max.

Double Q-learning splits the job. One estimator picks, the other prices. Noise that won the pick need not win the price. This page covers the split. Solo Q details live in Q-learning. The deep twin target version lives in double DQN.

The one idea you need first

An action value $Q(s, a)$ is expected discounted return after taking $a$ in $s$ . Learning rate $\alpha$ blends old guess with new target. Discount $\gamma$ weights near rewards over far ones. All updates below use those three symbols only.

Think of It Like This

Two separate house appraisers

You want one rental house. Ask one keen broker to both choose the best listing and name its rent, and the pick will be the listing whose rent guess happened to spike highest. Pick noise and price noise are the same number.

Hire two independent appraisers instead. Appraiser A walks all houses and names one winner. Appraiser B, who never saw the shortlist contest, prices only that winner. A lucky spike in A rarely repeats in B, so the price stays honest.

Where it stops: houses have true market rents. Q values have no fixed tag; both tables keep moving as the policy improves.

How It Actually Works

Keep $Q_1$ and $Q_2$ . Flip a fair coin each step. On heads update $Q_1$ with $Q_2$ pricing the pick:

A^\ast = \arg\max_a Q_1(s^\prime, a)

Q_1(s, a) \leftarrow Q_1(s, a) + \alpha [r + \gamma Q_2(s^\prime, A^\ast) - Q_1(s, a)]

On tails swap roles:

B^\ast = \arg\max_a Q_2(s^\prime, a)

Q_2(s, a) \leftarrow Q_2(s, a) + \alpha [r + \gamma Q_1(s^\prime, B^\ast) - Q_2(s, a)]

Act with the mean $(Q_1 + Q_2)/2$ . Because each table learned from different samples, the pricing table is roughly independent of the pick, so its value is centered instead of systematically high.

Worked example

Next state $s^\prime$ has three actions with true mean $0.00$ each. Current noisy tables read:

$Q_1 = [+0.40, -0.20, +0.10]$
$Q_2 = [-0.15, +0.30, -0.05]$

Solo Q with $Q_1$ backs up:

\max_a Q_1 = \max(+0.40, -0.20, +0.10) = +0.40

A phantom gain of $+0.40$ where truth is $0.00$ .

Double Q picks with $Q_1$ and prices with $Q_2$ . Pick is $A^\ast = a_1$ because $+0.40$ tops the row. Price is $Q_2(s^\prime, a_1) = -0.15$ . Backup uses $-0.15$ , not $+0.40$ . Gap closed is $0.40 - (-0.15) = 0.55$ . Reverse direction works the same: $Q_2$ would pick $a_2$ at $+0.30$ while $Q_1$ prices it at $-0.20$ . That pair is what the diagram draws.

Code

This snippet repeats the pick and price logic with plain lists. No packages needed.

1def solo_target(q: list[float]) -> float:2    return max(q)3
4
5def double_target(q_pick: list[float], q_price: list[float]) -> tuple[int, float]:6    best = max(range(len(q_pick)), key=lambda i: q_pick[i])7    return best, q_price[best]8
9
10q1 = [0.40, -0.20, 0.10]11q2 = [-0.15, 0.30, -0.05]12print(round(solo_target(q1), 2))  # -> 0.413idx, price = double_target(q1, q2)14print(idx)  # -> 015print(round(price, 2))  # -> -0.1516idx2, price2 = double_target(q2, q1)17print(idx2)  # -> 118print(round(price2, 2))  # -> -0.2

Solo target prints $0.40$ . Double pick is index $0$ priced at $-0.15$ . The reverse pick is index $1$ priced at $-0.20$ . Both remove the phantom high.

Watch Out For

Thinking two tables need twice the episodes

Newcomers halve updates per table and assume they must collect twice the play. Samples are shared. Each transition updates one table picked at random, so environment cost per step is unchanged. Only memory doubles.

Keep the coin fair at $0.50$ , update exactly one table per step, and act on the mean. If one table lags far behind the other, the coin or the learning rate drifted, not the data budget.

Sharing one target network across both deep heads

In tabular form independence comes from separate samples. In deep form teams often share the encoder or copy one target into both heads. Coupling returns and the bias creeps back.

Keep targets separate, sync them at different times or with different seeds, and log mean gap $\max Q_1 - Q_2(\arg\max Q_1)$ . When that gap stays positive for many batches, the heads share too much.

The Quick Version

Max over noisy values drifts high because expected max tops max of means.
Double Q keeps $Q_1$ and $Q_2$ , picks with one and prices with the other.
Worked numbers: solo target $+0.40$ , double target $-0.15$ , gap closed $0.55$ .
Acting uses the mean of both tables with one random table updated per step.
Sample cost matches solo Q; only memory for the second table is extra.