Independent Q-Learning

Let each agent run its own Q-learning and pretend teammates are weather. It scales with zero messaging but the ground shifts as everyone learns at once.

Two separate Q tables chase joint payoffs 10 and 5, but peer exploration drags Q of A from 2.0 to 1.6 while safe B sits at 1.0

Why Does This Exist?

Joint control blows up fast. With $N$ agents and $|A|$ moves each, the joint action count is $|A|^N$ . Two agents with 5 moves mean 25 joint moves. Ten agents mean almost ten million. A central brain that picks every joint move cannot scale, and in drone fleets or warehouses you cannot rely on instant messaging at run time.

Multi-agent RL frames the options, and Markov games give the formalism. Independent Q-learning from Tan in 1993 is the simplest decentralized answer. Each agent $i$ keeps its own $Q_i(s_i, a_i)$ on its local view $s_i$ and runs plain Q-learning. Teammates are treated as part of the scenery.

That simplicity breaks theory. Because peers update too, each agent faces a moving target. This page shows the update, the drift, and the classic trap where teams abandon the best joint move for a safe one. The fix family with shared training lives in value decomposition networks.

The one idea you need first

Single agent Q-learning assumes fixed dynamics $P(s' \mid s, a)$ . The Bellman backup $r + \gamma \max Q(s', a')$ converges because the world stays still while you learn. Multi-agent learning removes that stillness.

Think of It Like This

Two cooks who cannot taste together

Two cooks share one oven but work in separate kitchens with no phone. Each adjusts heat by watching only their own tray. Cook one raises heat because their half looks pale. Cook two lowers heat because their half looks dark. Both act sensibly alone, yet the shared oven swings wildly.

Neither cook can tell if a burnt edge came from their own knob or the other knob. So each blames their own last move and gets cautious. Soon both pick the lukewarm setting that never burns, even though a hotter joint setting would bake perfectly if they held it together.

Where the analogy stops: cooks share culture and can agree on a recipe in advance. These agents start with no shared plan, and their exploration itself is what shakes the oven.

How It Actually Works

Agent $i$ sees $s_i$ , picks $a_i$ , gets $r_i$ . It stores $Q_i(s_i, a_i)$ and updates with temporal difference error $\delta$ :

Q_i(s_i,a_i) \leftarrow Q_i(s_i,a_i) + \alpha[r_i + \gamma \max_{a'_i} Q_i(s'_i,a'_i) - Q_i(s_i,a_i)]

Symbols: $\alpha$ is the step size, $\gamma$ is the discount, $s'_i$ is the next local view. Joint actions of others, $a_{-i}$ , never appear. That is the point and the wound.

True dynamics depend on the joint move $\mathbf{a} = (a_1, \dots, a_N)$ :

P(s' \mid s, a_i) = \sum_{a_{-i}} \mathcal{P}(s' \mid s, a_i, a_{-i}) \prod_{j \ne i} \pi_j(a_j \mid s)

As peer policies $\pi_j$ change, the left side shifts every iteration. Replay buffers go stale, because old tuples carry rewards earned under dead peer behavior.

Worked example

Two agents pick A or B. Payoffs are $(A,A) = 10.0$ , $(B,B) = 5.0$ , mismatch $= 0.0$ . Step size $\alpha = 0.2$ , $\gamma = 0$ . Start $Q_1 = \{A: 0.0, B: 0.0\}$ , same for $Q_2$ .

Step 1, both play A and earn 10.0:

Q_1(A) = 0.0 + 0.2 \times 10.0 = 2.00

Same for $Q_2(A)$ . Good start toward the best joint move.

Step 2, agent 1 plays A while agent 2 explores B. Joint is $(A,B)$ , reward 0.0:

Q_1(A) = 2.00 + 0.2 \times (0.0 - 2.00) = 1.60

Q_2(B) = 0.0 + 0.2 \times 0.0 = 0.00

Agent 1 loses 0.40 for doing the right thing at the wrong joint moment.

Step 3, both play B and earn 5.0:

Q_1(B) = 0.0 + 0.2 \times 5.0 = 1.00

Now A looks risky because any peer wobble zeroes it, while B pays steadily. Teams slide to $(B,B)$ even though 10 beats 5. That slide is relative overgeneralization.

Hysteretic Q-learning softens the blow with a small step $\beta = 0.05$ on negative errors:

Q_1(A) = 2.00 + 0.05 \times (-2.00) = 1.90

Same miscoordination now costs 0.10 instead of 0.40, so A survives longer.

Code

This snippet replays the three steps and the hysteretic guard.

1"""Replay the 2-agent coordination trap from the page."""2
3alpha = 0.24beta = 0.055
6q1a = 0.07q1a = q1a + alpha * (10.0 - q1a)8print(round(q1a, 2))9# -> 2.010
11q1a_std = q1a + alpha * (0.0 - q1a)12print(round(q1a_std, 2))13# -> 1.614
15q1a_hys = q1a + beta * (0.0 - q1a)16print(round(q1a_hys, 2))17# -> 1.918
19q1b = 0.0 + alpha * (5.0 - 0.0)20print(round(q1b, 2))21# -> 1.0

Watch Out For

Replaying stale team behavior

Deep independent learners store $(s, a_i, r_i, s')$ and replay it weeks later in training time. The reward and next state depended on old peer moves that no longer happen. The net fits ghosts. The symptom is training that improves, then jitters or collapses as the buffer fills with outdated joint contexts.

Tag transitions with time or peer behavior, shorten buffers for tight coupling, or move to centralized training with decentralized execution. If agents rarely interact, plain replay can still pass, so match the fix to the coupling.

Settling for the safe joint move

Early tries at $(A,A)$ get punished by peer exploration, so $Q(A)$ sinks and $Q(B)$ looks steady. The team locks $(B,B)$ at payoff 5 and stops trying for 10. The symptom is stable but mediocre coordination that never recovers even with more training.

Use optimistic guards like hysteretic updates, longer exploration with restarts, or a mixing method that sees joint actions in training. Check the payoff gap first: if the best joint pays only a little more, safe may be rational.

The Quick Version

Independent Q-learning runs one Q update per agent with no peer messaging.
Peer learning makes dynamics non stationary, so Bellman guarantees and fresh replay both fail.
In this page payoff 10 joint action falls when exploration zeroes it, with $Q_1(A)$ moving $2.00$ to $1.60$ .
Safe payoff 5 joint action then sticks at $Q_1(B) = 1.00$ through relative overgeneralization.
Hysteretic steps with $\beta = 0.05$ keep $Q_1(A)$ at $1.90$ and protect promising moves longer.