Value Decomposition Networks (VDN)

Split the team score into per-agent utilities that add up to the total, so each agent can pick its own best move with no radio contact at game time.

Two agents scoring 3.50 and 4.00 sum to a team value of 7.50, and a shared error of 2.60 trains both utilities at once

Why Does This Exist?

A cooperative team shares one reward but runs on separate eyes and legs. Two naive designs both fail. Independent Q-learning lets each agent learn alone, so teammates look like weather: random, shifting, and impossible to credit, and slackers surf on teammates' goals. One giant joint Q-function is principled but explodes: 8 agents with 5 moves each means $5^8 = 390{,}625$ joint actions to maximize every step, plus perfect live communication at deployment.

VDN (Sunehag and colleagues, 2017) takes the middle path. Model the team value as a plain sum of per-agent utilities, $Q_{tot} = \sum_i Q_i$ , train the sum centrally against the shared reward, then deploy each $Q_i$ alone. Addition is what makes decentralized greed line up with the team optimum.

The one idea you need first

Multi-agent RL frames the team game: shared state, private observations, one team reward, and the CTDE mantra of training together while executing apart. VDN is the simplest CTDE value method. QMIX and QTRAN relax its one big assumption, so read them as the sequel.

Think of It Like This

Splitting one shared restaurant bill three ways

Three friends share a banquet and one bill for $150 arrives. A lazy treasurer either charges everyone $150 or guesses blindly, so the salad eater subsidizes the steak and learns salads are ruinous. A control freak memorizes every possible three-way order combination before anyone may bite.

VDN is the itemized tracker. Each diner keeps a personal tab ( $Q_1, Q_2, Q_3$ ) with one accounting rule: the tabs must sum to the real bill, $Q_{tot} = Q_1 + Q_2 + Q_3 = 150$ . The gap between the expected total and the actual receipt corrects every tab at once. Next dinner, each friend orders locally with confidence that picking their own best dish serves the table.

Where the analogy stops: restaurant prices really do add up, while agents interact: two robots lifting one couch are worth more together than apart. When teamwork breaks addition, VDN underfits and the monotonic mixers in the sequel exist for exactly that case.

How It Actually Works

Each agent $i$ sees only its own action-observation history $\tau_i$ and owns a utility net $Q_i(\tau_i, a_i)$ . The team value is their sum:

Q_{tot}(\boldsymbol{\tau}, \mathbf{a}) = \sum_{i=1}^N Q_i(\tau_i, a_i)

Why greedy locals make a greedy team

Addition is strictly increasing in each part ( $\partial Q_{tot} / \partial Q_i = 1 > 0$ ), so the joint argmax splits apart:

\arg\max_{\mathbf{a}} Q_{tot} = \big(\arg\max_{a_1} Q_1,\ \ldots,\ \arg\max_{a_N} Q_N\big)

This is the Individual-Global-Max (IGM) property: each agent maximizing its own utility maximizes the team total, with zero communication. Training minimizes the usual squared Bellman error on $Q_{tot}$ with target $y = r + \gamma \sum_i \max_{a_i'} Q_i'$ , which costs $O(N \cdot |\mathcal{A}|)$ instead of the joint $O(|\mathcal{A}|^N)$ . The shared TD error flows back into every utility with unit gradient, so one team surprise teaches all agents at once.

Worked example

Two agents, moves $\{A, B\}$ , $\gamma = 0.90$ . Current utilities: $Q_1: A = 3.50, B = 1.00$ so agent 1 picks $A$ ; $Q_2: A = 2.50, B = 4.00$ so agent 2 picks $B$ . Joint action $(A, B)$ , no coordination needed.

Team value: $Q_{tot} = 3.50 + 4.00 = 7.50$ . The team acts, gets shared reward $r = 2.00$ .

Next-state target nets read $Q_1': A = 4.00, B = 2.00$ (max $4.00$ ) and $Q_2': A = 3.00, B = 5.00$ (max $5.00$ ). Centralized target: $y = 2.00 + 0.90 \times (4.00 + 5.00) = 2.00 + 8.10 = 10.10$ .

TD error: $\delta = 10.10 - 7.50 = 2.60$ . Loss: $\tfrac{1}{2} \times 2.60^2 = 3.38$ . Both utilities get pushed up by the same $+2.60$ signal through the unit sum gradient.

Code

This snippet replays the worked team step with plain arithmetic. Only numpy is used.

1import numpy as np2
3
4def team_target(reward: float, gamma: float, next_maxes: list[float]) -> float:5    """Centralized VDN target: r + gamma * sum of per-agent next maxima."""6    return reward + gamma * float(np.sum(next_maxes))7
8
9actions = ["A" if 3.50 > 1.00 else "B", "B" if 4.00 > 2.50 else "A"]10print(actions)  # -> ['A', 'B']11q_tot = 3.50 + 4.0012print(round(q_tot, 2))  # -> 7.513y = team_target(2.00, 0.90, [4.00, 5.00])14print(round(y, 2))  # -> 10.115td = y - q_tot16print(round(td, 2))  # -> 2.617print(round(0.5 * td ** 2, 2))  # -> 3.38

Decentralized argmax picks $(A, B)$ , the sum gives 7.5, and the shared error of 2.6 prices the loss at 3.38.

Watch Out For

Forcing additive values onto a coordination game

VDN can only rank an agent's moves independently of teammates, because $Q_{tot} = Q_1 + Q_2$ has no interaction term. In a pure coordination task where $(A, A)$ pays $+8$ , $(B, B)$ pays $+6$ , and mismatches pay $-12$ , no additive split reproduces the matrix: fitting it underestimates the $(A, A)$ peak and the team settles on the worse equilibrium or oscillates. Symptom: training loss looks fine while the deployed team miscoordinates on exactly the states that need synchrony.

Upgrade to QMIX and QTRAN when payoffs hinge on matching. If joint-action plots show strong off-diagonal penalties, VDN was the wrong factorization from the start.

Training on local rewards and calling it VDN

The sum factorization is trained against the shared team reward with joint histories in one replay buffer. Swapping in per-agent rewards to "speed things up" breaks the accounting identity the IGM proof rests on: utilities stop summing to anything meaningful and decentralized greed drifts. Symptom: each agent improves its own metric while the team score stagnates.

Keep one shared reward and centralized training, even if execution is fully local. If only local rewards exist, you have a different problem (credit assignment from scratch), not a VDN shortcut.

The Quick Version

VDN writes the team value as a sum of per-agent utilities, $Q_{tot} = \sum_i Q_i$ , trained centrally on the shared reward.
The IGM property makes local argmax equal the team argmax, so deployment needs no communication.
Target computation drops from $O(|\mathcal{A}|^N)$ to $O(N \cdot |\mathcal{A}|)$ because the max splits across the sum.
One shared TD error (2.60 in the example) trains every utility at once through unit gradients.
Additive values cannot express coordination payoffs; hand those teams to QMIX or QTRAN.