Multi-Agent DDPG (MADDPG)

Each agent acts alone from its own view, but training gives each critic the full scene plus all actions, which steadies learning in continuous team play.

Centralized critic sees joint actions and builds target 5.6 with error 0.60 while the actor uses only its local view to climb with gradient 1.20

Why Does This Exist?

Independent learners chase a moving target. When agent 1 learns, agent 2 changes too, so transition $P(s' \mid s, a_1)$ shifts under your feet. Replay turns stale, Q updates wobble, and continuous control rarely converges.

A single joint controller fixes stationarity but needs global talk at runtime, which robots and game clients cannot afford, and its action space blows up with team size. Multi-agent RL frames the problem. This page covers the CTDE fix for continuous actions: train with global context, deploy with local views only. For the PPO cousin see MAPPO. For factored team values see value decomposition.

The one idea you need first

An actor $\mu(o)$ maps a local view $o$ to an action $a$ . A critic $Q(x, a)$ scores a global state $x$ plus actions. DDPG trains a deterministic actor by climbing the critic slope $\nabla_a Q$ . MADDPG keeps that climb but gives each critic the joint actions.

Think of It Like This

Film room versus live game

Your basketball team studies overhead drone film. The coach sees all ten players and tells you that when your teammate set a high screen you should have cut backdoor. The note is sharp because it uses full context.

In the live game the film room is gone. You play from eye level on your local view $o_1$ only. You still cut at the right time because practice wired the pattern in. Training was centralized, execution is local.

Where it stops: a human coach can still shout from the sideline. In MADDPG the critic is deleted at deployment. Zero talk, full autonomy.

How It Actually Works

With $N$ agents, call $o_i$ the local view, $a_i$ the continuous action, $x$ the global state, $a = (a_1, \dots, a_N)$ the joint action. Agent $i$ owns actor $\mu_i(o_i)$ and critic $Q_i(x, a)$ .

Conditioning on $a$ restores stationarity because $P(x' \mid x, a)$ does not shift when policies change. Only the action distribution shifts, which replay handles.

Critic target

Critic $i$ minimizes Bellman square error with target actors $\mu'_j$ and target critic $Q'_i$ :

$y_i = r_i + \gamma Q'_i(x', a'_1, \dots, a'_N)$

$a'_j = \mu'_j(o'_j)$

$\mathcal{L} = (Q_i(x, a) - y_i)^2$

Rewards $r_i$ stay individual, so cooperative, competitive, and mixed motives all fit.

Actor climb

The actor climbs through its own critic slope only:

$\nabla J = \mathbb{E}[\nabla_\theta \mu_i(o_i) \cdot \nabla_{a_i} Q_i(x, a)]$

Peer actions enter as fixed context. You do not differentiate through them.

Worked example

Two agents, $\gamma = 0.90$ . State $x = [1.0, 2.0]$ , views $o_1 = 1.0$ and $o_2 = 2.0$ , actions $a_1 = 0.50$ and $a_2 = -0.50$ , reward $r_1 = 2.0$ , next $x' = [1.2, 1.8]$ .

Target actions $a'_1 = 0.60$ and $a'_2 = -0.40$ . Target critic $Q'_1(x', a') = 4.00$ .

$y_1 = 2.0 + 0.90 \times 4.00 = 5.60$

Current critic $Q_1(x, a) = 5.00$ .

$\delta_1 = 5.60 - 5.00 = 0.60$

$\mathcal{L} = 0.60^2 = 0.36$

Critic slope $\nabla_{a_1} Q_1 = 1.50$ and actor sensitivity $\nabla_\theta \mu_1 = 0.80$ .

$\nabla J = 0.80 \times 1.50 = 1.20$

Actor weights move up by rate times $1.20$ . The diagram shows $5.60$ , $0.60$ , and $1.20$ on the same path.

Code

This replays the critic and actor arithmetic from the worked example.

1def bellman_target(r: float, gamma: float, q_next: float) -> float:2    """y = r + gamma * Q next."""3    return r + gamma * q_next4
5
6def policy_gradient(dmu_dtheta: float, dq_da: float) -> float:7    """Chain rule through own action only."""8    return dmu_dtheta * dq_da9
10
11if __name__ == "__main__":12    y1 = bellman_target(2.0, 0.90, 4.0)13    print(f"y1 = {y1:.2f}")  # -> y1 = 5.6014    delta = y1 - 5.0015    print(f"delta = {delta:.2f}")  # -> delta = 0.6016    print(f"loss = {delta**2:.4f}")  # -> loss = 0.360017    g = policy_gradient(0.80, 1.50)18    print(f"grad = {g:.2f}")  # -> grad = 1.20

Watch Out For

Critic input blows up with team size

Each critic takes all actions, so with 10 plus agents the input grows and credit assignment blurs. Symptom: sample hunger spikes and actor gradients swing wildly as team size grows.

Attend to neighbors instead of concatenating everyone, as in attention critics, or switch to factored team values in pure cooperative tasks. Do not just widen the critic and hope. The joint space is the bottleneck.

Leaking global state into the actor

It is tempting to pass $x$ or peer actions to the actor because training looks faster. Symptom: lab scores shine, deployment fails because those inputs are missing and the policy never learned a local mapping.

Keep the actor signature strictly $\mu_i(o_i)$ . Test with global inputs masked before you report results. If performance collapses under masking, the actor cheated.

The Quick Version

MADDPG trains centralized critics on global state plus joint actions but deploys actors on local views only.
Joint conditioning restores stationarity because $P(x' \mid x, a)$ stays fixed while policies change.
In the worked step target $y_1 = 5.60$ , error is $0.60$ , loss is $0.36$ , and actor gradient is $1.20$ .
Each actor climbs only its own critic slope $\nabla_{a_i} Q_i$ , with peer actions as fixed context.
Individual rewards support cooperative, competitive, and mixed play, but large teams need attention or factored critics.