Multi-Agent PPO (MAPPO)

One shared policy controls every agent from local views, while one central critic judges the team from full state. Clipped updates keep the team moving together.

A shared MAPPO actor reads local views while a central critic scores full state, and clipping at 1.20 caps the update at 3.30

Why Does This Exist?

Independent policy gradients fail on teams. Each agent treats teammates as noise, so from any one agent view the world keeps shifting as peers change. Advantage estimates swing, and small steps by many agents compound into a large joint jump.

MAPPO answers with centralization at train time and independence at run time. Training sees full state through one critic. Execution uses only local views through one shared actor. Clipping bounds each step. Yu and coauthors showed in 2022 that this plain recipe matches far more complex off policy team methods on StarCraft unit tasks, football, and Hanabi.

This page covers the team PPO loop. Single agent clipping lives in trust region and proximal methods. Team framing lives in multi-agent RL. Value factorization alternatives live in value decomposition networks.

The one idea you need first

A policy gradient raises the chance of actions that beat baseline. PPO keeps the new policy near the old one by clipping the probability ratio $r(\theta)$ to $[1 - \epsilon, 1 + \epsilon]$ . With $\epsilon = 0.20$ the ratio can move only between $0.80$ and $1.20$ . That bound is the whole stabilizer MAPPO reuses per agent.

Think of It Like This

One playbook, many players, one coach with the full field view

A youth football squad shares one playbook. Each child on the pitch sees only nearby players and the ball. That is the local view $o_i$ . Each child decides alone, with no time to ask the bench. That is the decentralized actor.

One coach stands on a tower and sees all 22 players. The coach does not shout moves during play. After the match the coach grades each choice against the full field picture. That grade is the central critic $V(s)$ . Practice changes are small by rule, at most one small tweak per drill, so the team keeps shape. That is clipping.

Where it stops: children differ in speed and skill. MAPPO agents with shared weights are identical copies told apart only by an identity tag in the input.

How It Actually Works

There are $N$ agents. Agent $i$ sees local view $o_i$ plus a one hot tag for its identity. One actor $\pi_\theta(a_i \mid o_i)$ with weights $\theta$ serves all agents. One critic $V_\phi(s)$ with weights $\phi$ reads full state $s$ during training only. $r_t$ is shared team reward, $\gamma$ discounts the future, $\lambda$ trades bias for variance in advantage estimation.

Rollouts collect $N$ times more steps per tick than one agent, because every teammate writes into the same buffer. Advantages come from the central critic with generalized advantage estimation. Actors update with the clipped PPO loss averaged over agents and time. The critic fits return targets with its own clipped loss.

Central advantage from full state

Define the one step residual from the central critic:

\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)

In words: reward plus discounted next value minus current value. Positive means the team did better than the critic thought. The advantage aggregates residuals:

\hat{A}_t = \sum_{l \ge 0} (\gamma \lambda)^l \delta_{t+l}

Because $V_\phi$ sees $s$ and not just $o_i$ , peer moves are in the baseline instead of appearing as noise.

Clipped team update

Let $r_i(\theta)$ be new chance divided by old chance for the taken action:

r_i(\theta) = \frac{\pi_\theta(a_i \mid o_i)}{\pi_{\theta_{old}}(a_i \mid o_i)}

The per sample loss keeps the smaller of plain and clipped:

L^{CLIP}_i = \min\left(r_i(\theta) \hat{A}_i, \text{clip}(r_i(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_i\right)

With $\hat{A}_i > 0$ the update raises action chance but stops paying once the ratio hits $1 + \epsilon$ . With $\hat{A}_i < 0$ it lowers chance but stops penalizing past $1 - \epsilon$ .

Worked example

Use $\gamma = 0.95$ , $\lambda = 1.0$ , $\epsilon = 0.20$ . The critic says $V(s) = 4.00$ and $V(s^\prime) = 5.00$ . Team reward is $r = 2.00$ .

Residual:

\delta = 2.00 + 0.95 \times 5.00 - 4.00 = 2.00 + 4.75 - 4.00 = 2.75

One step advantage $\hat{A} = 2.75$ . The action helped.

Old chance is $0.40$ . Candidate new chance $0.48$ gives ratio:

r = 0.48 / 0.40 = 1.20

Surrogate values:

r \hat{A} = 1.20 \times 2.75 = 3.30

Clipped ratio stays $1.20$ , so clipped value is also $3.30$ . Loss is $\min(3.30, 3.30) = 3.30$ .

Now try an aggressive jump to $0.52$ . Ratio is $0.52 / 0.40 = 1.30$ . Plain value is $1.30 \times 2.75 = 3.575$ . Clipped ratio snaps back to $1.20$ , clipped value stays $3.30$ . Loss is $\min(3.575, 3.30) = 3.30$ . The extra push earns nothing, so the gradient for pushing further is zero. That cap is what the diagram draws.

Code

This snippet recomputes the worked numbers with plain Python. It has no dependencies and prints the advantage plus both clipped cases.

1def td_residual(r: float, v: float, v_next: float, gamma: float) -> float:2    return r + gamma * v_next - v3
4
5def clipped_loss(ratio: float, adv: float, eps: float = 0.20) -> float:6    lo = 1.0 - eps7    hi = 1.0 + eps8    clipped = min(max(ratio, lo), hi)9    return min(ratio * adv, clipped * adv)10
11
12adv = td_residual(2.00, 4.00, 5.00, 0.95)13print(round(adv, 2))  # -> 2.7514print(round(0.48 / 0.40, 2))  # -> 1.215print(round(clipped_loss(0.48 / 0.40, adv), 2))  # -> 3.316print(round(0.52 / 0.40, 2))  # -> 1.317print(round(clipped_loss(0.52 / 0.40, adv), 2))  # -> 3.3

The prints match the hand calculation: advantage $2.75$ , moderate ratio $1.20$ with loss $3.30$ , aggressive ratio $1.30$ still capped at $3.30$ .

Watch Out For

Feeding the critic a concat of every view

Stacking all $N$ local views into the critic sounds thorough and trains poorly past a handful of agents. Input width grows with team size, noise repeats, and the critic fits sensor quirks instead of team state. Returns then swing and advantages follow.

Fix it with a compact global state: positions, health, counts, map flags. Keep views for actors only. Normalize value targets with a running mean and scale so early team return spikes do not throw huge critic gradients.

Running too many PPO epochs on stale team data

Single agent PPO often uses 10 to 20 passes per batch. MAPPO with 10 plus passes on the same joint batch drifts because peers moved while the data sat still. Policy ratios leave the trust band and the next rollout starts from a worse joint policy.

Use 5 to 10 epochs, watch the fraction of clipped ratios, and stop early when half the batch clips. Fresh team rollouts beat extra passes on old ones.

The Quick Version

MAPPO is PPO for teams: one shared actor per agent plus one central critic on full state.
The central critic removes teammate noise from advantages by conditioning on $s$ instead of $o_i$ .
Parameter sharing pools $N$ agent streams into one buffer, so data grows with team size.
Clipping at $1 \pm \epsilon$ caps joint jumps. Ratio $1.30$ with advantage $2.75$ still pays only $3.30$ .
Keep critic inputs compact, normalize value targets, and hold PPO to few epochs per batch.