Mean Field Multi-Agent RL

When hundreds of agents share one space, each one stops tracking every neighbor and steers by the average move of the few agents next to it.

Four neighbors average to 0.5, 0.25 and 0.25, the agent picks Up at 4.20 and the target reaches 5.55

Why Does This Exist?

Traffic, order books, and drone swarms put hundreds of agents in one loop. A central critic that reads every action at once blows up: input width grows with $N$ and the joint action count grows as $|\mathcal{A}|^N$ . Value mixing helps but credit still fades across a huge crowd, and independent learners treat each other as noise and wobble.

Mean field MARL, from Yang and coauthors in 2018, borrows a physics trick. Each agent $j$ averages the actions of its local neighbors $\mathcal{N}(j)$ into one mean action $\bar{a}^j$ and best responds to that average. An $N$ body problem becomes $N$ small two body problems. This page shows the average, the factorized update, and one hand computed step. For the general setting see multi-agent RL, for factored values see QMIX and QTRAN, and for the game model see Markov games.

You need one idea first. A one hot vector marks one choice with 1 and the rest with 0, so averaging one hots gives the share of neighbors picking each move.

Think of It Like This

One sardine in a large school

A sardine cannot track nine thousand other fish. It watches five or six neighbors and feels their average heading. When that local average turns left by 30 degrees, it turns left too.

Because every fish runs the same local rule, a turn ripples across the school as tight curves and flashes without any leader calling the move.

Where the analogy stops: sardines share one body and one goal. Engineered agents differ in battery, payload, and destination, so a plain average can wash out a critical neighbor unless you weight by importance.

How It Actually Works

Let $a_k$ be neighbor $k$ action as a one hot vector of width $D$ . Let $s$ be state, $r_j$ reward for agent $j$ , and $\gamma$ in $[0, 1]$ discount. The local mean action is:

$\bar{a}^j = \frac{1}{|\mathcal{N}(j)|} \sum_{k \in \mathcal{N}(j)} a_k$

Each entry of $\bar{a}^j$ is the share of neighbors taking that move. The joint value $Q_j(s, \mathbf{a})$ then drops the full joint action $\mathbf{a}$ and keeps only own action $a_j$ plus the mean:

$Q_j(s, \mathbf{a}) \approx Q_j(s, a_j, \bar{a}^j)$

Input width stays $2 \times D$ no matter how large the swarm grows. Learning uses the mean field target:

$y_j = r_j + \gamma v_j^{\text{MF}}(s')$

where $v_j^{\text{MF}}(s')$ averages $Q_j(s', a_j, \bar{a}'^j)$ under the current policy at the next mean $\bar{a}'^j$ . Actor versions condition the policy the same way, $\pi(a_j \mid s, \bar{a}^j)$ , and move along $\nabla \log \pi \times Q$ .

Worked example

Four neighbors vote Up, Up, Down, Left as $[1,0,0,0]$ , $[1,0,0,0]$ , $[0,1,0,0]$ , $[0,0,1,0]$ . The sum is $[2, 1, 1, 0]$ , so:

$\bar{a}^j = [0.50, 0.25, 0.25, 0.00]$

Half the street moves Up. Suppose the net scores Up $4.20$ , Down $2.10$ , Left $1.80$ , Right $1.20$ . The agent picks Up at $4.20$ . With reward $r_j = 1.50$ , discount $\gamma = 0.90$ , and next mean value $4.50$ :

$y_j = 1.50 + 0.90 \times 4.50 = 1.50 + 4.05 = 5.55$

The TD error is $5.55 - 4.20 = +1.35$ . The diagram shows the same average, the winning Up bar, and the 5.55 target.

Code

The snippet averages the neighbors and forms the same target.

1def mean_action(neighbors: list[list[float]]) -> list[float]:2    """Average one hot neighbor actions into the mean field."""3    n = len(neighbors)4    return [round(sum(col) / n, 4) for col in zip(*neighbors)]5
6
7def bellman_target(reward: float, next_value: float, gamma: float = 0.9) -> float:8    """Mean field target: immediate reward plus discounted next value."""9    return round(reward + gamma * next_value, 4)10
11
12if __name__ == "__main__":13    m = mean_action([[1, 0, 0, 0], [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0]])14    print(m)15    print(bellman_target(1.5, 4.5, gamma=0.9))16    print(round(5.55 - 4.2, 2))

1# -> expected output:2[0.5, 0.25, 0.25, 0.0]35.5541.35

Watch Out For

Averaging away the one neighbor that matters

The symptom: a crowd policy looks smooth in training and fails where one leader decides the outcome, like a tower directing planes or an ambulance in traffic. The plain mean dilutes that signal to one share among many.

The fix: weight neighbors by learned attention over state pairs, or train the few special agents with a full joint critic while the homogeneous crowd stays mean field.

The Quick Version

Mean field MARL replaces the joint action with own action plus local mean: $Q_j(s, a_j, \bar{a}^j)$ .
The mean is the average one hot over neighbors, so input size stays fixed as the swarm grows.
Learning targets $r + \gamma V(s', \bar{a}')$ over local moves only, which dodges the joint max.
In the example the mean is $[0.50, 0.25, 0.25, 0.00]$ , Up wins at $4.20$ , and the target is $5.55$ with error $+1.35$ .
The trick needs dense local crowds of similar agents and breaks on sparse graphs or dominant leaders.