Cooperative vs. Competitive Environments

When several agents share one world, the reward rules decide everything. Shared rewards need teamwork, opposite rewards need unexploitable play, and mixed rewards tempt everyone into mutual ruin.

Shared rewards pull agents toward one team optimum, zero-sum rewards settle at a 0.00 minimax value, and the dilemma traps selfish play at (1,1) instead of (3,3)

Why Does This Exist?

One agent learning alone faces a fixed world. Add a second learner and the world starts moving: the next state depends on the joint action of everyone, so $P(s' \mid s, a_i)$ as seen by agent $i$ shifts every time someone else updates. A move that scored well at step 100 can fail at step 200 because the opponent adapted.

Worse, "best" stops meaning one thing. It depends on how the rewards $R_1, \dots, R_N$ relate, and each relation needs its own machinery. This page maps the three relations and their fixes. The algorithms themselves live in multi-agent RL, the formalism in Markov games, and the classic failure in independent Q-learning.

Think of It Like This

A rowing crew, a chess match, and a highway at rush hour

A rowing eight shares one fate. All eight rowers win or lose together, so the job is pure sync: pull together and spot the rider who coasts while others haul.

A chess game is the opposite. White at $+1$ means Black at $-1$ , no sharing. You assume your opponent finds the most punishing reply to every move, so you play the line that survives that.

Rush hour is the trap. If everyone keeps speed and leaves a gap, all flow. But each driver gains by weaving, and when all weave, all jam. Each person was rational. All are stuck.

Where it stops: people use norms, talk, and contracts to escape the jam. Your agents have only numbers, so the escape has to be built into the loss.

How It Actually Works

A Markov game tuples up the setup: agents $\mathcal{N}$ , state $s$ , joint action $\mathbf{a} = (a_1, \dots, a_N)$ , transitions $P(s' \mid s, \mathbf{a})$ , one reward $R_i$ per agent, discount $\gamma$ . Agent $i$ earns $V_i$ , the expected discounted sum of its own $R_i$ . No single "optimal policy" exists. The replacement is Nash: a joint policy where nobody gains by deviating alone.

From there the reward relation splits the world in three.

Cooperative means $R_1 = \dots = R_N = R_{\text{team}}$ . The goal is Pareto: nobody can gain without someone losing. The headache is credit assignment. One team score arrives and nobody knows who earned it, so naive training breeds lazy riders who coast while one hero works. The fix family is centralized training with factorized critics, such as VDN and QMIX, plus counterfactual baselines that ask what changes if this one agent had acted differently.

Competitive zero-sum means $R_1 = -R_2$ . Von Neumann's minimax value $V_1^* = \max_{\pi_1} \min_{\pi_2} E[R_1]$ is the score player 1 can guarantee against any opponent. Pure strategies cycle: rock loses to paper loses to scissors loses to rock. The fix is stochastic mixed strategies learned against a history of opponents, through self-play, fictitious play, or response oracles.

Mixed general-sum means $R_1 \ne R_2$ with no fixed sum. Selfish best replies land in Nash traps that waste welfare. The fix family shapes motives: inequity penalties or opponent-aware updates that reward reciprocity.

Worked example

Take the dilemma with cooperate $C$ and defect $D$ : $(C,C)$ pays $(3.0, 3.0)$ , $(D,C)$ pays $(5.0, 0.0)$ , $(C,D)$ pays $(0.0, 5.0)$ , $(D,D)$ pays $(1.0, 1.0)$ .

Defect dominates. If the other cooperates, defect pays $5$ over $3$ . If the other defects, defect pays $1$ over $0$ . So both defect and land at $(1.0, 1.0)$ though both prefer $(3.0, 3.0)$ .

Welfare at the Nash point is $1.0 + 1.0 = 2.0$ . Welfare at cooperation is $3.0 + 3.0 = 6.0$ . The price of anarchy is $6.0 / 2.0 = 3.0$ . Selfish play destroys two thirds of the available welfare.

For contrast, matching pennies has value $0.00$ at fifty-fifty play, since $(2 \times 0.5 - 1)(2 \times 0.5 - 1) = 0.00$ . And the coordination game with $(0,0)$ paying $10.0$ and $(1,1)$ paying $6.0$ has its team optimum at $10.0$ . Same two players, three reward tables, three different answers.

Code

Two ratios carry the page. Compute the welfare ratio and the minimax value:

1pareto: float = 3.0 + 3.02nash: float = 1.0 + 1.03print(f"{pareto:.1f} {nash:.1f} {pareto / nash:.1f}")  # -> 6.0 2.0 3.04
5p: float = 0.56q: float = 0.57print(f"{(2 * p - 1) * (2 * q - 1):.2f}")  # -> 0.00

The first line matches the hand arithmetic at $3.0$ . The second confirms the fifty-fifty minimax value is zero, so neither side can be exploited there.

Watch Out For

Training each agent alone and watching the loop spin

Symptom: independent Q-learners in a zero-sum game oscillate forever, and in the dilemma they slide to $(1.0, 1.0)$ and stay. Loss curves look busy while policies chase each other in circles.

Fix: never treat co-learners as fixed noise. Train cooperators with a centralized critic that sees the joint state and actions, train competitors against a pool of past checkpoints, and add opponent-aware or inequity terms in mixed games.

Bringing a team algorithm to a fight

Symptom: QMIX or shared-reward PPO pasted into a zero-sum or mixed game converges fast to something dumb, like always cooperating into an opponent that always defects.

Fix: match the method to the regime before tuning anything. Shared reward with factorization for teams, self-play pools for adversaries, reciprocity shaping for mixed motives. The regime decides the algorithm, not the other way round.

The Quick Version

Shared rewards ( $R_1 = \dots = R_N$ ) call for Pareto teamwork with factorized critics, since one team score hides who did the work.
Zero-sum rewards ( $R_1 = -R_2$ ) call for minimax mixed strategies near value $0.00$ in matching pennies, learned against past opponents to stop cycling.
Mixed rewards trap selfish play at $(1.0, 1.0)$ though $(3.0, 3.0)$ is available, a welfare ratio of $3.0$ .
Independent single-agent learners go non-stationary and loop, so centralize, self-play, or shape motives depending on the regime.