Asynchronous Advantage Actor-Critic (A3C)

Many CPU workers play their own copies of the game at once and push gradients to one shared brain without locks. The mixed updates decorrelate the data, so no replay buffer is needed.

Two workers start at 1.00 2.00, push gradients in turn, and move the shared weights to 0.96 2.06 then 1.01 2.03 without locks

Why Does This Exist?

Deep Q methods broke correlation with a million step replay store. That works for off policy Q learning, but on policy methods cannot reuse old data without correction. Actor critic methods need fresh rollouts from the current policy, so the buffer does not fit.

A3C replaces memory with parallelism. Each of $K$ workers holds its own environment copy and its own local weights $\theta'$ for the policy and $\phi'$ for the value baseline. Workers roll out up to $t_{max} = 5$ steps, compute $n$ -step advantages, and push gradients to the shared model with no locks. Because the workers sit in different states at different times, the push stream is mixed enough to train stably on CPU.

This page covers the async push and the $n$ -step update. The sync follow up lives in advantage actor critic, the shared architecture in actor critic architecture, and continuous control notes in continuous action spaces.

You need policy gradients first: $\log \pi(a \mid s)$ times advantage moves probability toward better actions.

Think of It Like This

Scouts phoning edits to one shared playbook

One playbook sits at HQ. Four scouts watch four different opponents with yesterday's copy. Each scout tests a short drill, phones in a small edit the moment it finishes, then pulls the latest playbook before the next drill.

No scout waits for the others, and HQ never locks the book. Because the calls come from different games, the edit stream stays balanced instead of overfitting to one opponent.

Where it stops: scouts write notes one at a time while A3C threads overwrite shared floats with Hogwild races. Stale edits are also real here: a slow scout phones in advice for a playbook that already moved on.

How It Actually Works

Each worker repeats pull, rollout, bootstrap, push. Local gradients start at zero, sync $\theta' \leftarrow \theta$ and $\phi' \leftarrow \phi$ , then step the private env for up to $t_{max}$ steps.

Bootstrapping sets $R = 0$ at a terminal state else $R = V(s_t; \phi')$ . Walking backward with discount $\gamma$ :

R \leftarrow r + \gamma R

A = R - V(s; \phi')

Here $r$ is the one step reward, $V$ is the critic value, and $A$ is the advantage telling how much better this path ran than expected. Policy and value gradients accumulate with entropy bonus $H$ weighted by $\beta$ :

d\theta += \nabla \log \pi(a \mid s) A + \beta \nabla H

Then the Hogwild push writes straight to shared memory with step $\alpha$ , no mutex:

\theta \leftarrow \theta + \alpha d\theta

The price is staleness: $\theta'$ lags $\theta$ by the time the rollout took.

Worked example

Track shared weights $w$ in $\mathbb{R}^2$ with $\alpha = 0.10$ . Both workers pull:

w = [1.00, 2.00]

Worker 1 finishes fast with $g_1 = [0.40, -0.60]$ and pushes:

w \leftarrow [1.00, 2.00] - 0.10 \times [0.40, -0.60] = [0.96, 2.06]

Worker 2 ran longer on the old copy $[1.00, 2.00]$ and returns stale $g_2 = [-0.50, 0.30]$ . It still pushes:

w \leftarrow [0.96, 2.06] - 0.10 \times [-0.50, 0.30] = [1.01, 2.03]

Both edits land. The second used a one update old view, which is the staleness A3C accepts.

Code

Deterministic replay of the two pushes above.

1import numpy as np2
3
4def push(params: np.ndarray, grad: np.ndarray, lr: float = 0.10) -> np.ndarray:5    """One Hogwild style descent step, no locks in this model."""6    return params - lr * grad7
8
9w = np.array([1.00, 2.00])10g1 = np.array([0.40, -0.60])11w = push(w, g1)12print(np.round(w, 4))  # -> [0.96 2.06]13
14g2 = np.array([-0.50, 0.30])15w = push(w, g2)16print(np.round(w, 4))  # -> [1.01 2.03]

Outputs match the hand steps: $[0.96, 2.06]$ then $[1.01, 2.03]$ .

Watch Out For

Too many workers make gradients too stale

A3C assumes delays stay short. With 32 threads or uneven episode lengths, a slow worker returns advice for a model that moved dozens of updates ago.

Symptom: sudden score collapse after adding cores, worse with longer episodes. Fix: cap workers near core count, use shared RMSprop statistics to damp odd pushes, or switch to advantage actor critic sync batches on GPU.

The Quick Version

A3C decorrelates with parallel env copies, not with a replay store, so it stays on policy.
Workers use $n$ -step bootstrapped advantages with $t_{max} = 5$ plus an entropy bonus.
Hogwild pushes skip locks for CPU speed and accept mild staleness.
The traced run moves shared weights $[1.00, 2.00]$ to $[0.96, 2.06]$ to $[1.01, 2.03]$ .
Cap worker count or go sync A2C when staleness starts to hurt.