Advantage Actor-Critic (A2C)

A2C judges each action against what the critic expected, not against zero. Actions that beat the baseline get reinforced and actions that miss it get dialed back.

A critic values the states at 1.00 and 1.60 so the TD advantage of plus 2.44 pushes the actor toward the chosen action

Why Does This Exist?

Plain policy gradient methods such as REINFORCE learn from full-episode returns. The gradient points the right way on average, but the noise is brutal. The same state-action pair can score wildly different returns depending on lucky or unlucky steps hundreds of moves later. When every return is positive, the update pushes up the probability of everything it saw and separates good moves from mediocre ones only by degree.

The fix keeps the gradient unbiased while shrinking its noise. Subtract any baseline that depends on the state but not the action, and the expected update does not move, because the score function averages to zero over actions. The best such baseline is the state value $V(s)$ itself. That difference has a name: the advantage $A(s, a) = Q(s, a) - V(s)$ . Positive means better than expected. Negative means worse.

This page covers the synchronous version of that idea. You get the advantage estimator the critic actually computes, the two losses trained together, the lockstep batching that replaced A3C threads, and one transition worked by hand. For the architecture without the advantage math, read actor-critic architecture first. For the async predecessor, the sequel is asynchronous advantage actor-critic.

Think of It Like This

A darts coach who knows your average

You throw darts while a coach tracks your average score per visit.

Without the coach you only hear your raw total at the end of the night. A 60 sounds good until you learn everyone hits 60 when the board is kind. You cannot tell which throws were skill and which were luck.

With the coach every throw gets judged against your average. Hitting 85 when you average 60 earns praise and you repeat that stance. Hitting 40 earns a correction even though 40 still scores points. The coach never throws. They supply the baseline $V(s)$ and the verdict $\delta$ , and you adjust.

Where it stops: a coach sees your form and the room. The critic sees only numbers and learns its baseline from the same noisy rewards it judges, so early in training its averages are wrong and its verdicts mislead.

How It Actually Works

Symbols for this page: $s$ is the state, $a$ the action, $\pi_\theta(a \mid s)$ the actor policy with weights $\theta$ , $V_\phi(s)$ the critic value with weights $\phi$ , $R_{t+1}$ the reward, $\gamma$ the discount, and $\delta_t$ the one-step TD error that stands in for the advantage.

The policy gradient with a state baseline reads:

\nabla_\theta J(\theta) = \mathbb{E}[\nabla_\theta \log \pi_\theta(A_t \mid S_t)\, A(S_t, A_t)]

Nobody knows $Q$ exactly, so A2C estimates the advantage from the critic you already train:

\delta_t = R_{t+1} + \gamma V_\phi(S_{t+1}) - V_\phi(S_t)

Conditioned on the pair $(S_t, A_t)$ , the expectation of $\delta_t$ equals $Q(S_t, A_t) - V_\phi(S_t)$ , which is the advantage. One critic, no second $Q$ network.

The two losses trained together

The actor climbs the advantage-weighted log probability and keeps exploring through an entropy bonus $H(\pi)$ with strength $\beta$ :

\mathcal{L}_{actor} = -\mathbb{E}[\log \pi_\theta(A_t \mid S_t)\, \delta_t + \beta H(\pi_\theta)]

Entropy is highest for a uniform policy and zero for a deterministic one, so the bonus resists early collapse onto one action. The critic regresses toward the one-step Bellman target:

\mathcal{L}_{critic} = \tfrac{1}{2}\,\mathbb{E}[\delta_t^2]

Shared backbones add the two with a critic weight near $0.5$ . Batches come from $N$ environments stepped in lockstep: one batched forward pass on the GPU, $N$ simultaneous actions, one $(N \times T)$ tensor for the backward pass. No Hogwild threads, no stale writes, far better device use than A3C.

Worked example with advantage plus 2.44

State features are $x(S_t) = [1.0, 0.5]$ and $x(S_{t+1}) = [0.8, 1.2]$ . Critic weights are $w_c = [0.5, 1.0]$ , reward $R_{t+1} = 2.0$ , discount $\gamma = 0.9$ . Actor logits are $z = [0.6, 0.2]$ and the agent picks action $a_0$ . Entropy weight $\beta = 0.01$ .

Critic values come from dot products:

V(S_t) = 0.5 \times 1.0 + 1.0 \times 0.5 = 1.00

V(S_{t+1}) = 0.5 \times 0.8 + 1.0 \times 1.2 = 0.40 + 1.20 = 1.60

The TD advantage follows:

\delta_t = 2.0 + 0.9 \times 1.60 - 1.00 = 2.0 + 1.44 - 1.00 = 2.44

Positive and large: this transition beat expectations by a mile. Softmax over $[0.6, 0.2]$ gives exponentials $[1.8221, 1.2214]$ summing to $3.0435$ , so probabilities are $[0.5987, 0.4013]$ . The log probability of the chosen action is $\log(0.5987) \approx -0.5130$ . Entropy is:

H = -[0.5987 \times (-0.5130) + 0.4013 \times (-0.9130)] = 0.6735

Losses:

\mathcal{L}_{actor} = -(-0.5130) \times 2.44 - 0.01 \times 0.6735 = 1.2517 - 0.0067 = 1.2450

\mathcal{L}_{critic} = 0.5 \times 2.44^2 = 2.9768

Because $\delta_t$ is positive, the gradient step raises the probability of $a_0$ . That is the mechanism in one line: the critic sets the bar, the gap decides the push.

Code

A short check of the worked numbers above, with type hints throughout:

1import math2from typing import List3
4
5def dot(a: List[float], b: List[float]) -> float:6    return sum(x * y for x, y in zip(a, b))7
8
9def softmax(logits: List[float]) -> List[float]:10    exps = [math.exp(v) for v in logits]11    total = sum(exps)12    return [e / total for e in exps]13
14
15w_c = [0.5, 1.0]16s = [1.0, 0.5]17s_next = [0.8, 1.2]18reward = 2.019gamma = 0.920beta = 0.0121
22v = dot(w_c, s)23v_next = dot(w_c, s_next)24delta = reward + gamma * v_next - v25print([round(v, 4), round(v_next, 4), round(delta, 4)])26# -> [1.0, 1.6, 2.44]27
28probs = softmax([0.6, 0.2])29print([round(p, 4) for p in probs])30# -> [0.5987, 0.4013]31
32logp = math.log(probs[0])33entropy = -(probs[0] * math.log(probs[0]) + probs[1] * math.log(probs[1]))34print([round(logp, 4), round(entropy, 4)])35# -> [-0.513, 0.6735]36
37actor = -logp * delta - beta * entropy38critic = 0.5 * delta * delta39print([round(actor, 4), round(critic, 4)])40# -> [1.245, 2.9768]

1# -> expected output:2[1.0, 1.6, 2.44]3[0.5987, 0.4013]4[-0.513, 0.6735]5[1.245, 2.9768]

Watch Out For

Letting the actor sprint ahead of the critic

The advantage is only as honest as the baseline. If the actor learns faster than the critic, $\delta_t$ reflects critic error rather than action quality, and the policy chases states where the critic merely underpriced the value. The symptom is early collapse: entropy falls, one action dominates, and returns plateau below what the same budget should reach.

Give the critic the faster schedule, often two to five times the actor step size, and keep $\beta$ nonzero until values settle. When sharing a backbone, weight the critic loss so its features stay accurate. Distrust big advantages while the critic loss is still large.

The Quick Version

A2C scales each policy update by the advantage $A(s, a) = Q(s, a) - V(s)$ , so better-than-expected actions rise and worse ones fall.
The one-step TD error $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$ estimates the advantage with no separate $Q$ network.
Subtracting the baseline keeps the gradient unbiased because the score function averages to zero over actions.
One worked step gives values $1.00$ and $1.60$ , advantage $2.44$ , actor loss $1.2450$ and critic loss $2.9768$ .
$N$ environments step in lockstep for one batched GPU update, which replaced A3C threads and their stale writes.
Entropy bonus $\beta H(\pi)$ keeps the policy exploring until the critic becomes trustworthy.