Actor-Critic Architecture

One network picks actions and a second network grades them step by step. The actor learns from the critic instant feedback instead of waiting for the final score.

A shared backbone splits into an actor head and a critic head, and one TD error of 1.25 updates both

Why Does This Exist?

REINFORCE learns slowly for a mechanical reason: it judges an action taken at step zero by the total return of the whole episode. Every random event hundreds of steps later adds noise to that judgment. In long tasks the noise drowns the signal, so the policy gradient points almost anywhere.

Value methods have the opposite strength and weakness. They learn from every single transition by bootstrapping, which is sample-efficient, but they cannot directly represent a continuous action or a stochastic policy. You cannot ask a Q-table for a torque vector.

The actor-critic architecture wires both halves together. An actor holds the policy $\pi_\theta(a \mid s)$ and a critic holds a value estimate $V_\phi(s)$ . After each transition the critic emits a temporal difference error, and the actor treats that error as the grade for the action it just took. Learning happens every step instead of every episode.

This page is about the wiring: what each half computes, how the error flows, and how workers coordinate. The family of update rules that runs on this wiring has its own page at actor-critic methods, and the concrete canonical instance is Advantage Actor-Critic.

Think of It Like This

A stage actor with a director in the front row

An actor rehearses a play. Under the REINFORCE regime she performs all five acts, waits for the closing-night review, then tries to guess which line readings in act one caused the verdict. Progress is glacial.

With a director in the front row, every line gets an instant note: that pause landed, keep it; that shout fell flat, drop it. The actor adjusts the next line within seconds. Meanwhile the director sharpens her own sense of what each scene is worth as the rehearsal goes on.

Where it stops: a real director has taste independent of the actor. Our critic starts out knowing nothing and grades the actor using its own unfinished guesses. Early in training the blind leads the blind, and both only converge because the rewards slowly ground the critic in reality.

How It Actually Works

Most deep implementations share one backbone, a network $h = f(S_t)$ that turns the state into features, then split into two heads:

Actor head. Outputs the policy, such as action probabilities for discrete moves or a mean and spread for continuous ones.
Critic head. Outputs a single number, the estimated value $V(S_t)$ of the current state.

When the agent acts and the environment answers with reward $R_{t+1}$ and next state $S_{t+1}$ , the critic forms the one-step TD error. Here $\gamma$ is the discount factor that shrinks future value:

\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)

This $\delta_t$ is the whole message bus of the architecture. A positive value means the action beat expectations, so the actor should raise that action probability; a negative value means the opposite. Both heads train off it:

Critic update. Move the value estimate toward the bootstrapped target: $V(S_t) \leftarrow V(S_t) + \alpha \, \delta_t$ , with learning rate $\alpha$ .
Actor update. Climb the log-probability of the taken action, scaled by the grade: $\theta \leftarrow \theta + \alpha \, \delta_t \, \nabla_\theta \log \pi_\theta(A_t \mid S_t)$ .
Entropy bonus. A small reward for keeping the policy spread out, which stops the actor from locking onto its first lucky action.

How workers coordinate

Two classic layouts run this wiring. A3C gives each CPU worker its own copy of the environment and lets workers push gradients to shared weights whenever they finish, with no locks. That asynchrony decorrelates the data but lets slow workers apply stale gradients computed against old weights. A2C waits for all workers, stacks their rollouts into one batch, and updates once on the GPU. Same architecture, no staleness, far better hardware use. Modern code uses the synchronous form.

Worked example

State $S_0$ with critic value $V(S_0) = 2.0$ . The actor gives two actions probabilities 0.70 and 0.30, and samples the first. The environment pays $R_1 = 1.0$ and the critic rates the next state $V(S_1) = 2.5$ . With $\gamma = 0.9$ :

\delta_0 = 1.0 + 0.9 \times 2.5 - 2.0 = 1.0 + 2.25 - 2.0 = 1.25

Positive, so the action beat expectations. The critic target is $1.0 + 2.25 = 3.25$ , and with $\alpha = 0.1$ :

V(S_0) \leftarrow 2.0 + 0.1 \times 1.25 = 2.125

The actor raises the log-probability of the chosen action in proportion to $0.1 \times 1.25 = 0.125$ . One transition, two updates, no episode required. Contrast this with waiting hundreds of steps for a Monte Carlo return: the same reward would arrive mixed with everything else that happened later.

Code

This snippet replays the worked example in plain standard-library Python: one TD error, one critic move, one actor nudge.

1def actor_critic_step(2    v: float, v_next: float, reward: float, gamma: float = 0.9, alpha: float = 0.13) -> tuple:4    """Return (td error, bootstrapped target, updated value) for one transition."""5    target = reward + gamma * v_next6    delta = target - v7    return delta, target, v + alpha * delta8
9
10delta, target, v_new = actor_critic_step(2.0, 2.5, 1.0)11print(f"{delta:.2f}")12# -> 1.2513print(f"{target:.2f}")14# -> 3.2515print(f"{v_new:.4f}")16# -> 2.1250

The prints match the hand computation: error 1.25, target 3.25, value 2.1250.

Watch Out For

Stale gradients when workers run free

In the asynchronous layout, a worker that pauses on a slow reset computes its gradient against weights that are dozens of updates old, then writes that stale gradient over fresh weights. The symptom is a policy that improves for hours and then falls off a cliff for no apparent reason. The fix is the synchronous layout: batch all workers, update once, and the staleness class of bug disappears by construction.

The actor going deterministic too early

Without the entropy bonus, the actor often latches onto the first action that gets a positive grade and stops trying the rest. The symptom is a policy with one action near probability 1.0 while rewards plateau far below what the task allows. The fix is a small entropy coefficient that pays the actor to stay spread out, decayed as training matures. If your action histogram collapses in the first minutes, check this before touching the network.

The Quick Version

The actor holds the policy and the critic holds a value estimate, usually sharing one backbone with two heads.
One TD error $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$ grades each transition and trains both halves.
Positive error raises the taken action probability; negative error lowers it.
The worked step turns $V(S_0) = 2.0$ into 2.125 off a single reward of 1.0.
Synchronous batched workers (A2C) replaced lock-free async ones (A3C) by removing stale gradients.
Keep an entropy bonus or the actor locks onto its first lucky action.