Twin Delayed DDPG (TD3)

TD3 fixes DDPG optimism by asking two critics and believing the gloomier one, blurring target actions with noise, and holding the actor back until the critics settle.

Two target critics score 4.20 and 3.80, the minimum 3.80 builds a target of 4.762, and the actor waits while both critics learn

Why Does This Exist?

DDPG learns continuous control with a deterministic actor chasing one critic's estimates. That single critic is an imperfect neural net, so it has bumps of overestimation scattered across action space. The actor, doing gradient ascent on those estimates, steers straight into the tallest bump. The inflated value then bootstraps into the next target, and values spiral upward while the real behavior gets worse.

Fujimoto and colleagues (2018) traced DDPG failures to three causes: overestimation bias from the actor exploiting critic error, brittle sharp peaks in the value surface, and the actor and critic chasing each other every step. TD3 answers with three fixes: clipped double Q-learning, target policy smoothing, and delayed policy updates.

The one idea you need first

A deterministic policy gradient moves the actor in the direction that increases the critic's value. That is the engine TD3 keeps; everything here is about making the critic it climbs trustworthy. Read DDPG first for the actor-critic loop, then this page for the three stabilizers.

Think of It Like This

Two appraisers, a wobble test, and a patient builder

You are renovating a house to sell, and you need to know what each upgrade is worth.

First, you hire two independent appraisers ( $Q_1$ and $Q_2$ ) and the bank lends on the lower number. One appraiser's enthusiasm cannot inflate the loan, which is clipped double Q-learning: the Bellman target uses $\min(Q_1, Q_2)$ .

Second, an upgrade only counts if nearby variants are worth roughly as much. A kitchen worth a fortune at exactly one shade of white but nothing at any other shade is a pricing glitch, not value. Shaking each estimate with a little noise ( $\epsilon$ ) is target policy smoothing: sharp phantom peaks get averaged away.

Third, the builder (the actor) only gets new blueprints every second appraisal round ( $d = 2$ ). Rebuilding daily on half-checked numbers wastes concrete; waiting one round lets the numbers settle. That is the delayed update.

Where the analogy stops: appraisers value the house from outside, while TD3 critics train on their own past targets, so errors can still echo. The three fixes shrink the echo; they do not remove bootstrapping.

How It Actually Works

TD3 keeps DDPG's actor $\mu_\phi$ and replay loop, and changes how targets are built and when the actor moves.

Twin critics take the minimum

Two critics $Q_{\theta_1}, Q_{\theta_2}$ with target nets each see the same transition. The Bellman target uses the smaller next value:

y = r + \gamma (1 - done) \min_{i=1,2} Q_{\theta_i'}(s', \tilde{a})

Both critics regress toward this one shared $y$ . The minimum favors underestimation, and underestimation does not compound through bootstrapping the way overestimation does.

Noise smooths the target action

Before the target critics score the next action, TD3 perturbs it with clipped Gaussian noise and clips to the legal range:

\tilde{a} = \mathrm{clip}(\mu_{\phi'}(s') + \epsilon, a_{low}, a_{high}), \quad \epsilon \sim \mathrm{clip}(\mathcal{N}(0, 0.2^2), -0.5, 0.5)

Averaging value over the neighborhood flattens phantom spikes. Defaults $\sigma = 0.2$ , $c = 0.5$ come from the original paper.

The actor waits its turn

Critics update every step. The actor and all target nets update only when $t \bmod 2 = 0$ , with the actor climbing $Q_{\theta_1}$ alone and targets drifting by Polyak averaging ( $\tau \approx 0.005$ ). The policy then improves against settled values instead of a moving target. Soft actor-critic later kept the twins but swapped the delay for entropy; the comparison is worth knowing once TD3 clicks.

Worked example

Reward $r = 1.0$ , $\gamma = 0.99$ , non-terminal. Target actor outputs 0.60, noise draw $\epsilon = +0.08$ (inside the $\pm 0.5$ clip), bounds $[-1, 1]$ :

Smoothed action: $\tilde{a} = 0.60 + 0.08 = 0.68$ . No bound clipping needed.

Target critics score it: $Q_1' = 4.20$ , $Q_2' = 3.80$ . Minimum: $3.80$ .

Target: $y = 1.0 + 0.99 \times 3.80 = 1.0 + 3.762 = 4.762$ .

Online critics predict $Q_1 = 4.90$ , $Q_2 = 4.60$ for the sampled transition. Losses: $\mathcal{L}_1 = (4.762 - 4.90)^2 = (-0.138)^2 = 0.0190$ and $\mathcal{L}_2 = (4.762 - 4.60)^2 = 0.162^2 = 0.0262$ .

Update gate: at step 1 ( $1 \bmod 2 \neq 0$ ) only critics move; at step 2 the actor climbs $Q_1$ and targets Polyak-drift. The gloomy critic set the target, noise vetted the action, and the actor sat out round one.

Code

This snippet replays the worked numbers: smooth one action, take the twin minimum, and compute both critic losses.

1def td3_target(reward: float, gamma: float, q1_next: float, q2_next: float) -> float:2    """Clipped double Q target: bootstrap on the gloomier critic."""3    return reward + gamma * min(q1_next, q2_next)4
5
6raw_action = 0.607noise = 0.08  # drawn from clip(N(0, 0.2^2), -0.5, 0.5)8smoothed = min(1.0, max(-1.0, raw_action + noise))9print(round(smoothed, 2))  # -> 0.6810
11y = td3_target(1.0, 0.99, 4.20, 3.80)12print(round(y, 4))  # -> 4.76213print(round((y - 4.90) ** 2, 4))  # -> 0.01914print(round((y - 4.60) ** 2, 4))  # -> 0.0262

The minimum (3.80, not 4.20) caps the target at 4.762, and both losses stay small because neither online critic had drifted far.

Watch Out For

Drowning precision in smoothing noise

The defaults $\sigma = 0.2$ , $c = 0.5$ suit arm-scale MuJoCo tasks. On a precision task like inserting a peg, that much blur erases the very peak the actor must find: values flatten, the actor wobbles around the hole, and training plateaus below the required tolerance. Symptom: returns stall with the actor hovering near, never in, the goal.

Scale noise to the action range instead of copying defaults. Halve $\sigma$ and $c$ together, keep $c \approx 2.5\sigma$ so clipping rarely bites, and confirm the unsmoothed evaluation policy (noise is training-only) actually sharpens.

Setting the delay to 1 and rebuilding DDPG

With $d = 1$ the actor updates every step and TD3 silently degrades into DDPG with extra critics: the moving-target oscillation returns, minus most of the benefit you implemented the algorithm for. Symptom: critic losses seesaw and evaluation returns swing wildly between checkpoints.

Keep $d = 2$ unless you have ablated otherwise. If updates feel slow, raise the critic learning rate or batch size first; the delay is load-bearing, not a speed knob.

The Quick Version

TD3 builds its Bellman target on $\min(Q_1', Q_2')$ so one critic's optimism cannot inflate values.
Clipped Gaussian noise on the target action smooths away sharp phantom peaks in the value surface.
Critics learn every step while the actor and target nets move every second step, breaking the chase cycle.
The worked target 4.762 comes from $1.0 + 0.99 \times 3.80$ , with the gloomier critic setting the number.
Keep $d = 2$ and scale smoothing noise to your action range; both knobs fail quietly when mis-set.