Deep Deterministic Policy Gradient (DDPG)

DDPG outputs one exact continuous action per state instead of searching over action values, and keeps learning stable with slowly moving target networks.

A DDPG actor proposes one continuous action, a critic scores it, and slowly moving target networks supply the Bellman target

Why Does This Exist?

Q-learning picks actions with an argmax over values. That works for a few buttons and fails for a robot arm, where the action is a vector of torques. Searching a continuous space for the max at every step means solving a fresh optimization problem thousands of times per episode.

Discretizing does not save you. Ten bins per joint over eight joints gives one hundred million combinations, and the coarse grid still misses the smooth motion the hardware needs.

DDPG, introduced by Lillicrap and colleagues in 2015, combines three fixes. A deterministic actor writes the action directly. A replay buffer reuses past transitions off policy. Soft target updates move the learning target slowly instead of copying weights in jumps. You get DQN style learning for continuous control.

This page covers the actor critic update, the Bellman target and the soft update rule. For the gradient theorem behind the actor see deterministic policy gradients. For the discrete method it extends see deep Q-networks. For the action spaces it serves see continuous action spaces.

The one idea you need first

An actor critic has two networks. The actor $\mu(s)$ maps a state $s$ to an action $a$ . The critic $Q(s, a)$ maps a state plus an action to expected return. The actor proposes, the critic judges, and the judgment trains the proposal.

Think of It Like This

A driving student and a patient instructor

A student driver holds the wheel and picks one exact steering angle. An instructor in the passenger seat watches and scores each choice. After the turn the instructor says which way would have been smoother, and the student adjusts their grip for next time.

The student is the actor. The instructor is the critic. Old lesson notes are the replay buffer. A slowly updated answer key is the target network, changed a little each lesson so the student never chases a shifting standard.

Where it stops: a human instructor understands physics. The critic starts knowing nothing and learns only from reward numbers, so early advice can be wrong and the actor can follow it with full confidence.

How It Actually Works

DDPG keeps four networks. Online actor $\mu(s \mid \theta^{\mu})$ outputs action vector $a$ in $\mathbb{R}^d$ . Online critic $Q(s, a \mid \theta^{Q})$ outputs a scalar value. Target actor $\mu^{\prime}$ and target critic $Q^{\prime}$ are slow copies used only for targets. Here $r$ is reward, $s^{\prime}$ is the next state, $d$ is $1$ when the episode ended and $0$ otherwise, $\gamma$ is the discount factor and $\tau$ is the small soft update rate.

Critic update from the Bellman target

Sample a batch of $(s, a, r, s^{\prime}, d)$ from replay. Build the target with the slow networks:

y = r + \gamma (1 - d) Q^{\prime}(s^{\prime}, \mu^{\prime}(s^{\prime}))

In plain words: reward now plus discounted critic opinion about the next state and the target actor choice there, zeroed out at termination. Train the online critic to match it with squared error:

\mathcal{L}(\theta^{Q}) = \frac{1}{N} \sum_{i=1}^{N} (y_i - Q(s_i, a_i \mid \theta^{Q}))^2

Actor update through the critic

The actor wants actions the critic rates highly. By the chain rule the gradient is the critic slope with respect to action times the actor slope with respect to weights:

\nabla_{\theta^{\mu}} J \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_a Q(s_i, a) \mid_{a=\mu(s_i)} \nabla_{\theta^{\mu}} \mu(s_i)

Here $\nabla_a Q$ points toward higher value in action space and $\nabla_{\theta^{\mu}} \mu$ converts that direction into a weight change. Exploration adds noise to the deterministic output during data collection, often Gaussian noise or temporally correlated noise for hardware with inertia.

Soft target tracking

Instead of freezing targets for thousands of steps, DDPG nudges them after every update with Polyak averaging and a small $\tau$ such as $0.005$ :

\theta^{Q^{\prime}} \leftarrow \tau \theta^{Q} + (1 - \tau) \theta^{Q^{\prime}}

\theta^{\mu^{\prime}} \leftarrow \tau \theta^{\mu} + (1 - \tau) \theta^{\mu^{\prime}}

Small $\tau$ keeps the regression target almost stationary, which calms the feedback loop between actor and critic.

Worked example

Use one transition with $\gamma = 0.99$ , $\tau = 0.1$ and $d = 0$ . Reward $r = 1.0$ . Target actor output $a^{\prime} = 0.70$ . Target critic value $Q^{\prime}(s^{\prime}, a^{\prime}) = 3.0000$ . Online critic prediction $Q(s, a) = 3.5000$ .

Bellman target:

y = 1.0 + 0.99 \times 3.0000 = 1.0 + 2.9700 = 3.9700

Temporal difference error and loss:

\delta = 3.9700 - 3.5000 = 0.4700

\mathcal{L} = 0.4700^2 = 0.2209

Actor chain rule with $\nabla_a Q = 2.0000$ and $\nabla_{\theta^{\mu}} \mu = 0.5000$ :

\nabla_{\theta^{\mu}} J = 2.0000 \times 0.5000 = 1.0000

Polyak step with online weight $w = 10.0000$ and target weight $w^{\prime} = 8.0000$ :

w^{\prime}_{new} = 0.1 \times 10.0000 + 0.9 \times 8.0000 = 1.0000 + 7.2000 = 8.2000

The target glides from $8.0$ to $8.2$ instead of jumping.

Code

This snippet repeats the hand computation above.

1def bellman_target(reward: float, gamma: float, done: float, q_next: float) -> float:2    """One DDPG critic target: y = r + gamma * (1 - d) * Qnext."""3    return reward + gamma * (1.0 - done) * q_next4
5
6def critic_loss(target: float, pred: float) -> tuple[float, float]:7    """Return temporal difference error and squared loss."""8    td = target - pred9    return td, td**210
11
12def actor_grad(grad_q_a: float, grad_mu_theta: float) -> float:13    """Deterministic policy gradient chain rule product."""14    return grad_q_a * grad_mu_theta15
16
17def polyak(online: float, target: float, tau: float) -> float:18    """Soft update: target <- tau * online + (1 - tau) * target."""19    return tau * online + (1.0 - tau) * target20
21
22y = bellman_target(1.0, 0.99, 0.0, 3.0)23print(f"{y:.4f}")  # -> 3.970024td, loss = critic_loss(y, 3.5)25print(f"{td:.4f}")  # -> 0.470026print(f"{loss:.4f}")  # -> 0.220927print(f"{actor_grad(2.0, 0.5):.4f}")  # -> 1.000028print(f"{polyak(10.0, 8.0, 0.1):.4f}")  # -> 8.2000

Each printed value matches the worked section, so you can trust the arithmetic before wiring it into a full trainer.

Watch Out For

Chasing an overestimated critic peak

The actor is trained to maximize the critic output, so it steers toward whatever action the critic currently rates highest. Early in training those peaks are often approximation errors in rarely visited regions. The next Bellman target then bootstraps from the inflated value and the error compounds.

Symptom: Q values climb while true returns stall or fall, and actions slam into their bounds. Fix it by moving to twin critics with a minimum, delaying actor updates relative to critic updates and smoothing the target action with clipped noise, which is exactly the TD3 recipe.

The Quick Version

DDPG replaces the discrete argmax with a deterministic actor that writes the continuous action directly.
The critic learns from $y = r + \gamma Q^{\prime}(s^{\prime}, \mu^{\prime}(s^{\prime}))$ sampled from replay.
The actor climbs by chaining the critic action slope into its own weights.
Target networks track slowly with $\tau$ near $0.005$ instead of copying in jumps.
Deterministic output needs added noise during collection, or the agent never tries anything new.
Vanilla DDPG overestimates values, which motivated twin critics and delayed updates in TD3.