Deterministic Policy Gradients

Deterministic policy gradients steer one exact action per state by chaining the critic value slope into the actor weights, with no sampling over actions.

The critic slope over actions chains into the actor Jacobian, moving actor weights toward higher value without sampling actions

Why Does This Exist?

Stochastic policy gradient integrates over states and actions. For a policy $\pi_{\theta}(a \mid s)$ the gradient averages over every action the distribution might sample. In a high dimensional continuous space that average needs many samples and carries high variance.

Off policy reuse is also painful there. Weighting old data by density ratios over continuous vectors swings wildly and destabilizes updates.

Deterministic Policy Gradient, proved by Silver and colleagues in 2014, removes the action integral. The policy is a map $\mu_{\theta}(s)$ to one action vector. The gradient averages only over states and chains the critic slope $\nabla_a Q$ into actor weights. No action sampling, no density ratio. DDPG and TD3 build directly on it.

This page covers the theorem, the chain rule and the exploration fix. For the stochastic baseline see policy gradient. For the flagship application see deep deterministic policy gradient. For the spaces it unlocks see continuous action spaces.

The one idea you need first

A Jacobian $\nabla_{\theta} \mu(s)$ lists how each actor weight changes the action output. The critic slope $\nabla_a Q(s, a)$ lists how each action entry changes value. Multiplying them converts a wish in action space into a weight change.

Think of It Like This

A hand on a power steering wheel

You hold a wheel linked straight to the tires. Mid turn you feel resistance in your palms telling you which way grip improves. You do not try fifty random angles. You follow the pressure and your arms learn the better line.

Wheel resistance is $\nabla_a Q$ . Arm adjustment is $\nabla_{\theta} \mu$ . The chain of the two is the update. The road you practiced on is the replay buffer, and you still swerve a little on purpose during practice so you learn more than one line.

Where it stops: real resistance comes from tire physics. Here the slope comes from a learned critic that starts wrong, so early steering advice can point the wrong way.

How It Actually Works

Let states follow $s \sim \rho^{\mu}$ under the deterministic policy. Performance is expected value at the actor choice:

J(\mu_{\theta}) = \mathbb{E}_{s \sim \rho^{\mu}}[Q^{\mu}(s, \mu_{\theta}(s))]

The Deterministic Policy Gradient theorem states:

\nabla_{\theta} J(\mu_{\theta}) = \mathbb{E}_{s \sim \rho^{\mu}}[\nabla_{\theta} \mu_{\theta}(s) \nabla_a Q^{\mu}(s, a) \mid_{a=\mu_{\theta}(s)}]

In plain words: slope of weights into actions times slope of actions into value, averaged over visited states. Off policy training swaps the state distribution to the behavior data $\rho^{\beta}$ and keeps the same chain, with no importance ratio over continuous actions.

Why the action integral disappears

A stochastic policy spreads probability across actions, so its gradient must average over that spread. A deterministic policy puts all mass on one point $a = \mu_{\theta}(s)$ , so composition $Q(s, \mu_{\theta}(s))$ differentiates by the ordinary chain rule. Fewer integrals means fewer samples for the same signal, which is the sample efficiency win.

Exploration still needs noise

One output per state never tries anything new on its own. Collection adds noise $N_t$ to the actor output with behavior $\beta(a \mid s) = \mu_{\theta}(s) + N_t$ , often Gaussian or temporally correlated noise. Replay stores those noisy transitions. The critic trains on them while the actor updates through $\nabla_a Q(s, \mu_{\theta}(s))$ at sampled states.

Worked example

One dimensional control with $s = 2.0$ . Actor $\mu_{\theta}(s) = \theta_1 s + \theta_2$ with $\theta_1 = 1.0$ and $\theta_2 = 0.5$ . Critic $Q(s, a) = -(a - 5.0)^2$ with best action $5.0$ . Step size $\alpha = 0.1$ .

Forward action and value:

a = 1.0 \times 2.0 + 0.5 = 2.50

Q = -(2.50 - 5.0)^2 = -6.2500

Critic slope with respect to action:

\nabla_a Q = -2(a - 5.0) = -2 \times (-2.50) = 5.00

Actor Jacobian entries:

\nabla_{\theta_1} \mu = s = 2.00, \quad \nabla_{\theta_2} \mu = 1.00

Chain into gradients:

\nabla_{\theta_1} J = 2.00 \times 5.00 = 10.0000

\nabla_{\theta_2} J = 1.00 \times 5.00 = 5.0000

Ascent step:

\theta_1 \leftarrow 1.00 + 0.1 \times 10.00 = 2.00

\theta_2 \leftarrow 0.50 + 0.1 \times 5.00 = 1.00

New action and value:

a_{new} = 2.00 \times 2.0 + 1.00 = 5.00

Q_{new} = -(5.00 - 5.00)^2 = 0.0000

One step reaches the optimum because the critic is a clean quadratic and the actor is linear in this toy setup.

Code

This snippet repeats the chain rule and update with plain Python.

1def actor(s: float, t1: float, t2: float) -> float:2    """Deterministic linear actor: mu(s) = t1 * s + t2."""3    return t1 * s + t24
5
6def critic(a: float, star: float = 5.0) -> float:7    """Quadratic critic with best action at star."""8    return -((a - star) ** 2) + 0.09
10
11def critic_slope(a: float, star: float = 5.0) -> float:12    """Slope of critic over action: -2 * (a - star)."""13    return -2.0 * (a - star)14
15
16s = 2.017t1, t2 = 1.0, 0.518a = actor(s, t1, t2)19print(f"{a:.2f}")  # -> 2.5020print(f"{critic(a):.4f}")  # -> -6.250021ga = critic_slope(a)22print(f"{ga:.2f}")  # -> 5.0023print(f"{s * ga:.4f}")  # -> 10.000024print(f"{1.0 * ga:.4f}")  # -> 5.000025t1 += 0.1 * s * ga26t2 += 0.1 * 1.0 * ga27print(f"{t1:.2f} {t2:.2f}")  # -> 2.00 1.0028a_new = actor(s, t1, t2)29print(f"{a_new:.2f} {critic(a_new):.4f}")  # -> 5.00 0.0000

Slopes, updated weights and the final optimal action match the hand derivation line for line.

Watch Out For

Training the deterministic actor on its own narrow track

Without noise the actor visits a thin slice of state action space. The critic looks accurate there and wrong elsewhere, yet the chain rule trusts it everywhere. Learning stalls at the first decent line and never discovers sharper ones.

Symptom: fast early gains then a flat plateau with near zero action diversity. Fix it by collecting with added Gaussian noise, keeping that noisy data in replay and training the critic on it. Check action histograms during collection. If they collapse to a spike, raise the noise scale before touching learning rates.

The Quick Version

DPG integrates over states only: $\nabla_{\theta} J = \mathbb{E}[\nabla_{\theta} \mu \nabla_a Q]$ .
The critic slope $\nabla_a Q$ says which action direction helps, and the actor Jacobian turns it into a weight change.
Off policy replay needs no action density ratio because the target policy is a single point.
Worked numbers: $a = 2.50$ , $Q = -6.25$ , slopes $10.0$ and $5.0$ , new weights $2.0$ and $1.0$ , optimum $5.0$ .
Deterministic output cannot explore alone, so add noise during collection and store it.
DDPG applies this theorem with replay plus soft targets, and TD3 hardens it further.