Continuous Action Spaces in RL

Discrete buttons cannot steer a robot arm, so the policy outputs smooth real-valued controls instead. It predicts a Gaussian per joint, samples with a trick that keeps gradients alive, and squashes into motor limits.

A Gaussian policy head predicts mean and spread, samples with the reparameterization trick, and squashes through tanh into bounded motor commands

Why Does This Exist?

DQN-style methods pick actions by scoring every option and taking the max:

a^* = \arg\max_a Q(s, a)

That breaks the moment actions turn continuous. A 7-joint arm with 10 bins per joint needs $10^7$ scores per step, and the winning command still jumps in coarse steps that chatter real gears. Robots, cars, and valves need smooth vectors like torques and steering angles, not button presses.

Continuous methods output the vector directly. A stochastic policy predicts a distribution $\pi_\theta(a \mid s)$ and samples it; a deterministic one predicts the vector itself. One forward pass, no search over actions.

This page covers the stochastic Gaussian route with bounded outputs. For the deterministic twin see deterministic policy gradients and DDPG. For the gradient theorem underneath see policy gradient.

You need one setup idea: a policy maps states to actions, and learning adjusts its weights to raise the expected return.

Think of It Like This

A ship wheel versus three gamepad buttons

Steer a yacht through a narrow channel with three buttons: hard left, center, hard right. To hold $4.2$ degrees you slam left, then right, then left, lurching down the channel and straining the rudder cables.

A real wheel turns to any angle and stays there. Small correction, smooth water, quiet cables.

Map it point for point: buttons are a discretized action grid, the wheel is a continuous policy output, and the lurching is the bang-bang chatter that discretization causes on real hardware.

Where it stops: a wheel is one axle, while a humanoid outputs 18 torques at once with coordination across joints, which is why the policy predicts a whole vector with per-joint spreads.

How It Actually Works

The standard stochastic choice is a diagonal Gaussian. The network predicts a mean head $\mu_\theta(s)$ and a log-spread head $\log \sigma_\theta(s)$ , with spread clamped (often to $[-20, 2]$ ) so exploration can neither explode nor die. Predicting log-spread keeps $\sigma = \exp(\log \sigma)$ positive for free.

Two gradient paths

Score-function methods (REINFORCE, PPO, TRPO) use the likelihood ratio:

\nabla_\theta J = E[\nabla_\theta \log \pi_\theta(a \mid s) \, \hat{A}(s, a)]

For one Gaussian dimension the mean score is $(a - \mu)/\sigma^2$ and the log-spread score is $(a - \mu)^2/\sigma^2 - 1$ . Good actions pull the mean toward them; samples beyond one spread widen exploration, nearer ones tighten it.

Pathwise methods (SAC, DDPG-style critics) use the reparameterization trick instead. Sample noise $\varepsilon \sim \mathcal{N}(0, 1)$ once, then write the action as arithmetic:

u = \mu_\theta(s) + \sigma_\theta(s) \varepsilon

Gradients now flow through $u$ into $\theta$ by the chain rule through the critic $Q(s, a)$ , with far less variance than the score route.

Squash into limits, then fix the density

Motors have hard bounds but Gaussians run to infinity. Clipping kills gradients outside the range, so modern actors squash smoothly:

a = a_{\max} \tanh(u)

Squashing compresses probability near the edges, so the logged density must shrink accordingly. The change-of-variables correction for one dimension is:

\log \pi(a \mid s) = \log p(u \mid s) - \log(a_{\max}(1 - \tanh^2(u)))

Skip that subtraction and entropy bonuses in SAC read the wrong density, which pushes pre-squash values to extremes until gradients saturate.

Worked example in 1D

Mean $\mu = 2.0$ , spread $\sigma = 0.5$ , bound $a_{\max} = 1.0$ , latent sample $u = 2.5$ .

Base log density:

\log p(u) = -\tfrac{1}{2}\log(2\pi) - \log 0.5 - \frac{(2.5-2.0)^2}{2(0.25)} = -0.9189 + 0.6931 - 0.5000 = -0.7258

So $p(u) \approx 0.4839$ . Scores: mean score $(0.5)/(0.25) = 2.0$ ; log-spread score $0.25/0.25 - 1 = 0.0$ , because the sample sits exactly one spread away.

Squash: $a = \tanh(2.5) \approx 0.9866$ . Slope $da/du = 1 - 0.9866^2 \approx 0.0266$ , so $\log |J| \approx -3.6271$ . Corrected log density:

\log \pi(a \mid s) = -0.7258 - (-3.6271) = 2.9013

That is density $\approx 18.20$ . It looks startling next to $0.48$ , but tanh packs a wide latent range into a sliver near $1.0$ , so density there must be tall. The diagram traces this exact sample.

Code

The snippet below replays the worked sample with only the standard library.

1import math2
3
4def gaussian_logp(u: float, mu: float, sigma: float) -> float:5    """Log density of a 1D Gaussian at u."""6    return -0.5 * math.log(2.0 * math.pi) - math.log(sigma) - ((u - mu) ** 2) / (2.0 * sigma ** 2)7
8
9mu, sigma, u, a_max = 2.0, 0.5, 2.5, 1.010logp = gaussian_logp(u, mu, sigma)11print(round(logp, 4))12# -> -0.725813print(round((u - mu) / sigma ** 2, 4))14# -> 2.015print(round(((u - mu) ** 2) / sigma ** 2 - 1.0, 4))16# -> 0.017
18a = math.tanh(u)19slope = 1.0 - a ** 220print(round(a, 4))21# -> 0.986622print(round(math.log(a_max * slope), 4))23# -> -3.627124print(round(logp - math.log(a_max * slope), 4))25# -> 2.9013

Watch Out For

Forgetting the tanh Jacobian in entropy code

SAC tempers exploration with an entropy bonus computed from $\log \pi(a \mid s)$ . Plug in the raw Gaussian $\log p(u \mid s)$ and the bonus ignores the squash: near the bounds the policy looks more uncertain than it is, training shoves $u$ outward, tanh saturates, and gradients vanish.

Symptom: pre-squash magnitudes drift to large values while actions pin at the limits. Fix: always subtract $\sum \log(1 - \tanh^2(u))$ (plus the $a_{\max}$ scale) wherever a squashed log-probability is needed.

Letting the spread collapse before exploring

An unconstrained log-spread head can dive early on noisy gradients. Spread hits zero, the policy turns deterministic before seeing the space, and the arm masters one bad motion.

Symptom: actions repeat almost exactly across episodes while returns plateau low. Fix: clamp log-spread (such as to $[-20, 2]$ ), start near zero spread rather than very negative, and watch entropy per joint, not just the mean return.

The Quick Version

Continuous policies emit real-valued control vectors in one forward pass, dodging the $K^m$ blowup of discretized grids.
The network predicts per-joint Gaussian means and clamped log-spreads, keeping variance positive and bounded.
The reparameterization $u = \mu + \sigma \varepsilon$ routes critic gradients straight through the sample with low variance.
Tanh squashing enforces motor bounds, and the log density must subtract the Jacobian term to stay exact.
In the worked sample, $u = 2.5$ with mean 2.0 and sigma 0.5 squashes to $0.9866$ with corrected log density $2.9013$ .