Soft Actor-Critic (SAC)

An off-policy actor-critic that stays curious on purpose: it maximizes reward plus randomness, so one policy covers many good motions instead of one brittle trick.

SAC samples action 0.7616 with log probability 0.6417, takes the smaller Q of 4.2, and builds soft Bellman target 5.0309

Why Does This Exist?

Deterministic off-policy methods learn fast from old data but collapse onto one narrow motion. The policy goes near-deterministic early, value errors pile up, and the final controller shatters the first time the world wobbles. On-policy methods stay flexible but throw every batch away after one update, so real robots need millions of extra steps.

Soft Actor-Critic takes the third option. It keeps off-policy data reuse and adds a randomness bonus to the objective, so the policy earns reward for succeeding and for staying broad. The framework behind the bonus is maximum entropy RL, the twin-critic trick it borrows is shared with TD3, and the deterministic method it improves on is DDPG.

The one idea you need first

Entropy measures how spread out a policy is. A policy that always picks one action has low entropy; one that spreads probability across many actions has high entropy. SAC adds $\alpha$ times that spread to the reward, where $\alpha$ is a temperature trading exploration against return. The full treatment of the objective lives in maximum entropy RL.

Think of It Like This

The Chef Who Keeps Five Good Sauces Ready

A rigid chef perfects one sauce for one dish. It scores top marks until the night the cream delivery fails, and then service collapses because no backup exists.

A soft chef keeps five good sauces ready and rotates them. Each one scores slightly below the rigid chef's best on a perfect night, but when the cream fails the kitchen keeps serving. The menu bonus pays for variety, not just tonight's plate.

SAC is the soft chef. Its entropy bonus pays the policy to keep several good motions alive, and the temperature turns that pay up early and down once the task is nearly solved. Where the analogy stops: a chef tastes and remembers, while SAC writes variety as a number, $\alpha \mathcal{H}(\pi)$ , inside the Bellman target itself.

How It Actually Works

SAC maximizes reward plus entropy:

$J(\pi) = \sum_t \mathbb{E}[r(s_t, a_t) + \alpha \, \mathcal{H}(\pi(\cdot \mid s_t))]$

Three pieces make it run. Twin critics $Q_1$ and $Q_2$ fight overestimation by backing up the smaller prediction. A squashed Gaussian actor keeps actions inside $[-1, 1]$ through $a = \tanh(u)$ with $u = \mu + \sigma \epsilon$ . And the temperature $\alpha$ auto-tunes against a target entropy of minus the action dimension.

The soft target

For a next action $\tilde{a}'$ sampled from the current policy, with $\gamma$ the discount:

$y = r + \gamma \, (\min(Q_1, Q_2) - \alpha \log \pi(\tilde{a}' \mid s'))$

The $\alpha \log \pi$ term is the price of certainty: confident actions shrink the target, spread-out actions grow it. Both critics regress toward $y$ , and the actor climbs the smaller critic minus its own log probability.

The squashing correction

Because $a = \tanh(u)$ squeezes the whole real line into $[-1, 1]$ , probability piles up near the edges, and the raw Gaussian density misstates it. The fix subtracts the squeeze factor per action dimension:

$\log \pi(a \mid s) = \log p(u \mid s) - \sum_i \log(1 - a_i^2)$

Skip that subtraction and entropy readings near the boundaries are wrong, which you will feel next as a misbehaving temperature.

Worked example

One dimension, so target entropy is $-1.0$ . Policy outputs mean $\mu = 1.0$ and spread $\sigma = 0.5$ , sampled noise $\epsilon = 0.0$ , temperature $\alpha = 0.2$ , reward $r = 1.0$ , discount $\gamma = 0.99$ .

Sample: $u = 1.0 + 0.5 \times 0.0 = 1.0000$ , and $a' = \tanh(1.0) = 0.7616$ .

Densities: latent log probability $\log p(u) = -0.2258$ , squeeze factor $\log(1 - 0.7616^2) = \log(0.4200) = -0.8675$ , so corrected $\log \pi = -0.2258 + 0.8675 = 0.6417$ .

Critics read $Q_1 = 5.0$ and $Q_2 = 4.2$ , so the backup uses $\min = 4.2$ . Soft value: $4.2 - 0.2 \times 0.6417 = 4.0717$ . Target: $1.0 + 0.99 \times 4.0717 = 5.0309$ .

Temperature check: $-\log \pi = -0.6417$ sits above the target $-1.0$ by $0.3583$ , so the policy holds more randomness than asked and $\alpha$ steps down.

Code

The worked numbers in plain Python using only the standard library:

1import math2
3mu, sigma, eps = 1.0, 0.5, 0.04u = mu + sigma * eps5a = math.tanh(u)6log_p = -0.5 * math.log(2 * math.pi * sigma**2)7log_pi = log_p - math.log(1 - a**2)8soft_value = min(5.0, 4.2) - 0.2 * log_pi9target = 1.0 + 0.99 * soft_value10
11print(f"a {a:.4f}")           # -> a 0.761612print(f"log_pi {log_pi:.4f}")  # -> log_pi 0.641713print(f"target {target:.4f}")  # -> target 5.0309

Watch Out For

Dropping the tanh correction corrupts the temperature

Computing $\log \pi$ from the raw Gaussian and skipping the $\log(1 - a^2)$ subtraction reads fine near zero but lies near the action limits, where squashing piles mass. The symptom is a temperature that dives toward zero for no reason while boundary actions look more random than they are.

Fix it by always applying the change-of-variables term with a small stabilizer inside the log, and by unit-testing one boundary action such as $u = 1.0$ against the hand values above.

Fixing the temperature by hand wastes the method

Hard-coding $\alpha$ works in one environment and silently mistunes the next: too high and the policy never commits, too low and it collapses before exploring. The symptom is a sweep over $\alpha$ per task that eats the sample savings SAC promised.

Fix it by enabling automatic tuning against target entropy $-\dim(\mathcal{A})$ and watching $\alpha$ fall as training shifts from exploring to executing.

The Quick Version

SAC maximizes reward plus an entropy bonus, so the policy stays broad while reusing old off-policy data.
Twin critics back up the smaller Q prediction, which curbs the overestimation that breaks deterministic methods.
The actor samples $a = \tanh(\mu + \sigma \epsilon)$ and must correct $\log \pi$ for the squash factor.
In the worked step, action 0.7616 with $\log \pi$ 0.6417 and Q values 5.0 and 4.2 give soft target 5.0309.
The temperature auto-tunes toward target entropy $-\dim(\mathcal{A})$ , falling as the policy commits.