Soft Q-Learning

Soft Q-learning adds an entropy bonus to every backup, so the agent keeps several good actions alive instead of collapsing onto one greedy choice.

Why Does This Exist?

Ordinary Q-learning backs up the single best next value: $y = r + \gamma \max_{a'} Q(s', a')$ . That hard max has two sharp edges. First, it overestimates, because the maximum over noisy estimates is biased upward. Second, it collapses the policy onto one action. If steering left and steering right around an obstacle are both good, greedy Q-learning picks one and forgets the other. Block that corridor later and the agent has no fallback.

Soft Q-learning fixes both edges with one change: maximize reward plus entropy. The agent is paid for high return and for staying unpredictable, with a temperature $\alpha > 0$ setting the exchange rate. This page covers that mechanism and the soft backup it produces. The broader case for entropy objectives lives in maximum entropy RL, and the actor-critic successor that made the idea practical at scale is soft actor-critic.

The one prerequisite is the Bellman backup itself: a Q-value is learned by regressing toward reward plus the discounted value of the next state, as in deep Q-networks. Everything here is that same loop with a softer target.

Think of It Like This

Two soup stalls and a hungry regular

You eat lunch on a street with two soup stalls. Stall A serves a 9 out of 10 bowl, stall B a 7 out of 10. A greedy friend tries A once, then eats at A every day for a year and never learns B improved its recipe.

You play it softer. You visit A about three times out of four and B about one time in four. You still eat very well on average, and you keep tasting both menus, so when A closes for renovations you already know exactly where your second-favorite bowl is.

The visit frequencies are the policy, the soup scores are the Q-values, and your willingness to keep tasting B is the temperature $\alpha$ . Turn it down and you become your greedy friend. Turn it up and you eat everywhere at random.

Where it stops: soup scores stay fixed while you taste around. Q-values move as the agent learns, so the real algorithm is tasting menus that the tasting itself keeps rewriting.

How It Actually Works

Standard RL maximizes expected discounted return. Soft Q-learning adds the policy entropy $\mathcal{H}(\pi(\cdot \mid s))$ at every step, scaled by $\alpha$ :

J(\pi) = \sum_t \mathbb{E}\left[ r(s_t, a_t) + \alpha \mathcal{H}(\pi(\cdot \mid s_t)) \right]

Here $\mathcal{H}$ is Shannon entropy, a number that is large when the policy spreads probability across many actions and zero when it always picks one. The optimal policy for this objective has a closed form: it is the Boltzmann distribution with Q-values as negative energies,

\pi^*(a \mid s) = \exp\left( (Q^*(s, a) - V^{\text{soft}}(s)) / \alpha \right)

where the soft state value is the log-sum-exp over actions:

V^{\text{soft}}(s) = \alpha \log \sum_i \exp(Q(s, a_i) / \alpha)

In plain words: exponentiate each Q-value divided by temperature, add them up, take the log, multiply back by temperature. When $\alpha$ goes to zero this recovers the hard $\max$ . When $\alpha$ is large the policy goes nearly uniform. The network is trained with the usual squared error against the soft target $y^{\text{soft}} = r + \gamma V^{\text{soft}}(s')$ , reusing a frozen target network for the $V^{\text{soft}}(s')$ term.

Worked example

Two actions, temperature $\alpha = 2.0$ , discount $\gamma = 0.9$ , reward $r = 1.5$ , current estimate $Q(s, a) = 5.0$ , next-state values $Q(s', a_1) = 4.0$ and $Q(s', a_2) = 2.0$ .

Scale by temperature: $4.0 / 2.0 = 2.0$ and $2.0 / 2.0 = 1.0$ . Exponentiate: $\exp(2.0) \approx 7.3891$ and $\exp(1.0) \approx 2.7183$ . Add: $Z = 7.3891 + 2.7183 = 10.1074$ . Log and rescale: $V^{\text{soft}}(s') = 2.0 \times \ln(10.1074) \approx 2.0 \times 2.3133 = 4.6265$ .

Notice $4.6265$ sits above the hard max $4.0$ . The surplus is the entropy bonus: the policy below has entropy $\approx 0.5822$ , and $3.4621 + 2.0 \times 0.5822 = 4.6265$ , where $3.4621$ is the expected Q under the policy.

The backup: $y^{\text{soft}} = 1.5 + 0.9 \times 4.6265 = 1.5 + 4.1639 = 5.6639$ . Against the current $5.0$ , the TD error is $0.6639$ and the half-squared loss is $0.2204$ . The implied policy is $\pi(a_1 \mid s') = 7.3891 / 10.1074 \approx 0.7311$ and $\pi(a_2 \mid s') \approx 0.2689$ : mostly the better action, with the runner-up kept firmly alive.

Why continuous actions needed a successor

In continuous action spaces the sum becomes an integral with no closed form, and sampling from $\pi(a \mid s) \propto \exp(Q(s, a)/\alpha)$ means sampling an unnormalized density. The original paper used Stein variational gradient descent particles for this, which was fiddly to tune. Soft actor-critic kept the entropy objective and replaced the sampler with a plain reparameterized Gaussian actor, which is why SAC is what people actually run.

Code

1import math2
3
4def soft_value(q: list[float], alpha: float) -> float:5    """Log-sum-exp soft value, stabilized by subtracting the max."""6    m = max(q)7    return m + alpha * math.log(sum(math.exp((v - m) / alpha) for v in q))8
9
10def soft_policy(q: list[float], alpha: float) -> list[float]:11    """Boltzmann policy over a discrete action set."""12    m = max(q)13    exps = [math.exp((v - m) / alpha) for v in q]14    z = sum(exps)15    return [v / z for v in exps]16
17
18alpha: float = 2.019gamma: float = 0.920q_next = [4.0, 2.0]21v_soft = soft_value(q_next, alpha)22target = 1.5 + gamma * v_soft23td_error = target - 5.024loss = 0.5 * td_error**225print(f"{v_soft:.4f}")  # -> 4.626526print(f"{target:.4f}")  # -> 5.663927print(f"{td_error:.4f}")  # -> 0.663928print(f"{loss:.4f}")  # -> 0.220429print([round(p, 4) for p in soft_policy(q_next, alpha)])  # -> [0.7311, 0.2689]

Watch Out For

Setting the temperature by feel instead of tuning it

The failure looks like this: with $\alpha$ too high the agent dithers and never commits, and returns plateau far below greedy Q-learning. With $\alpha$ too low you paid for the soft machinery and got hard Q-learning back, overestimation included. Both regimes look like ordinary bad training, so people blame the network size or the learning rate first.

The symptom that points at temperature is the policy entropy itself. Log the mean entropy of $\pi(\cdot \mid s)$ over evaluation states. If it sits near $\log K$ for $K$ actions, you are close to uniform and $\alpha$ is too high. If it sits near zero from the first thousand steps, $\alpha$ is too low. Tune $\alpha$ until entropy declines slowly over training, or use the automatic temperature tuning from the SAC follow-up, which treats a target entropy as a constraint.

Letting the soft target leak gradients into the target network

The soft target $y^{\text{soft}}$ is computed from a frozen copy of the Q-network, and that copy must stay frozen during the gradient step. If the implementation accidentally differentiates through $V^{\text{soft}}(s')$ with respect to the online weights, the loss gets a second pathway that pushes values up without bound, and training diverges upward instead of converging.

The symptom is unmistakable once you know it: Q-values climb past any plausible return scale within a few thousand steps while the loss stays small. The fix is to detach or stop-gradient the target branch before computing the loss, and to assert after each optimizer step that the target weights did not move.

The Quick Version

Soft Q-learning maximizes reward plus policy entropy, with temperature $\alpha$ setting the exchange rate between the two.
The hard $\max$ becomes a log-sum-exp soft value, which sits above the max by exactly the entropy bonus.
The optimal policy is Boltzmann in the Q-values, so near-optimal actions keep real probability instead of being discarded.
As $\alpha$ goes to zero the method reduces to ordinary Q-learning, overestimation and all.
Continuous action spaces need approximate samplers, which is why soft actor-critic replaced the original SVGD scheme.
Tune $\alpha$ by watching policy entropy, and always detach the frozen target branch from the gradient.