Maximum Entropy RL

Score the goal and the spread. The policy keeps every good path alive in proportion to its value instead of collapsing to one narrow line.

Q values 2.0 and 1.0 give Boltzmann chances 0.7311 and 0.2689, entropy 0.5822, and soft value 2.3132

Why Does This Exist?

Plain RL maxes expected reward:

J(\pi) = \mathbb{E}_{\tau \sim \pi}\left[\sum_t \gamma^t R(s_t, a_t)\right]

That push collapses to one deterministic pick. Spread hits zero, backup routes vanish, and a small gust or slip breaks the run. Tasks with two equal doors keep only one.

Max entropy RL from Ziebart plus Haarnoja and Levine adds spread to the goal. Each step pays reward plus $\alpha$ times entropy $\mathcal{H}(\pi(\cdot \mid s))$ . Policies stay smooth and multimodal, with weight on every viable path. This page covers the objective and the soft values. The actor critic built on it lives in soft actor-critic. The tabular value form lives in soft Q-learning. Plain gradients live in policy gradient.

The one idea you need first

Entropy $\mathcal{H}$ measures spread: $\mathcal{H}(\pi) = \mathbb{E}_{a \sim \pi}[-\log \pi(a \mid s)]$ . Uniform is high, single spike is zero. Temperature $\alpha > 0$ prices spread in reward units. Near zero recovers plain RL. Large values wander.

Think of It Like This

One ridge trail versus a mapped valley

A plain explorer finds one narrow ridge over a ravine and declares it the only route. All traffic walks single file. One fallen rock strands the party.

An entropy surveyor maps ridge, river path, and forest track, then splits traffic by safety and speed. A blocked ridge shifts walkers to the river with no panic.

Where it stops: hikers pick one path per trip. A MaxEnt policy is a full spread over actions at once, hedging every step instead of booking one trail.

How It Actually Works

Objective with temperature $\alpha$ :

J_{MaxEnt}(\pi) = \sum_t \mathbb{E}_{(s_t, a_t) \sim \pi}[\gamma^t (R(s_t, a_t) + \alpha \mathcal{H}(\pi(\cdot \mid s_t)))]

Hard max $V(s) = \max_a Q(s, a)$ turns soft:

V_{soft}(s) = \alpha \log \sum_a \exp(Q_{soft}(s, a) / \alpha)

Action values still back up reward plus next soft value:

Q_{soft}(s, a) = R(s, a) + \gamma \mathbb{E}_{s^\prime}[V_{soft}(s^\prime)]

Optimal policy is Boltzmann, in words exponentiated value over its sum:

\pi^\ast(a \mid s) = \exp(Q_{soft}(s, a) / \alpha) / Z(s)

with $Z(s) = \sum_{a^\prime} \exp(Q_{soft}(s, a^\prime) / \alpha)$ . Better actions get exponentially more chance, weaker ones keep nonzero mass. As $\alpha$ drops to zero the soft value returns to hard max.

Worked example

One state, two actions, $\gamma = 0$ , $\alpha = 1.0$ . Rewards give $Q = [2.0, 1.0]$ .

Partition:

Z = \exp(2.0) + \exp(1.0) = 7.389056 + 2.718282 = 10.107338

Chances:

\pi_1 = 7.389056 / 10.107338 \approx 0.7311

\pi_2 = 2.718282 / 10.107338 \approx 0.2689

Sum is $1.0000$ . Entropy in nats:

\mathcal{H} = -[0.7311 \log 0.7311 + 0.2689 \log 0.2689]

\mathcal{H} = -[0.7311 \times (-0.3132) + 0.2689 \times (-1.3134)] \approx 0.5822

Mean reward:

\mathbb{E}[R] = 0.7311 \times 2.0 + 0.2689 \times 1.0 = 1.4622 + 0.2689 = 1.7311

Total objective:

J = 1.7311 + 1.0 \times 0.5822 = 2.3133

Soft value check:

V_{soft} = 1.0 \times \log(10.107338) \approx 2.3132

Matches $J$ to four places. Plain max would pay $2.00$ on one spike. Soft pays $2.3132$ because it counts reward plus spread. Those are the numbers the diagram draws.

Code

This snippet uses only the standard library and prints the same partition, chances, entropy, mean, objective, and soft value.

1import math2
3q1 = 2.04q2 = 1.05alpha = 1.06e1 = math.exp(q1 / alpha)7e2 = math.exp(q2 / alpha)8z = e1 + e29print(round(z, 4))  # -> 10.107310p1 = e1 / z11p2 = e2 / z12print([round(p1, 4), round(p2, 4)])  # -> [0.7311, 0.2689]13h = -(p1 * math.log(p1) + p2 * math.log(p2))14print(round(h, 4))  # -> 0.582215mean = p1 * q1 + p2 * q216print(round(mean, 4))  # -> 1.731117print(round(mean + alpha * h, 4))  # -> 2.313318print(round(alpha * math.log(z), 4))  # -> 2.3132

Soft value $2.3132$ matches objective $2.3133$ up to rounding, which is the identity the page claims.

Watch Out For

Leaving temperature fixed for the whole run

One $\alpha$ cannot serve early roaming and late control. High $\alpha$ drowns reward in noise and walks at random. Low $\alpha$ collapses early to one spike and keeps plain RL flaws. A single static value picks one failure.

Track spread and move $\alpha$ against a target. In soft actor-critic style tuning, raise $\alpha$ when entropy falls below target and lower it when entropy runs hot. Log $\alpha$ , entropy, and mean reward together.

Reading soft value as plain best value

$V_{soft}$ at $2.3132$ tops plain max $2.00$ because it adds spread value. Treating it as achievable reward overstates the policy and breaks comparisons with hard backups.

Label soft and hard separately. Compare policies by sampled mean reward plus measured entropy, not by soft value alone. Convert only when $\alpha$ is near zero.

The Quick Version

Objective is reward plus $\alpha$ times entropy at every step.
Hard max turns into log sum exp with temperature $\alpha$ .
Optimal policy is Boltzmann: chance proportional to $\exp(Q / \alpha)$ .
Worked numbers: $Z \approx 10.1073$ , chances $0.7311$ and $0.2689$ , entropy $0.5822$ .
Soft value $2.3132$ matches mean $1.7311$ plus entropy $0.5822$ .