Maximum Entropy Inverse Reinforcement Learning

Watch an expert, count what they visit, then pick the reward that makes those counts likely while assuming nothing else. Higher return means higher chance.

Expert counts 1.80 and 1.20 meet learner counts 1.6225 and 1.3775, so weights shift by plus 0.1775 and minus 0.1775

Why Does This Exist?

Writing rewards by hand is brittle. A small misspecification invites reward hacking, where the agent maxes the score while driving badly. Inverse RL flips the task: infer the reward from expert demos.

Many rewards fit the same demos, including the zero reward that makes everything optimal. That is reward ambiguity. Margin methods also break ties arbitrarily and assume a perfect expert. Maximum entropy IRL from Ziebart and coauthors in 2008 answers both. It matches expert feature counts while keeping trajectory spread as wide as possible. Noisy demos stay likely, equal return paths stay equal, and linear rewards give one global optimum.

This page covers the entropy fix. The base inverse problem lives in inverse RL. Copying actions without rewards lives in imitation learning. The adversarial shortcut lives in GAIL.

The one idea you need first

A feature map $\phi(s)$ turns a state into counts like road type or speed. A path total $\mu_\tau = \sum_t \phi(s_t)$ sums those counts along the run. Reward weights $\theta$ price each feature with $R_\theta(s) = \theta^T \phi(s)$ , so path return is $\theta^T \mu_\tau$ . Everything below is matching these counts.

Think of It Like This

Guessing a soup recipe from many bowls

Fifty cooks trained by one chef serve fifty balanced bowls, no two with identical salt and acid. A rigid rule that demands one exact recipe calls the data contradictory. A loose rule that accepts any edible bowl also accepts plain water.

You act like a calm taster. You assume tastier bowls appear exponentially more often, with natural wobble around the best mix. You match the mean salt and acid across bowls, then pick the widest recipe spread that still hits those means. You add no truffle oil the data never demanded.

Where it stops: soup mixes at once in a bowl. Demos unfold step by step, where an early turn changes which states remain reachable.

How It Actually Works

Expert set $\mathcal{D}_E$ holds $N$ paths. Mean expert count is:

\tilde{\mu}_E = \frac{1}{N} \sum_{\tau \in \mathcal{D}_E} \mu_\tau

Max entropy asks for the path spread $P(\tau)$ with largest Shannon entropy $H(P) = -\sum_\tau P(\tau) \log P(\tau)$ that still matches $\tilde{\mu}_E$ . The unique answer is Boltzmann:

P(\tau \mid \theta) = \frac{1}{Z(\theta)} \exp(\theta^T \mu_\tau)

Normalizer $Z(\theta) = \sum_\tau \exp(\theta^T \mu_\tau)$ sums over all paths. Log likelihood of demos is:

\mathcal{L}(\theta) = \theta^T \tilde{\mu}_E - \log Z(\theta)

Gradient is expert minus learner:

\nabla_\theta \mathcal{L} = \tilde{\mu}_E - \mathbb{E}_{P(\tau \mid \theta)}[\mu]

In words: raise weights on features the expert visits more than the current model, lower the rest. Linear rewards make $\mathcal{L}$ concave, so ascent reaches the global peak. Large grids compute the learner mean with soft backward values plus forward visit counts instead of listing every path.

Worked example

Three stops with two features: $\phi(s_1) = [1.0, 0.0]$ , $\phi(s_2) = [0.0, 1.0]$ , $\phi(s_3) = [1.0, 1.0]$ . Two routes reach the goal:

Route 1 through $s_1$ : $\mu_1 = [2.0, 1.0]$ .
Route 2 through $s_2$ : $\mu_2 = [1.0, 2.0]$ .

Weights start at $\theta = [1.0, 0.5]$ . Ten demos split 8 to route 1 and 2 to route 2.

Returns:

R_1 = 1.0 \times 2.0 + 0.5 \times 1.0 = 2.50

R_2 = 1.0 \times 1.0 + 0.5 \times 2.0 = 2.00

Energies $\exp(2.50) \approx 12.1825$ and $\exp(2.00) \approx 7.3891$ . Normalizer:

Z = 12.1825 + 7.3891 = 19.5716

Model chances:

P_1 = 12.1825 / 19.5716 \approx 0.6225

P_2 = 7.3891 / 19.5716 \approx 0.3775

Expert mean:

\tilde{\mu}_E = 0.80 \times [2.0, 1.0] + 0.20 \times [1.0, 2.0] = [1.80, 1.20]

Learner mean:

\mathbb{E}[\mu] = 0.6225 \times [2.0, 1.0] + 0.3775 \times [1.0, 2.0] = [1.6225, 1.3775]

Gradient:

\nabla = [1.80 - 1.6225, 1.20 - 1.3775] = [+0.1775, -0.1775]

Step with rate $1.0$ :

\theta_{new} = [1.0, 0.5] + [+0.1775, -0.1775] = [1.1775, 0.3225]

Expert favors feature one more than the model, so weight one rises and weight two falls. Repeating closes the gap. Those six numbers are what the diagram draws.

Code

This snippet uses only the standard library and prints the same partition, chances, means, gradient, and update.

1import math2
3
4mu1 = [2.0, 1.0]5mu2 = [1.0, 2.0]6theta = [1.0, 0.5]7r1 = theta[0] * mu1[0] + theta[1] * mu1[1]8r2 = theta[0] * mu2[0] + theta[1] * mu2[1]9print([round(r1, 4), round(r2, 4)])  # -> [2.5, 2.0]10e1 = math.exp(r1)11e2 = math.exp(r2)12z = e1 + e213print(round(z, 4))  # -> 19.571614p1 = e1 / z15p2 = e2 / z16print([round(p1, 4), round(p2, 4)])  # -> [0.6225, 0.3775]17expert = [0.8 * mu1[0] + 0.2 * mu2[0], 0.8 * mu1[1] + 0.2 * mu2[1]]18learner = [p1 * mu1[0] + p2 * mu2[0], p1 * mu1[1] + p2 * mu2[1]]19print([round(v, 4) for v in expert])  # -> [1.8, 1.2]20print([round(v, 4) for v in learner])  # -> [1.6225, 1.3775]21grad = [expert[0] - learner[0], expert[1] - learner[1]]22print([round(v, 4) for v in grad])  # -> [0.1775, -0.1775]23print([round(theta[0] + grad[0], 4), round(theta[1] + grad[1], 4)])  # -> [1.1775, 0.3225]

Each print matches the hand path: $Z$ at $19.5716$ , chances $0.6225$ and $0.3775$ , gradient $+0.1775$ and $-0.1775$ , new weights $1.1775$ and $0.3225$ .

Watch Out For

Summing every path on a large grid

Listing all paths to get $Z$ works for two routes and explodes on long horizons. Path count grows exponentially with steps, memory follows, and naive sampling misses the narrow high reward corridors.

Use soft backward values plus forward visit counts on tabular tasks. On continuous control switch to guided sampling that focuses on high reward corridors, or skip $Z$ with an adversarial matcher. Exact sums are for teaching grids only.

Reading matched counts as proof of true reward

Many weight vectors can match the same counts when features overlap. Matching counts guarantees matching return under linear rewards, not recovery of the chef private weights.

Report the fitted weights as one consistent explanation, test the implied policy on held out starts, and add features only when demos demand them. Extra features invent distinctions the expert never showed.

The Quick Version

Reward ambiguity means many rewards fit the same demos, including zero everywhere.
Max entropy picks the widest path spread that still matches expert feature means.
Path chance is $\exp(\theta^T \mu_\tau) / Z(\theta)$ with $Z \approx 19.5716$ in the worked pair.
Gradient is expert mean minus learner mean, here $[+0.1775, -0.1775]$ .
Update moves weights to $[1.1775, 0.3225]$ and repeats until means agree.