Adversarial Inverse RL (AIRL)

AIRL copies the goal instead of the motions. It splits what the expert wanted from how the old world moved, so the aim still works when the world changes.

A discriminator score of 0.9813 splits into portable reward 2.00 plus shaping 0.35, and the 2.00 survives when the next-state potential drops

Why Does This Exist?

Copying motions fails when the world shifts. GAIL matches the expert's state-action visits, and its discriminator soaks up the training world's physics along with the behavior. Move one hurdle, change the friction, jam a motor, and the copied motions land in empty air. The agent learned how to move, not what to want.

Wanting is a reward function, and recovering rewards from demonstrations is inverse RL. Classical maximum entropy IRL does this honestly and slowly: every gradient step re-solves a forward RL problem, which is hopeless for hard continuous control.

AIRL keeps the adversarial sample efficiency and repairs the transfer. It structures the discriminator so the portable state reward separates from the dynamics-dependent shaping term. The reward exports to new physics. The shaping stays behind.

Think of It Like This

The agility dog that learns the prize

A champion dog runs an agility course: tunnel, red hurdle, finish line with food.

The mimicking student memorizes paw placements and leap timing. While the hurdles stay put, it shines. Slide the red hurdle three meters left and it leaps into empty air. It copied the route, not the reason.

The AIRL student splits the demo in two. The hurdle jump was local maneuvering for that arena layout. The food at the finish was the point. In a new arena it plans a fresh path around the moved hurdles and still sprints to the bowl.

Where it stops: a real dog smells the food. AIRL gets zero reward signal during demos and must isolate the goal purely from the discriminator math, which is why the state-only restriction below carries the whole guarantee.

How It Actually Works

Symbols for this page: $s$ is the state, $a$ the action, $s'$ the next state, $\pi(a \mid s)$ the learner policy, $g_\theta(s)$ the portable state reward network, $h_\phi(s)$ the shaping potential network, $\gamma$ the discount, and $D$ the discriminator.

Maximum entropy IRL says the expert policy looks like $\pi^*(a \mid s) \propto \exp(Q^*(s,a) - V^*(s))$ . AIRL bakes that shape into the discriminator:

D(s, a, s') = \frac{\exp(f(s, a, s'))}{\exp(f(s, a, s')) + \pi(a \mid s)}

Reward split that survives new physics

Reward shaping theory says two rewards share the optimal policy exactly when they differ by $\gamma \Phi(s') - \Phi(s)$ . AIRL parameterizes the advantage to expose that split:

f(s, a, s') = g_\theta(s) + \gamma h_\phi(s') - h_\phi(s)

The $g_\theta(s)$ term sees only the state and carries the goal. The $\gamma h_\phi(s') - h_\phi(s)$ term absorbs layout quirks and value offsets. When $g$ stays state-only, theory says its optimum equals the true reward up to a constant, free of transition dynamics. That is the export license: train in world one, carry $g$ into world two, re-plan there.

The two-step training loop

The discriminator learns binary cross-entropy on expert transitions against policy transitions. The policy trains with policy gradients on the surrogate reward:

\hat{r}(s, a, s') = \log D - \log(1 - D) = f(s, a, s') - \log \pi(a \mid s)

Low policy probability under the current $\pi$ raises the surrogate, which pushes the policy toward expert-like transitions without ever copying joint angles directly.

Worked example with discriminator 0.9813

One transition carries $g_\theta(s) = 2.0$ , potentials $h_\phi(s) = 1.0$ and $h_\phi(s') = 1.5$ , discount $\gamma = 0.9$ , and policy probability $\pi(a \mid s) = 0.20$ .

Shaped advantage:

f = 2.0 + 0.9 \times 1.5 - 1.0 = 2.0 + 1.35 - 1.00 = 2.35

The portable part contributes $2.00$ and the shaping contributes $0.35$ . Discriminator:

\exp(2.35) \approx 10.4856, \quad D = \frac{10.4856}{10.4856 + 0.20} \approx 0.9813

Near one, so the discriminator calls this expert-like. Surrogate reward with $\log(0.20) \approx -1.6094$ :

\hat{r} = 2.35 - (-1.6094) = 3.9594

Now change the world. Higher friction routes the same $(s, a)$ to $s''$ with $h_\phi(s'') = 0.80$ . The shaping becomes $0.9 \times 0.80 - 1.0 = -0.28$ and the advantage drops to $1.72$ . The portable $g_\theta(s) = 2.00$ does not move. The agent re-optimizes $g$ under the new physics instead of replaying stale motions.

Code

A short check of the worked numbers above, with type hints throughout:

1import math2
3
4def shaped_advantage(g: float, h: float, h_next: float, gamma: float) -> float:5    return g + gamma * h_next - h6
7
8def discriminator(f: float, pi: float) -> float:9    expf = math.exp(f)10    return expf / (expf + pi)11
12
13g = 2.014h = 1.015h_next = 1.516gamma = 0.917pi = 0.2018
19f = shaped_advantage(g, h, h_next, gamma)20d = discriminator(f, pi)21r = f - math.log(pi)22print([round(f, 2), round(math.exp(f), 4), round(d, 4), round(r, 4)])23# -> [2.35, 10.4856, 0.9813, 3.9594]24
25f2 = shaped_advantage(g, h, 0.80, gamma)26print([round(f2, 2), round(g, 2)])27# -> [1.72, 2.0]

1# -> expected output:2[2.35, 10.4856, 0.9813, 3.9594]3[1.72, 2.0]

Watch Out For

Letting the reward see the action

Widening $g$ to $g(s, a)$ feels more expressive and breaks the guarantee. With actions inside, the shaping term can no longer absorb dynamics uniquely, and the learned reward entangles with the training world's transitions. Transfer to new friction or payloads then produces erratic policies that exploit stale shaping.

Keep $g$ state-only. That single restriction is what makes the recovered reward portable.

The Quick Version

AIRL structures its discriminator as $D = \exp(f)/(\exp(f) + \pi)$ with $f(s,a,s') = g(s) + \gamma h(s') - h(s)$ .
The state-only $g(s)$ carries the goal and exports across dynamics, while $h$ absorbs training-world shaping.
Unlike GAIL motion matching, the recovered goal re-plans under new physics instead of replaying stale paths.
One worked step gives advantage $2.35$ , discriminator $0.9813$ and surrogate reward $3.9594$ , with $g = 2.00$ unchanged when shaping drops the total to $1.72$ .
Conditioning the reward on actions voids the disentanglement, so keep $g$ state-only.