Generative Adversarial Imitation Learning (GAIL)

GAIL copies an expert without writing a reward function. A discriminator learns to spot learner moves, and its suspicion becomes the training reward.

A discriminator scores expert and learner moves, and scores 0.8 and 0.1 turn into rewards 1.61 and 0.11 that train the policy

Why Does This Exist?

Writing rewards by hand is brittle. A driving reward that says stay centered can produce a car that never changes lanes. Imitation learning avoids hand rewards by copying demos, but plain copying drifts: small errors put the learner where the expert never went, and errors compound.

Classical inverse RL tried to infer the reward first, then solve a full RL problem inside every update. In high dimensional control that inner loop is far too slow. Ho and Ermon showed in 2016 that the regularized inverse problem is dual to matching state action visitation directly. GAIL uses that duality and trains like a GAN, so no inner MDP solve is needed. Use maximum entropy IRL when you want the reward itself explained; use GAIL when you want the policy with less compute.

The one idea you need first

The occupancy measure $\rho_{\pi}(s, a)$ is the discounted frequency of visiting $(s, a)$ under $\pi$ . If two policies visit states the same way, they earn the same return under any reward. GAIL matches those frequencies instead of recovering a reward.

Think of It Like This

A forger and a sharp curator

A forger paints in the style of a master. The curator knows the real collection and scores each canvas for authenticity. The forger never sees a chemistry report, only the score. High score means keep that brushwork, low score means change it.

Over rounds the forger gets harder to catch. At the limit the curator guesses at chance. That limit is the GAIL equilibrium at $D = 0.5$ . The forger is $\pi$ , the curator is $D$ , and the score turns into reward.

Where it stops: paintings do not fight back. Actions change the next state, so GAIL must match whole trajectories, not single stills.

How It Actually Works

GAIL plays a minimax game between policy $\pi_{\theta}$ and discriminator $D_{\psi}(s, a)$ in $(0, 1)$ :

\min_{\theta} \max_{\psi} \mathbb{E}_E[\log D_{\psi}] + \mathbb{E}_{\pi_{\theta}}[\log(1 - D_{\psi})] - \lambda \mathcal{H}(\pi_{\theta})

In words: $E_E$ averages over expert demos, $E_{\pi}$ averages over learner rollouts, $\mathcal{H}$ is entropy to keep exploring, $\lambda$ weights it. The discriminator wants $1$ on expert pairs and $0$ on learner pairs. The policy wants learner pairs that score near $1$ .

Discriminator step

Binary cross entropy on one expert and one learner pair is $-\log D_E - \log(1 - D_L)$ . For fixed $\pi$ the best discriminator is the occupancy ratio $\rho_E / (\rho_E + \rho_{\pi})$ . When visitations match, that ratio is $0.5$ everywhere.

Policy step with surrogate reward

Freeze $D$ , then reward each learner step by:

r_{\text{GAIL}}(s, a) = -\log(1 - D_{\psi}(s, a))

In words: confident expert-like moves get large reward, obvious learner moves get near zero. Update $\pi$ with PPO or TRPO on those rewards.

Worked example

Take $D = 0.80$ for a near expert move and $D = 0.10$ for a noisy move:

$r(0.80) = -\log(1.0 - 0.80) = -\log(0.20) = 1.6094$
$r(0.10) = -\log(1.0 - 0.10) = -\log(0.90) = 0.1054$

The expert-like move pays over 15 times more, so gradients point toward expert behavior.

Now one discriminator batch with $D_E = 0.70$ and $D_L = 0.40$ . Loss is $-\log(0.70) - \log(0.60) = 0.3567 + 0.5108 = 0.8675$ . Gradients are $-1 / 0.70 = -1.4286$ on the expert output and $+1 / 0.60 = +1.6667$ on the learner output, pushing the two apart.

Code

Same arithmetic in code, with checks:

1import math2
3
4def surrogate_reward(d: float) -> float:5    """GAIL reward r = -log(1 - D), clipped for safety."""6    c = min(1.0 - 1e-7, max(1e-7, d))7    return -math.log(1.0 - c)8
9
10def bce_loss(d_expert: float, d_learner: float) -> tuple[float, float, float]:11    """Return loss, dL/dDE, dL/dDL for one pair."""12    loss = -math.log(d_expert) - math.log(1.0 - d_learner)13    return loss, -1.0 / d_expert, 1.0 / (1.0 - d_learner)14
15
16print(round(surrogate_reward(0.80), 4))  # -> 1.609417print(round(surrogate_reward(0.10), 4))  # -> 0.105418print([round(x, 4) for x in bce_loss(0.70, 0.40)])  # -> [0.8675, -1.4286, 1.6667]

Outputs match the hand math: $1.6094$ , $0.1054$ , and loss $0.8675$ with the two gradients.

Watch Out For

Letting the discriminator get too strong

Early learner rollouts look nothing like demos, so classification is easy. The discriminator hits near perfect accuracy, outputs near $0$ on learner moves, and reward variance collapses. The symptom is loss near zero while the policy stops improving.

Train the discriminator one small step per policy round, never to convergence. Add spectral norm or a gradient penalty, and keep entropy bonus on so the policy keeps visiting new pairs.

Reading the surrogate as the true reward

GAIL reward measures fooling ability, not task progress. A policy can match occupancy quirks while missing intent, especially with few demos. The symptom is rising surrogate reward with flat or falling task score.

Track task return separately, grow demo coverage, and compare against inverse RL baselines when you need a transferable reward rather than a copier.

The Quick Version

GAIL skips inner loop inverse RL by matching state action visitation directly.
A discriminator learns expert $1$ versus learner $0$ , with optimum at occupancy ratio and $0.5$ at match.
The policy trains on $r = -\log(1 - D)$ , so $0.8$ pays $1.61$ and $0.1$ pays $0.11$ .
One batch at $0.70$ and $0.40$ gives loss $0.8675$ with gradients minus $1.43$ and plus $1.67$ .
Keep discriminator weak and entropy on, or gradients vanish early.