Apprenticeship Learning via Inverse RL

You do not need the expert's hidden reward to match them. Match how often their behavior visits each feature, and the return gap stays small no matter the reward weights.

Learner feature points move toward the expert point in 2D, shrinking the gap from 0.7211 to 0.3536 to 0.0943 until it passes below 0.10

Why Does This Exist?

Copying actions directly fails in a familiar way. Behavioral cloning fits state action pairs, then one small steering mistake puts the car somewhere the expert never visited. Errors compound along the trajectory, so performance can degrade with the square of the horizon.

Guessing the hidden reward is no easier. Inverse reinforcement learning tries to recover the reward that made the expert optimal, but the problem is underdetermined. Many rewards, including all zeros, make any demo look optimal.

Apprenticeship learning asks for less and still gets a guarantee. Assume reward is linear in known features, $R(s) = w^T \phi(s)$ with feature vector $\phi(s)$ and weights $w$ . Then matching the expert's discounted feature counts is enough. If your feature averages sit within $\epsilon$ of the expert's, your return sits within $\epsilon$ of the expert's for any valid $w$ . This page covers that matching loop. Neighbor pages cover inferring a full reward in inverse reinforcement learning, handling ambiguity in maximum entropy IRL, and skipping rewards with adversarial matching in GAIL.

You only need Markov decision processes first: states, actions, transitions, discount $\gamma$ , and expected return.

Think of It Like This

An apprentice cook matching the master's pantry totals

You apprentice under a chef whose exact taste weights you will never know. Copying every wrist flick fails the moment the kitchen changes. Instead you match pantry totals over a month: kilos of tomatoes, liters of stock, grams of salt.

Those totals are the feature expectations $\mu$ . The judge's hidden scoring weights are $w$ . If your monthly totals sit close to the master's on every ingredient, any judge who scores as a weighted mix of those ingredients will score you close to the master.

Where it stops: a real pantry has substitutes and spoilage, and totals hide order and timing. The math here assumes the listed features capture everything the reward cares about, and that you can roll out fresh trajectories to measure your own totals exactly.

How It Actually Works

Write $\mu(\pi)$ for the discounted feature expectation of a policy $\pi$ :

\mu(\pi) = E\left[ \sum_{t=0}^{\infty} \gamma^t \phi(s_t) \mid \pi \right]

Here $\gamma$ in $[0, 1)$ is the discount factor, $s_t$ is the state at step $t$ , and $E$ averages over start states and transitions. The expert's version $\mu_E$ is the same average measured from demos. Return under weights $w$ is then a dot product:

J(\pi) = w^T \mu(\pi)

So the gap obeys Cauchy-Schwarz. With $\|w\|_2 \le 1$ :

|J(\pi) - J(\pi_E)| \le \|\mu(\pi) - \mu_E\|_2

That is the whole guarantee. Get the feature distance below $\epsilon$ and the return gap follows.

The projection loop

The Abbeel and Ng projection method builds this match iteratively. Keep the convex hull of feature points found so far, project the expert onto it to get $\bar{\mu}$ , point the next reward from $\bar{\mu}$ toward $\mu_E$ , solve that forward RL problem, add the new point, repeat. Stop when $t = \|\mu_E - \bar{\mu}\|_2 \le \epsilon$ .

Worked example

Take two features, $\epsilon = 0.10$ , and expert point:

\mu_E = [0.80, 0.60]

Start from a random policy with $\mu^{(0)} = [0.20, 0.20]$ , so $\bar{\mu}^{(0)} = [0.20, 0.20]$ . The first gap vector is $[0.60, 0.40]$ and:

t_0 = \sqrt{0.60^2 + 0.40^2} = \sqrt{0.52} \approx 0.7211

That exceeds $0.10$ , so solve the induced reward and get $\mu^{(1)} = [0.55, 0.35]$ . The new gap is $[0.25, 0.25]$ and:

t_1 = \sqrt{0.25^2 + 0.25^2} = \sqrt{0.125} \approx 0.3536

Still above threshold. Solve again and get $\mu^{(2)} = [0.75, 0.52]$ . The gap is $[0.05, 0.08]$ and:

t_2 = \sqrt{0.05^2 + 0.08^2} = \sqrt{0.0089} \approx 0.0943

Now $0.0943 \le 0.10$ , so stop. The return gap to the expert is at most $0.0943$ for any allowed $w$ .

Code

This snippet recomputes the three margins above. Nothing else is hidden inside.

1import numpy as np2
3
4def margin(mu_e: np.ndarray, mu_bar: np.ndarray) -> float:5    """Euclidean distance between expert and learner feature points."""6    return float(np.linalg.norm(mu_e - mu_bar))7
8
9mu_e = np.array([0.80, 0.60])10mu0 = np.array([0.20, 0.20])11mu1 = np.array([0.55, 0.35])12mu2 = np.array([0.75, 0.52])13
14print(f"{margin(mu_e, mu0):.4f}")  # -> 0.721115print(f"{margin(mu_e, mu1):.4f}")  # -> 0.353616print(f"{margin(mu_e, mu2):.4f}")  # -> 0.0943

The prints match the hand arithmetic: $0.7211$ , then $0.3536$ , then $0.0943$ below the $0.10$ stop line.

Watch Out For

Features that miss what the expert cares about

The guarantee assumes the true reward is linear in your $\phi$ . If the expert cares about something you never encoded, say smoothness or risk, matching your features proves nothing.

Symptom: margins shrink to zero yet the learned policy still drives badly or looks nothing like the demos. Fix: add the missing feature and rerun. The failure is in representation, not in the projection math.

Paying for a full RL solve every round

Each outer step trains a fresh policy for a new reward guess. With deep policies that means millions of env steps per iteration.

Symptom: two outer loops take longer than the whole baseline budget. Fix: warm start each policy from the previous one, or move to a single loop method such as GAIL when the budget bites.

The Quick Version

Apprenticeship learning matches discounted feature averages $\mu$ , not raw actions and not the hidden weights $w$ .
The bound $|J - J_E| \le \|\mu - \mu_E\|_2$ holds for any allowed linear reward once features match.
The projection loop points each new reward from the current hull toward $\mu_E$ and stops below $\epsilon$ .
In the 2D example the gap falls $0.7211$ to $0.3536$ to $0.0943$ , so the return gap is at most $0.0943$ .
Bad features or a costly inner RL loop are the two failures to check first.