Dataset Aggregation (DAgger)

Why Does This Exist?

Copying an expert looks like supervised learning: gather state-action pairs and fit. That is behavioral cloning, and it works until the first mistake. Training saw only expert states, so the first wobble lands somewhere the dataset never covered. The next move is worse, the one after worse still, and errors stack quadratically in horizon $T$.

Concretely, a per-step slip rate $\epsilon$ grows into total cost near $\epsilon T^2$ under cloning, because one early slip poisons every later step. This page teaches the interactive repair. For the broader family, see imitation learning and its adversarial cousin GAIL.

Think of It Like This

How It Actually Works

Round one is plain cloning on expert demos $\mathcal{D}_1$. Later rounds mix execution as $\pi_n = \beta_n \pi^* + (1 - \beta_n) \hat{\pi}_n$, with $\beta_n$ decaying so the learner soon drives alone. Each round rolls the mixer, visits states $s_t$ from its own distribution, asks the expert for $a_t^* = \pi^*(s_t)$ on exactly those states, merges them as $\mathcal{D}_n = \mathcal{D}_{n-1} \cup \{(s_t, a_t^*)\}$, and retrains. Training finally matches deployment because the data comes from the learner's own wobbles.

Theory reads this as online learning with no regret. Average training slip $\epsilon_N$ plus a $1/N$ term bounds the deployed slip, so total cost grows like $\epsilon T$ instead of $\epsilon T^2$. The square is gone because recovery moves are now in the data.

Worked example

Lane position $x$ with dynamics $x_{t+1} = x_t + a_t$ and expert steer $\pi^*(x) = -x$. Demos cover only $x$ in $[-0.10, 0.10]$.

A gust drops the car at $x_0 = 0.400$ where the expert wants $-0.400$. The cloned policy never trained there and steers $-0.109$, landing at $0.291$. Squared error on the shoulder: $(-0.109 + 0.400)^2 = 0.291^2 \approx 0.0847$. Drift continues through $0.141$ before crawling home.

DAgger labels the visited trio $0.400$, $0.291$, $0.141$ with expert steers $-0.400$, $-0.291$, $-0.141$ and retrains. From $0.400$ again it now steers $-0.396$, landing at $0.004$. New error: $(-0.396 + 0.400)^2 = 0.004^2 \approx 0.000016$. Three corrections turn a long slide into a one-step save.

Code

The shoulder check before and after, in plain Python:

1expert: float = -0.4002before: float = -0.1093after: float = -0.3964print(f"{(before - expert) ** 2:.4f}")  # -> 0.08475print(f"{(after - expert) ** 2:.6f}")  # -> 0.000016

The first line is the cloning wobble, the second the retrained save. Same start, same expert, three extra labels between them.

Watch Out For

The Quick Version

• Cloning trains on expert states but deploys on learner states, so one slip compounds toward $\epsilon T^2$.
• DAgger rolls the learner, labels its own visits with $a_t^* = \pi^*(s_t)$, merges, and retrains.
• Our lane trio of corrections cuts shoulder error from $0.0847$ to $0.000016$.
• The bound drops to $\epsilon T$ because recovery moves join the data.
• Gate queries on doubt or danger when the expert is human.

What to Read Next

• Behavioral Cloning
• Imitation Learning
• Generative Adversarial Imitation Learning (GAIL)