Behavioral Cloning

Instead of designing rewards, record an expert and train a policy to copy each move with ordinary supervised learning. It works until one small mistake lands somewhere the expert never visited.

A cloning policy trained on centered driving drifts to position 0.55 for 1.25 meters of error while DAgger corrections hold it to 0.08

Why Does This Exist?

Writing a reward that produces the behavior you want is brutally hard. A driving reward that pays for speed invites tailgating, one that punishes steering jitter produces a car that never turns, and every patch adds a new exploit. When an expert can already do the task, copying them skips reward design entirely.

Behavioral cloning (Pomerleau, ALVINN, 1989) does exactly that. Log expert pairs $\mathcal{D}_E = \{(s_i, a_i^*)\}$ , fit a policy $\pi_\theta(s)$ with classification or regression loss, and deploy with no environment interaction and no value functions. The broader family is surveyed in imitation learning, and the method that fixes its main wound is DAgger. When demonstrations should instead reveal the hidden reward, the route is inverse RL.

The wound is covariate shift. Supervised learning assumes train and test inputs come from the same distribution. A cloned policy breaks that assumption on its first mistake: its own action selects its next input, so one error moves it onto states the expert never showed it. With no recovery examples in the log, the next prediction is worse, and the trajectory unravels.

Think of It Like This

Tracing a calligraphy master with no correction strokes

A student learns calligraphy by tracing a master's clean strokes. Every training sheet shows perfect form, centered on the page. The student copies beautifully while the brush stays where the master put it.

One day the brush slips a millimeter right. No training sheet shows a stroke starting from that wrong spot, because the master never slips. The student has learned perfect strokes but zero recovery strokes, so the correction overshoots left, the next one overshoots further right, and the character collapses off the margin.

Where the analogy stops: ink shows the error plainly and a teacher can grab the student's hand. A neural policy fails silently on unfamiliar pixel patterns and outputs confident nonsense, which is why the fix must supply the missing recovery labels rather than more perfect demonstrations.

How It Actually Works

Let $s$ be a state, $a^*$ the expert's action there, $\pi_\theta(s)$ the learner's prediction, and $\ell$ a supervised loss. Training minimizes error under the expert's own state distribution $d^{\pi^*}$ :

$\min_\theta \, \mathbb{E}_{s \sim d^{\pi^*}}[\ell(\pi_\theta(s), \pi^*(s))]$

Here $\ell$ is mean squared error $\tfrac{1}{2}\lVert \pi_\theta(s) - a^* \rVert^2$ for continuous control or cross entropy $-\log \pi_\theta(a^* \mid s)$ for discrete choices. At deployment the policy instead visits states from its own distribution $d^{\pi_\theta}$ , which is the mismatch that matters.

Why errors grow quadratically

Suppose the per step mistake rate on expert states is $\epsilon$ . Early on the learner matches the expert with probability near $1 - \epsilon$ . But the first slip switches the input distribution from $d^{\pi^*}$ to $d^{\pi_\theta}$ , where the error bound $\epsilon$ no longer applies and mistakes become likely. Each step can add a fresh deviation on top of the accumulated drift, so total regret over horizon $T$ scales as $\mathcal{O}(\epsilon T^2)$ (Ross and Bagnell, 2010). Interactive fixes that ask the expert about the learner's own states restore linear scaling $\mathcal{O}(\epsilon T)$ .

Worked example

A car holds lane center $x^* = 0.0$ with expert law $a^*(x) = -x$ and update $x_{t+1} = x_t + a_t$ . The log only covers $x$ in $[-0.20, 0.20]$ . The cloned policy carries a bias of $+0.05$ .

Step 0 at $x_0 = 0.00$ : expert says $0.00$ , learner does $+0.05$ , next $x_1 = 0.05$ .
Step 1 at $x_1 = 0.05$ : expert says $-0.05$ , learner does $0.00$ , next $x_2 = 0.10$ .
Step 2 at $x_2 = 0.10$ : learner does $+0.10$ , next $x_3 = 0.20$ , the edge of the log.
Step 3 at $x_3 = 0.20$ : learner does $+0.15$ , next $x_4 = 0.35$ , outside everything logged.
Step 4 at $x_4 = 0.35$ : the network saturates and outputs $+0.20$ , next $x_5 = 0.55$ , off the lane.

Cumulative deviation is $0.05 + 0.10 + 0.20 + 0.35 + 0.55 = 1.25$ meters. With DAgger the expert labels the visited $x_1 = 0.05$ with $-0.05$ , the car returns near center, and the trajectory $[0.00, 0.05, 0.01, 0.00, 0.02, 0.00]$ totals $0.05 + 0.01 + 0.00 + 0.02 + 0.00 = 0.08$ meters. The diagram draws the straight expert line, the veering learner through these six positions, and the two totals side by side.

Code

The snippet replays both rollouts so the two totals can be checked directly.

1def cumulative_deviation(path: list[float]) -> float:2    """Sum of absolute lateral positions after the start state."""3    return round(sum(abs(x) for x in path[1:]), 2)4
5
6if __name__ == "__main__":7    bc_path = [0.00, 0.05, 0.10, 0.20, 0.35, 0.55]8    dagger_path = [0.00, 0.05, 0.01, 0.00, 0.02, 0.00]9    bc_total = cumulative_deviation(bc_path)10    dagger_total = cumulative_deviation(dagger_path)11    print(bc_path)12    print(bc_total)13    print(dagger_path)14    print(dagger_total)

1# -> expected output:2[0.0, 0.05, 0.1, 0.2, 0.35, 0.55]31.254[0.0, 0.05, 0.01, 0.0, 0.02, 0.0]50.08

Watch Out For

Copying dashboard lights instead of the road

The symptom: training loss looks excellent but the deployed policy waits for a consequence of the expert's action rather than its cause. The classic case is a brake light indicator that turns on after the expert brakes: the network learns to brake when the light is on, so at deployment it never brakes first and the light never turns on.

The fix: remove non causal signals from the policy inputs, such as internal indicators, history buffers that leak the expert's past action, or telemetry computed from the label. Then test with interventions: perturb the scene in a simulator and confirm the action follows the true cause, like obstacle distance, rather than the spurious cue.

Fixing drift with more perfect demonstrations

The symptom: the team records twice as many flawless expert runs, validation loss drops, and closed loop driving is unchanged. Every new log covers the same centered states, so the recovery region stays empty.

The fix: log the missing distribution instead. Roll out the current policy, have the expert label the visited off center states, and aggregate those pairs into training, which is the DAgger loop. A small amount of on policy correction data beats a large amount of additional perfect data.

The Quick Version

Behavioral cloning turns imitation into supervised learning on logged expert state action pairs.
Deployment breaks the supervised assumption because the learner's actions choose its next inputs.
Small per step errors compound to $\mathcal{O}(\epsilon T^2)$ total regret over horizon $T$ without correction.
DAgger restores $\mathcal{O}(\epsilon T)$ by labeling states the learner actually visits.
In the worked example, pure cloning drifts to 0.55 for 1.25 meters of error while DAgger corrections hold the total to 0.08.