Model-Agnostic Meta-Learning (MAML) in RL

MAML learns a starting policy that sits close to many tasks at once, so one or two gradient steps on a new task land near good behavior.

Shared start 2.50 adapts to 2.40 and 2.80 in one step and meta update moves start to 2.628 with gradient 1.28

Why Does This Exist?

Deep RL is hungry. A new friction, speed, or mass can force retraining from scratch for millions of steps. Meta-RL asks instead for fast learning across a task family $p(T)$ , so a new task needs only a few trials.

Early meta methods hid adaptation inside recurrent memory. That ties learning to one architecture and struggles outside the training range. MAML grounds adaptation in gradient descent itself. It learns an init $\theta$ that is not best anywhere but is easy to adjust everywhere, so one policy gradient step per task does the work. Any differentiable policy fits. For the policy gradient used inside see policy gradient. For memory based meta learning see RL squared.

The one idea you need first

Inner loop adapts to one task. Outer loop improves the starting point across tasks. Call $\alpha$ the inner rate and $\beta$ the outer rate. Inner data comes from the start policy, outer scores come from adapted policies.

Think of It Like This

Base camp on the saddle

You must guide climbs on several peaks. Camp in the east valley and peak 1 is quick but peak 2 takes days to reach. Camp on the central saddle and no summit is at your door, yet each summit is one short ridge sprint away.

MAML picks the saddle. Zero shot views are modest. One step sprints $\theta_1'$ and $\theta_2'$ reach each peak fast. Each week you shift camp by the average miss, which is the outer update with rate $\beta$ .

Where it stops: mountains stay put. In RL moving camp changes the scouting paths you collect, so the start also shapes exploration, not just distance.

How It Actually Works

Policy $\pi_\theta(a \mid s)$ has weights $\theta$ . For task $T_i$ , collect rollouts $D_i$ from $\pi_\theta$ , form return $J_i(\theta)$ , and take one ascent step:

$\theta_i' = \theta + \alpha \nabla_\theta J_i(\theta)$

Then sample fresh rollouts from adapted $\pi_{\theta_i'}$ and score post performance. The meta objective sums it:

$\mathcal{J}(\theta) = \sum_i J_i(\theta_i')$

Outer update with rate $\beta$ :

$\theta \leftarrow \theta + \beta \sum_i \nabla_\theta J_i(\theta_i')$

Chain rule gives a Hessian term $I + \alpha \nabla^2 J_i(\theta)$ , so full MAML is second order. First order MAML drops the Hessian and treats $d\theta_i'/d\theta$ as identity, which cuts variance at small cost in speed.

Worked example

Linear policy with $s = 1.0$ , reward $r_i = -(\theta - k_i)^2$ , targets $k_1 = 2.0$ and $k_2 = 4.0$ , $\alpha = 0.10$ , $\beta = 0.10$ , start $\theta = 2.50$ .

Task 1: $r_1 = -(0.50)^2 = -0.2500$ . Gradient $-2(0.50) = -1.0000$ . Adapt $\theta_1' = 2.50 - 0.10 = 2.4000$ . New $r_1' = -(0.40)^2 = -0.1600$ .

Task 2: $r_2 = -(-1.50)^2 = -2.2500$ . Gradient $-2(-1.50) = +3.0000$ . Adapt $\theta_2' = 2.50 + 0.30 = 2.8000$ . New $r_2' = -(-1.20)^2 = -1.4400$ .

Totals: zero shot $-2.5000$ , adapted $-1.6000$ . Meta slopes with $d\theta'/d\theta = 0.80$ : task 1 gives $-0.80 \times 0.80 = -0.6400$ , task 2 gives $+2.40 \times 0.80 = +1.9200$ , sum $+1.2800$ . New start:

$\theta_{new} = 2.50 + 0.10 \times 1.28 = 2.6280$

Start shifts toward midpoint $3.0$ . The diagram draws $2.50$ , $2.40$ , $2.80$ , and $2.628$ on one line.

Code

This replays the inner and outer arithmetic with the same numbers.

1from typing import List, Tuple2
3
4def reward(theta: float, k: float) -> float:5    """Quadratic task reward."""6    return -((theta - k) ** 2)7
8
9def grad(theta: float, k: float) -> float:10    """d reward / d theta."""11    return -2.0 * (theta - k)12
13
14if __name__ == "__main__":15    theta, alpha, beta = 2.50, 0.10, 0.1016    t1, t2 = 2.0, 4.017    th1 = theta + alpha * grad(theta, t1)18    th2 = theta + alpha * grad(theta, t2)19    print(f"th1 = {th1:.4f}")  # -> th1 = 2.400020    print(f"th2 = {th2:.4f}")  # -> th2 = 2.800021    print(f"adapt = {reward(th1, t1) + reward(th2, t2):.4f}")  # -> adapt = -1.600022    mg = (-2.0 * (th1 - t1) * 0.8) + (-2.0 * (th2 - t2) * 0.8)23    print(f"meta grad = {mg:.4f}")  # -> meta grad = 1.280024    print(f"theta new = {theta + beta * mg:.4f}")  # -> theta new = 2.6280

Watch Out For

Noisy Hessians shake the outer loop

Full MAML multiplies two sampled gradients through the Hessian, so variance compounds. Symptom: outer loss swings, entropy collapses, and exploration dies early.

Use first order MAML for most RL runs, normalize advantages per task so large reward scales do not dominate, and keep inner steps to one or two until the start stabilizes. Add trust region clipping when tasks fight each other.

The Quick Version

MAML learns an init $\theta$ so one gradient step per task reaches high return, with inner rate $\alpha$ and outer rate $\beta$ .
Adaptation is plain gradient ascent, so any differentiable policy works with no special memory.
In the 1D check start $2.50$ adapts to $2.40$ and $2.80$ , lifting total from $-2.50$ to $-1.60$ .
Meta gradient $+1.28$ moves start to $2.628$ , toward the task midpoint.
In RL the start also shapes exploration, and first order updates tame Hessian noise.