Meta-Reinforcement Learning

Meta RL trains one starting policy across many small worlds so a new world takes a few practice tries instead of months of retraining.

Start 1.0 adapts to 1.2 and 1.6 on tasks 2.0 and 4.0, then the meta update moves the start to 1.128

Why Does This Exist?

Plain RL starts from scratch on every task. A walker tuned for speed 2.0 falls when friction shifts or the target moves to 3.5, and retraining burns millions of steps again. That is fine for one lab demo and painful everywhere else.

Meta RL changes the objective. Instead of one fixed MDP, it trains over a task distribution $p(\mathcal{T})$ where members share states and actions but differ in dynamics or rewards. The score is post adaptation return after one or a few trial episodes, not day one return. This page shows the two loop math and one hand computed meta step. For the gradient starter method see MAML in RL, for reusing skills see transfer and generalization, and for no shot behavior see zero and few shot RL.

You need one idea first. A task here is one MDP. The distribution is the bag of MDPs you train over, with some held out to test fast adaptation.

Think of It Like This

A decathlete learning a new event

A sprinter tuned for one dry track trips on wet grass with hurdles. A decathlete trains across ten events and builds shared primitives: balance, drive, pacing, quick calibration.

Hand the decathlete a new event and two warm up tries are enough to set stride and post a solid score. No ten year rebuild.

Where the analogy stops: athletes keep fixed bodies while meta RL updates weights through nested gradients, memory states, or inferred task codes, and it can forget if the training bag is too narrow.

How It Actually Works

Let $\theta$ be shared start weights, $\alpha$ inner step size, and $\beta$ outer step size. Let $\mathcal{D}_i^{\text{train}}$ be trial episodes on task $\mathcal{T}_i$ and $\mathcal{D}_i^{\text{test}}$ fresh episodes after adaptation. Let $\mathcal{L}_i$ be task loss.

Inner loop adapts once per task:

$\theta_i' = \theta - \alpha \nabla_\theta \mathcal{L}_i(\theta, \mathcal{D}_i^{\text{train}})$

Outer loop scores the adapted weights and moves the start:

$\min_\theta \mathbb{E}_{\mathcal{T}_i \sim p(\mathcal{T})} [\mathcal{L}_i(\theta_i', \mathcal{D}_i^{\text{test}})]$

$\theta \leftarrow \theta - \beta \nabla_\theta \sum_i \mathcal{L}_i(\theta_i', \mathcal{D}_i^{\text{test}})$

In words: adapt fast inside, then shift the start toward points from which many tasks are one step away. Three builds use this differently: gradient starts like MAML take explicit steps, recurrent policies carry memory across episodes, and context methods infer a task code $z$ and condition on it.

Worked example

One weight policy $a = \theta \times s$ with $s = 1.0$ , so $a = \theta$ . Loss is $\mathcal{L}_i = \frac{1}{2}(\theta - v_i^*)^2$ with targets $v_1^* = 2.0$ and $v_2^* = 4.0$ . Start $\theta = 1.0$ , $\alpha = 0.20$ , $\beta = 0.10$ .

Task 1 gradient $1.0 - 2.0 = -1.0$ . Adapted $\theta_1' = 1.0 - 0.20 \times (-1.0) = 1.2$ . Loss $\frac{1}{2}(1.2 - 2.0)^2 = 0.32$ .
Task 2 gradient $1.0 - 4.0 = -3.0$ . Adapted $\theta_2' = 1.0 - 0.20 \times (-3.0) = 1.6$ . Loss $\frac{1}{2}(1.6 - 4.0)^2 = 2.88$ .
Meta loss $(0.32 + 2.88) / 2 = 1.60$ .

Through the inner step each loss derivative gains factor $1 - \alpha = 0.80$ . Task grads become $-0.64$ and $-1.92$ , average $-1.28$ . New start $1.0 - 0.10 \times (-1.28) = 1.128$ . The diagram shows the same 1.2, 1.6, and 1.128.

Code

The snippet runs one inner adapt plus one meta update.

1def adapt(theta: float, target: float, alpha: float = 0.2) -> float:2    """One inner gradient step on squared tracking loss."""3    return theta - alpha * (theta - target)4
5
6if __name__ == "__main__":7    t1 = adapt(1.0, 2.0)8    t2 = adapt(1.0, 4.0)9    print(round(t1, 4))10    print(round(t2, 4))11    print(round((0.5 * (t1 - 2.0) ** 2 + 0.5 * (t2 - 4.0) ** 2) / 2, 4))12    print(round(1.0 - 0.1 * -1.28, 4))

1# -> expected output:21.231.641.651.128

Watch Out For

Training on a narrow task bag and memorizing the average

The symptom: meta training loss falls, but a held out friction or goal breaks adaptation. The start collapsed to a compromise policy for the few speeds you sampled rather than a real adaptation rule.

The fix: randomize dynamics and goals over wide ranges, and split tasks into train and test bands. Ship only when post adapt return beats zero shot return on unseen bands.

The Quick Version

Meta RL trains a shared start $\theta$ so new tasks from $p(\mathcal{T})$ solve in one to three tries.
The inner loop adapts per task while the outer loop moves the start to help all tasks at once.
From 1.0 the example reaches 1.2 and 1.6 on targets 2.0 and 4.0, with meta loss 1.60.
The meta gradient is $-1.28$ , so the start moves to $1.128$ in one outer step.
Wide task variety plus held out test tasks keeps the method honest.