RL Squared (Fast RL via Slow RL)

RL squared trains one recurrent policy slowly across many tasks so that, on a new task, it adapts fast inside its own memory with no weight updates.

Slow training builds memory that carries Trial 1 evidence across the boundary so Trial 2 turns nearly greedy at 0.98

Why Does This Exist?

Standard RL starts over on every new task. Teach a robot one goal and it needs millions of steps. Move the goal across the room and training restarts from zero. That is fine for one fixed job and painful for a family of related jobs.

Meta-reinforcement learning changes the objective: train across a task distribution $p(\mathcal{T})$ so a new task takes a trial or two. MAML in RL answers with gradients: learn weights that adapt in one gradient step at test time. That works and still pays for gradients plus fresh rollouts on every new task.

RL squared, written RL2 and introduced by Duan and coauthors in 2016 alongside closely related work by Wang and coauthors, asks a sharper question. What if adaptation lived in memory instead of gradients. Train one recurrent policy slowly with ordinary RL. At test time freeze every weight and let the hidden state do the fast learning as actions, rewards, and done flags stream past. No optimizer runs at deployment. The policy just remembers.

This page covers that memory trick: the trial as sequence setup, the input format that makes it work, and a two trial bandit trace you can check. For the parent framing read meta-reinforcement learning first. For the memory cell see recurrent networks in RL. For the gradient sibling compare MAML in RL.

The one idea you need first is a hidden state. Write the memory vector as $h_t$ . A recurrent update $f_\theta$ with weights $\theta$ maps $h_t$ plus the current input to $h_{t+1}$ . Here $\theta$ changes only in slow training, while $h$ changes every step. RL2 uses $h$ as the fast learner and $\theta$ as the slow learner.

Think of It Like This

The relief cook with one sticky note

A relief cook covers shifts at many small cafes. Cooking school was the slow loop: months across kitchens that taught knife grip, heat control, and how to read tickets. No single menu was the point. The point was becoming someone who walks into a strange kitchen and gets fast by lunch.

Each shift runs two rushes. The cook carries one sticky note, the fast loop. First rush: table 4 sends back soup, table 7 wants extra salt, the grill runs hot. Every clue goes on the note. Between rushes the cook keeps the note. Second rush: same room, and the note says salty crowd plus hot grill, so seasoning drops and proteins go on later. Orders land.

Shift ends, note goes in the trash. Next cafe means a blank note, because yesterday's regulars say nothing about today. And the cook never rewrites cooking school mid shift. School shaped how the note gets used. The note does the day's learning.

Where it stops: the cook chooses words, but a hidden state is a vector shaped only by gradients that nobody reads directly. You infer what it kept from how Trial 2 behavior changes, the way you judge the note by the second rush rather than by reading it.

How It Actually Works

The trial as sequence trick

Fix notation. A task $\mathcal{T}_i$ is one MDP drawn from $p(\mathcal{T})$ . At step $t$ the policy sees observation $o_t$ plus its own recent past: previous action $a_{t-1}$ , previous reward $r_{t-1}$ , and done flag $d_{t-1}$ marking trial ends. With $h_t$ the running memory:

$h_{t+1} = f_\theta(h_t, o_t, a_{t-1}, r_{t-1}, d_{t-1})$

$a_t \sim \pi_\theta(a_t \mid o_t, h_t)$

Here $f_\theta$ is the recurrent update, a GRU in the original paper, and $\pi_\theta$ is the action head reading memory. The extra inputs matter. Without previous reward and done flag the memory cannot tell which trial it is in or what paid off, and adaptation never appears.

Training, the slow loop, runs a standard optimizer over episodes where one episode spans several trials on the same sampled task. Trial 1 ends, $d$ marks the boundary, and $h$ is kept. Only the episode end clears $h$ to zero. That kept across trials and cleared across episodes rule turns a plain recurrent policy into a meta learner. Trial 2 starts already knowing what Trial 1 found.

Test time freezes $\theta$ fully. Trial 1 explores and loads evidence into $h$ . Trial 2 conditions on that loaded $h$ and exploits. MAML moves weights by gradient descent on the new task. RL2 moves nothing. It remembers.

The trade is clear. You buy adaptation with zero test time gradients and pay with harder training plus a narrower comfort zone. The dynamics must discover near Bayes trial behavior from scratch, and only for the trained family. Outside that family Trial 2 looks no smarter than Trial 1. Use RL2 when test tasks come from the same parametric family and test compute forbids fine-tuning.

Worked example

Two mirrored bandit tasks with prior 0.5 each. Task A pays 1 on arm L with probability 0.8 and on arm R with 0.2. Task B mirrors it. One episode means two trials of three pulls on the same hidden task. Suppose the hidden task is A and Trial 1 observes L to 1, R to 0, L to 1.

Likelihood under A multiplies per pull, with the middle term $P(R = 0 \mid A) = 1 - 0.2 = 0.8$ :

$P(\text{obs} \mid A) = 0.8 \times 0.8 \times 0.8 = 0.5120$

Under B each matching term is 0.2:

$P(\text{obs} \mid B) = 0.2 \times 0.2 \times 0.2 = 0.0080$

Flat prior gives posterior:

$P(A \mid \text{obs}) = \frac{0.5120}{0.5120 + 0.0080} = 0.9846$

After three pulls the agent should be 98.5 percent sure the left arm is good. A trained RL2 policy shows this shape: Trial 1 near $P(L) = 0.50$ , Trial 2 near $P(L) = 0.98$ , exploiting through $h$ alone. Expected return makes the gain concrete. An adaptive second trial earns 0.8 per pull when right and 0.2 when wrong:

$0.9846 \times 0.8 + 0.0154 \times 0.2 = 0.7908$

That is $2.3723$ over three pulls against $1.5000$ for a uniform policy at 0.5 per pull. Memory is worth about 0.87 extra rewards per trial here. The diagram draws this trace: Trial 1 evidence in, posterior 0.9846 at the boundary, greedy Trial 2 out.

Code

Standalone check of the Trial 1 posterior and the Trial 2 gain with the same numbers:

1from typing import List, Tuple2
3Pull = Tuple[str, int]4
5
6def likelihood(pulls: List[Pull], good_arm: str, p_good: float = 0.8, p_bad: float = 0.2) -> float:7    """Likelihood of outcomes if good_arm is the high-pay arm."""8    like = 1.09    for arm, reward in pulls:10        p_one = p_good if arm == good_arm else p_bad11        like *= p_one if reward == 1 else (1.0 - p_one)12    return like13
14
15def trial_posterior(pulls: List[Pull]) -> Tuple[float, float, float]:16    """Return likelihood under A, likelihood under B, and posterior of A."""17    like_a = likelihood(pulls, good_arm="L")18    like_b = likelihood(pulls, good_arm="R")19    return like_a, like_b, like_a / (like_a + like_b)20
21
22if __name__ == "__main__":23    trial1: List[Pull] = [("L", 1), ("R", 0), ("L", 1)]24    like_a, like_b, post_a = trial_posterior(trial1)25    print(f"like_a={like_a:.4f} like_b={like_b:.4f} post_a={post_a:.4f}")26    # -> like_a=0.5120 like_b=0.0080 post_a=0.984627    adaptive = post_a * 0.8 + (1.0 - post_a) * 0.228    print(f"adaptive per pull={adaptive:.4f} x3={adaptive * 3:.4f} uniform x3=1.5000")29    # -> adaptive per pull=0.7908 x3=2.3723 uniform x3=1.5000

Both prints match the hand trace: 0.5120 versus 0.0080 into 0.9846, then 0.7908 per pull into 2.3723.

Watch Out For

Clearing memory at the trial boundary erases the lesson

The symptom is Trial 2 performing exactly like Trial 1, with the adaptation gap near zero on held out tasks while training return looks healthy. The policy explores forever and never exploits.

The usual cause is a reset in the wrong place. Rollout code zeroes $h$ whenever $d$ fires, so the trial boundary wipes the evidence Trial 1 just collected. The mirror bug also happens: $h$ never clears at the true episode end, so a new task starts with the old summary loaded and Trial 1 acts on ghosts.

The fix is mechanical. Keep $h$ across trials inside one episode and clear only at episode ends. Feed separate flags when the framework collapses them: trial done keeps memory, episode done clears it. In batched rollouts mask per environment because environments end episodes on different steps.

Meta-overfitting to a narrow task family

The symptom is a fat Trial 2 lift on training tasks and none on a slightly wider held out range. The hidden state memorized which training task it was in rather than learning how to identify tasks. A larger network changes nothing.

The cause is a task distribution too small to force identification. Two fixed biases let the policy memorize arms. Five fixed mazes let it memorize layouts. The outer loop then rewards recognition, not adaptation.

The fix starts with measurement: report the adaptation gap on held out tasks from a wider range than training, not just training return. Then widen the distribution, hold out a slice permanently, and stop when the held out gap stalls. If the gap will not open, the family may need more trials per episode before memory has enough evidence to matter.

The Quick Version

RL squared trains one recurrent policy slowly across a task family so its hidden state adapts fast with weights frozen.
Each step feeds previous action, previous reward, and done flag beside the observation, and memory persists across trial boundaries.
Trial 1 pulls of L to 1, R to 0, L to 1 give likelihoods 0.5120 versus 0.0080 and posterior 0.9846 for the left-good task.
Trial 2 turns nearly greedy at 0.98, earning 2.3723 expected over three pulls against 1.5000 uniform.
It fails when memory clears at trial ends, when stale memory leaks across episodes, and when the family is too narrow to force identification.