Advanced RL Paradigms

Why Does This Exist?

A flat policy mapping states to primitive actions works while rewards are dense, the world sits still, and the task never changes. Four realities break it:

• Late rewards. A bonus after 100,000 correct micro-moves is nearly unreachable by random thrashing, so nothing ever reinforces the early moves.
• Moving peers. Warehouse robots and traffic fleets learn simultaneously, so from any one agent view the world dynamics keep shifting underfoot.
• Empty rewards. Some rooms pay nothing at all, and a flat agent twitches in place forever waiting for a signal.
• New tasks. A maze policy restarts from zero when the walls shift, because it memorized moves instead of how to adapt.

Each of the four below reframes the problem to remove one breakage: hierarchy compresses time, team training steadies peers, curiosity manufactures signal, and meta-learning compresses adaptation. The six-frontier research map lives at advanced and PhD-level topics; this page teaches the four mechanisms.

Think of It Like This

How It Actually Works

Options compress time

An option $\omega$ is a macro-action with three parts $\langle \mathcal{I}_\omega, \pi_\omega, \beta_\omega \rangle$: the states where it can start ($\mathcal{I}$), its internal policy ($\pi$), and its stop rule ($\beta$). A top-level policy picks an option, the option runs its own moves for $k$ steps, then control returns. Choosing among a dozen options every few seconds replaces choosing among thousands of torques every millisecond, which is what makes hour-long tasks learnable. See the options framework and hierarchical RL.

Centralized training steadies peers

With $N$ agents learning at once, independent learners chase a moving target because each agent policy shift rewrites everyone else world. Centralized training with decentralized execution splits the difference: during training one critic sees the joint state and all joint actions, while at runtime each actor sees only its own observation $o_i$ and picks $a_i \sim \pi_i(\cdot \mid o_i)$. Training is stable because nothing is hidden; execution is independent because nothing needs a radio. See multi-agent RL.

Curiosity manufactures signal

With no outside reward, the agent pays itself for surprise. An encoder maps states to features $\phi(S)$, a forward model predicts the next features $\hat{\phi}(S_{t+1})$ from current features and the action, and the bonus is the scaled prediction error. Here $\eta$ sets the bonus size:

New corridors predict badly, pay well, and get explored; familiar halls predict cleanly and stop paying. See curiosity-driven exploration.

Meta-learning compresses adaptation

Instead of one fixed task, the agent trains across a distribution of tasks and learns the adaptation itself: a recurrent or context-conditioned policy reads its own recent trials and adjusts within the episode, no weight updates needed at test time. A maze policy that infers the wall layout from three bumps beats one that memorizes a single layout. See meta-reinforcement learning.

Worked example

Zero outside reward, $\eta = 0.5$, $\gamma = 0.9$, critic value $V(S_{t+1}) = 0.80$. True next features $\phi = [1.0, 2.0, 0.0]$, predicted $\hat{\phi} = [1.2, 1.6, 0.5]$. Error vector $[0.2, -0.4, 0.5]$, squared length:

Bonus and Bellman target:

An agent that earned exactly nothing now has a target of 0.8325 pulling it toward the unexplored corridor.

Code

The curiosity bonus and target from the worked example, in standard-library Python:

1def curiosity_update(2    phi_true: list, phi_pred: list, eta: float = 0.5, gamma: float = 0.9, v_next: float = 0.803) -> tuple:4    """Return (squared error, intrinsic bonus, total reward, bellman target)."""5    sq_err = sum((p - t) ** 2 for p, t in zip(phi_pred, phi_true))6    r_int = (eta / 2.0) * sq_err7    return sq_err, r_int, r_int, r_int + gamma * v_next8
9
10sq_err, r_int, r_tot, y = curiosity_update([1.0, 2.0, 0.0], [1.2, 1.6, 0.5])11print(f"{sq_err:.2f}")12# -> 0.4513print(f"{r_int:.4f}")14# -> 0.112515print(f"{r_tot:.4f}")16# -> 0.112517print(f"{y:.4f}")18# -> 0.8325

Every print matches the hand computation step for step.

Watch Out For

The Quick Version

• Options $\langle \mathcal{I}, \pi, \beta \rangle$ turn hundred-thousand-step horizons into dozens of high-level picks.
• Centralized critics see joint states in training while actors run on local views alone.
• Curiosity pays prediction error: the worked 0.45 error becomes a 0.1125 bonus and a 0.8325 target from zero outside reward.
• Meta-learning trains across task distributions so new tasks need trials, not retraining.
• Random noise hypnotizes curiosity agents, so use distillation or controllable features near stochasticity.
• Options collapse to single steps without switching costs, so watch mean option length.

What to Read Next

• Hierarchical RL
• Multi-Agent RL
• Curiosity-Driven Exploration (ICM)
• Meta-Reinforcement Learning