Dreamer: World Models for RL

Learn a small simulator of the world from past play, then train the player inside that simulator where mistakes cost nothing and steps run in bulk.

The world model learns from replay while the actor critic trains on 15 step imagined rollouts scored with lambda returns

Why Does This Exist?

Model free methods learn by doing. That works until doing costs too much. Robots break, simulators crawl, and pixel environments eat millions of steps before the policy looks sane. Planning in pixels has the mirror problem: predicting full images many steps out is slow and blurry.

Dreamer (Hafner and colleagues, 2019 to 2023) splits the job. First learn a compact world model from logged experience. Then train an actor critic purely inside that model's latent space, where a step is a matrix multiply and you can dream tens of thousands of transitions per second. Start with learning latent dynamics for the model half and model-based deep RL for the family, then compare planning styles in MuZero and AlphaZero.

Three loops run together: act in the world and log $(x_t, a_t, r_t)$ , fit the world model on replay, and improve the policy on imagination. This page follows that order.

Think of It Like This

A pilot and a flight simulator built from flight logs

Training a pilot only in a real jet means every lesson risks the jet. Showing them cockpit videos is safe but passive: the video cannot answer what happens if they pull up now.

Dreamer builds the middle option. Engineers compress years of flight telemetry into a simulator, then the student flies thousands of simulated hours, stalls included, with an instructor grading each landing. Real flights still happen, and their logs keep improving the simulator.

The analogy stops at authorship. A flight simulator is hand coded from physics. Dreamer writes its own simulator from raw sensations, so a flawed model teaches confident nonsense. Horizon limits and value bootstrapping exist to contain exactly that.

How It Actually Works

The world model is a recurrent state space model. Latent state $z_t$ pairs a deterministic memory $h_t$ with a stochastic part $s_t$ . An encoder maps pixels $x_t$ to $z_t$ , dynamics predict $z_{t+1}$ from $z_t$ and action $a_t$ , and small heads predict reward $\hat{r}$ , discount $\hat{\gamma}$ , and reconstruction. Training balances reconstruction plus reward and discount losses against a KL term that keeps the prior close to the posterior without collapsing it.

DreamerV2 made $s_t$ discrete: 32 categorical variables of 32 classes each, sampled with a straight through estimator so gradients still flow. DreamerV3 added scale handling with the symlog map $\text{symlog}(x) = \text{sign}(x) \ln(|x| + 1)$ and its inverse, plus two hot value targets over binned returns. Discrete latents hold multimodal futures better, and symlog keeps one setting working across games with tiny and huge rewards.

Policy learning never renders pixels. From a real encoded state $z_0$ , the actor $\pi(a \mid z)$ rolls $H = 15$ latent steps through the frozen dynamics, collecting predicted rewards. The critic $V(z)$ scores states, and $n$ step bootstraps blend into lambda returns:

G_t^{(n)} = \sum_{k=0}^{n-1} \gamma^k \hat{r}_{t+k+1} + \gamma^n V(z_{t+n})

V^{\lambda}(z_t) = (1 - \lambda) \sum_{n} \lambda^{n-1} G_t^{(n)}

Here $\gamma$ discounts the future and $\lambda$ weights short bootstraps against long rollouts. The actor climbs $V^{\lambda}$ by analytic gradients through the differentiable model, and the critic regresses toward it.

Worked example

Take a short imagined slice with $\gamma = 0.9$ and $\lambda = 0.8$ . Rewards are $\hat{r}_1 = 1.0$ and $\hat{r}_2 = 2.0$ , critic values are $V(z_2) = 2.5$ and $V(z_3) = 1.0$ .

One step bootstrap:

G_1^{(1)} = 1.0 + 0.9 \times 2.5 = 1.0 + 2.25 = 3.25

Two step bootstrap:

G_1^{(2)} = 1.0 + 0.9 \times 2.0 + 0.81 \times 1.0 = 1.0 + 1.80 + 0.81 = 3.61

Lambda blend:

V^{\lambda}(z_1) = 0.2 \times 3.25 + 0.8 \times 3.61 = 0.650 + 2.888 = 3.538

DreamerV3 compresses that target before the critic fits it:

\text{symlog}(3.538) = \ln(4.538) \approx 1.5125

Inverting restores $3.538$ , so large and small returns share one loss scale.

Code

This snippet repeats the worked slice: two bootstraps, the lambda blend, and the symlog round trip.

1import math2
3
4def symlog(x: float) -> float:5    sign = 1.0 if x > 0 else (-1.0 if x < 0 else 0.0)6    return sign * math.log(abs(x) + 1.0)7
8
9def symexp(y: float) -> float:10    sign = 1.0 if y > 0 else (-1.0 if y < 0 else 0.0)11    return sign * (math.exp(abs(y)) - 1.0)12
13
14def main() -> None:15    gamma, lam = 0.9, 0.816    r1, r2, v2, v3 = 1.0, 2.0, 2.5, 1.017
18    g1 = r1 + gamma * v219    g2 = r1 + gamma * r2 + gamma**2 * v320    vlam = (1.0 - lam) * g1 + lam * g221    s = symlog(vlam)22
23    print(f"G1: {g1:.2f}")  # -> G1: 3.2524    print(f"G2: {g2:.2f}")  # -> G2: 3.6125    print(f"Vlambda: {vlam:.3f}")  # -> Vlambda: 3.53826    print(f"symlog: {s:.4f}")  # -> symlog: 1.512527    print(f"restored: {symexp(s):.3f}")  # -> restored: 3.53828
29
30if __name__ == "__main__":31    main()

Watch Out For

Stretching the imagination horizon

Longer dreams feel like deeper planning, so teams raise $H$ from 15 to 50 or 100. Small model errors compound with each imagined step, and the actor soon finds states where the model predicts free reward. Imagination scores soar while real play collapses.

Keep $H$ near 15 and let the critic carry the far future through $V^{\lambda}$ with $\lambda$ near 0.95. If imagined returns climb but real returns do not, shorten the horizon before touching the actor.

The Quick Version

Dreamer learns a latent world model from replay, then trains actor and critic on imagined rollouts inside it.
Latent state pairs recurrent memory with stochastic variables, discrete in V2 and V3 for sharper futures.
Policy rollouts run $H = 15$ steps and are scored with lambda returns that blend short bootstraps and long dreams.
V3 uses symlog and two hot targets so one setting spans games with very different reward scales.
In the worked slice, $3.25$ and $3.61$ blend to $3.538$ , which symlog maps to $1.5125$ .
Long horizons invite hallucination. Anchor them with the critic and keep the horizon modest.