Dyna Architecture

Learn from each real step once, then replay it many times inside a small learned model, so one trip to the goal teaches the whole path.

One real step updates values and the model, then five planning steps replay the model to carry the goal reward back two states

Why Does This Exist?

Real steps cost money and time. A robot that learns only from Q-learning needs many physical episodes for a reward at the end of a corridor to reach the start. Pure planning has the mirror flaw: it is fast but needs a perfect model up front, which you never have.

Dyna (Sutton, 1990) runs both on every step. Each real transition $(s, a, r, s^{\prime})$ does two jobs: a direct Q update from the world, and a model update that stores $(r, s^{\prime})$ for $(s, a)$ . Between real steps the agent replays $N$ simulated transitions sampled from visited pairs and applies the same Q update to them. With $N = 0$ this is plain Q-learning; with $N > 0$ each physical step fans out into many background updates. See planning and learning for the split and prioritized sweeping for smarter sampling than uniform replay.

Think of It Like This

Day shifts plus night rehearsals

A trainee pilot flies one real route by day. The flight recorder saves every turn. At night the trainee sits in a simulator loaded with that same recording and flies the tricky approach fifty more times, at zero fuel cost.

Day flying is direct RL: slow, honest, limited. Night rehearsal is planning: fast, cheap, limited by how well the simulator matches the day. Dyna is the school that schedules both, every day, with the same grading rubric for real and simulated landings.

The analogy stops at the model. A flight simulator ships with physics built in, while Dyna builds its model one observed transition at a time and must never rehearse routes it has never flown.

How It Actually Works

Keep a table $Q(s, a)$ and a model $\text{Model}(s, a) \to (r, s^{\prime})$ . Here $s$ is the state, $a$ the action, $r$ the reward, $s^{\prime}$ the next state, $\alpha$ the learning rate, and $\gamma$ the discount. Each real step runs direct RL, stores the model entry, then loops $N$ planning updates over previously visited pairs:

Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a^{\prime}} Q(s^{\prime}, a^{\prime}) - Q(s, a) \right]

Real and simulated updates share this exact rule; only the source of $(r, s^{\prime})$ differs. Search control samples $s$ from visited states and $a$ from actions tried there, never from unseen pairs.

Worlds change, so Dyna-Q+ adds a recency bonus during planning. Let $\tau(s, a)$ count steps since the pair was tried in the world and $\kappa$ be a small constant. Simulated reward becomes $r + \kappa \sqrt{\tau(s, a)}$ , which makes long untried moves look tempting in simulation and pulls the agent back to check them in the world.

Worked example

Corridor $S_0 \to S_1 \to S_2$ with goal reward $+10.0$ at $S_2$ , otherwise $0.0$ . Set $\gamma = 0.9$ , $\alpha = 0.5$ , all Q at $0.0$ .

Real step into the goal from $S_1$ :

Q(S_1) \leftarrow 0.0 + 0.5 \times (10.0 - 0.0) = 5.00

Plain Q-learning stops here, so $Q(S_0) = 0.00$ after episode one. Dyna with $N = 5$ keeps going, alternating samples:

Sample $S_0$ : target $0.0 + 0.9 \times 5.00 = 4.50$ , so $Q(S_0) \leftarrow 0.0 + 0.5 \times 4.50 = 2.25$ .
Sample $S_1$ : target $10.0$ , so $Q(S_1) \leftarrow 5.00 + 0.5 \times 5.00 = 7.50$ .
Sample $S_0$ : target $0.9 \times 7.50 = 6.75$ , so $Q(S_0) \leftarrow 2.25 + 0.5 \times 4.50 = 4.50$ .
Sample $S_1$ : target $10.0$ , so $Q(S_1) \leftarrow 7.50 + 0.5 \times 2.50 = 8.75$ .
Sample $S_0$ : target $0.9 \times 8.75 = 7.875$ , so $Q(S_0) \leftarrow 4.50 + 0.5 \times 3.375 = 6.1875$ .

After one physical episode, Dyna holds $Q(S_0) = 6.1875$ and $Q(S_1) = 8.75$ against $0.00$ and $5.00$ without planning. The goal reward crossed two states in one episode.

Code

This snippet replays the worked episode with a fixed planning order, so the numbers land exactly as derived.

1def q_update(q: float, r: float, gamma: float, best_next: float, alpha: float) -> float:2    return q + alpha * (r + gamma * best_next - q)3
4
5def main() -> None:6    gamma, alpha = 0.9, 0.57    q_s0, q_s1 = 0.0, 0.08
9    q_s1 = q_update(q_s1, 10.0, gamma, 0.0, alpha)10    print(f"direct Q(S1): {q_s1:.2f}")  # -> direct Q(S1): 5.0011
12    q_s0 = q_update(q_s0, 0.0, gamma, q_s1, alpha)13    q_s1 = q_update(q_s1, 10.0, gamma, 0.0, alpha)14    q_s0 = q_update(q_s0, 0.0, gamma, q_s1, alpha)15    q_s1 = q_update(q_s1, 10.0, gamma, 0.0, alpha)16    q_s0 = q_update(q_s0, 0.0, gamma, q_s1, alpha)17
18    print(f"planned Q(S1): {q_s1:.2f}")  # -> planned Q(S1): 8.7519    print(f"planned Q(S0): {q_s0:.4f}")  # -> planned Q(S0): 6.187520    print(f"plain Q(S0) stays: 0.0000")  # -> plain Q(S0) stays: 0.000021
22
23if __name__ == "__main__":24    main()

Watch Out For

Planning on states you never visited

Uniform sampling over the full grid feels thorough, but the model has no entry for unseen pairs and returns a default. Planning then teaches Q from fiction, often zeros that wipe real values.

Sample only visited states and tried actions, keep $N$ modest early (5 to 20 works), and reach for the Dyna-Q+ bonus when the world may have changed. Fiction in, fiction out.

The Quick Version

Dyna stores each real transition in a model and replays $N$ simulated transitions per real step.
Real and simulated updates use the same Q rule; $N = 0$ is plain Q-learning.
Sampling stays on visited pairs, since unseen pairs have no model entry worth trusting.
Dyna-Q+ adds $r + \kappa \sqrt{\tau(s, a)}$ in simulation so stale moves get rechecked in the world.
In the worked corridor, one episode with $N = 5$ reaches $Q(S_0) = 6.1875$ while plain learning sits at $0.00$ .