Planning and Learning in RL
Real steps teach you and train your mental model, then the model replays fake steps so values travel back in one episode instead of many.
Why Does This Exist?
Trial and error in the real world is slow and sometimes unsafe. A robot cannot crash a thousand times to learn that walls hurt, and a car cannot run red lights to price them. Pure model free learning needs exactly that kind of contact, one reward at a time.
Planning offers the opposite trade. If you own a model of next states and rewards, you can rehearse without touching anything. The problem is that the model starts wrong, and rehearsing a wrong model teaches confident nonsense.
The Sutton view that unites them is simple: learning and planning run the same Bellman backup, and only the source of the transition differs. Real transitions come from the world. Simulated transitions come from your model. Dyna uses each real step twice, once to improve values directly and once to improve the model, then runs N extra simulated backups before the next real move.
This page covers that loop, the two model types, and a corridor example where planning moves goal news back to the start in the same episode. For the pure model free side, see temporal difference learning. For the pure planning side, see dynamic programming. For the canonical instance, see Dyna architecture.
You need Q learning updates before this page. A Q value estimates future return for a state action pair, and the update nudges it toward reward plus discounted next value with step size alpha.
Think of It Like This
One drive, many replays
You drive a new route once and film it. That night you replay the film five times, pausing at each turn to ask what you should have done. The next morning you drive better, though you only drove the road once.
The film is the model. Replays are planning backups. Real driving still matters, because a wrong map or a stale film teaches bad turns.
Where the analogy stops: your brain knows film from road. A Dyna agent applies the same Q update to both, so model bias flows straight into values. Short replays and fresh films are the fix, and the pitfalls say how.
How It Actually Works
Write Q(s, a) for action value, alpha for step size, gamma for discount, R for reward, S prime for next state. The one step Q backup is:
Learning feeds this update with real tuples from the environment. Planning feeds the same update with fake tuples from an internal model M.
Sample models and distribution models
A distribution model stores full probabilities P(s prime given s, a) and expected rewards. It supports sweeping backups over every outcome, which is exact and heavy. Classic dynamic programming lives here.
A sample model is a simulator. Give it s and a, it draws one (R, S prime) from its learned dynamics. It supports cheap sample backups that look exactly like real experience. Dyna lives here because sampling is fast, parallel, and works in large spaces where full distributions do not fit.
The Dyna loop
Each real step does three jobs in order:
- Act in the world with your current policy and observe the real transition.
- Direct RL: apply the Q backup to the real tuple.
- Model learning: store or fit M(s, a) to the observed (R, S prime).
- Planning: repeat N times by sampling a seen pair, querying M, and applying the same Q backup to the simulated tuple.
Worked example
Corridor with states S0, S1, S2 where S2 pays 10.0. Start all Q at 0.0, alpha 0.5, gamma 0.9.
Direct RL alone first walks S0 to S1 for reward 0, so Q(S0) stays 0.0 because 0 plus 0.9 times 0 minus 0 is 0. Then it walks S1 to S2 for reward 10.0, so Q(S1) moves to 0.0 plus 0.5 times (10.0 minus 0.0), which is 5.0. Episode ends with Q(S0) at 0.0. The start learned nothing.
Dyna with N equal 2 keeps the same two real updates, then plans immediately. First simulated backup replays S1 to S2: 5.0 plus 0.5 times (10.0 minus 5.0), which is 7.5. Second simulated backup replays S0 to S1 with the fresh 7.5: 0.0 plus 0.5 times (0.0 plus 0.9 times 7.5), which is 3.375. Same physical walk, but the start now holds 3.375 instead of 0.0. The diagram draws this chain with the two planning arrows carrying 7.5 and 3.375 back.
Code
This snippet replays the corridor math with no extra packages.
def q_update(q: float, r: float, next_q: float, alpha: float = 0.5, gamma: float = 0.9) -> float: """One tabular Q backup.""" return q + alpha * (r + gamma * next_q - q)
q0 = 0.0q1 = 0.0q1 = q_update(q1, 10.0, 0.0)print(f"{q1:.4f}")# -> 5.0000q1 = q_update(q1, 10.0, 0.0)print(f"{q1:.4f}")# -> 7.5000q0 = q_update(q0, 0.0, q1)print(f"{q0:.4f}")# -> 3.3750Watch Out For
Hallucinated shortcuts from a stale model
The symptom is a policy that shines in simulation and fails outside. Values climb on fake replays, the agent chases a shortcut the world never offered, and real returns fall when you deploy.
The cause is compounding bias. A small model error on step one becomes a large value error after five simulated steps, and N large replays amplify it. Early in training the model is thinnest, which is exactly when planning helps most.
Fix it by keeping simulated horizons short, from 1 to 5 steps, by rechecking disputed pairs in the world, and by planning only where the model has data. If ensemble models disagree on a transition, skip it until the world settles the question.
The Quick Version
- Learning and planning run the same Bellman backup and differ only in where the transition comes from.
- A distribution model lists every outcome, while a sample model draws one fake outcome like a simulator.
- Dyna turns each real step into direct RL plus model learning plus N simulated backups.
- The corridor walk gives Q(S1) 5.0 from real experience, then planning lifts it to 7.5 and pulls Q(S0) to 3.375 in the same episode.
- Planning saves real steps but spreads model bias, so keep horizons short and recheck thin states in the world.