Exploration and Intrinsic Motivation

Why Does This Exist?

On dense benchmarks like CartPole the environment pays at every step, and dumb exploration works: flip random actions $\epsilon$-greedy style or jitter the motors with Gaussian noise and you will stumble into good trajectories.

Sparse tasks break that deal. Montezuma's Revenge, long-horizon manipulation, maze navigation: thousands of coordinated moves before the first nonzero reward. Random action sequences find a length-$T$ goal trajectory with probability around $|\mathcal{A}|^{-T}$, so at $T = 100$ with 4 actions you are waiting on an event at $10^{-60}$. Concretely: every reward reads $r = 0$, every TD error reads zero, policy gradients carry no signal, and the agent diffuses around the start like a lost tourist.

Intrinsic motivation adds a second reward engine inside the agent. The environment still pays its sparse task reward $r_{\text{ext}}$, but the agent also pays itself $r_{\text{int}}$ for novelty, surprise, or information gain. Early formulations go back to Schmidhuber (1991) and Oudeyer and Kaplan (2007); modern deep RL turned them into modules you bolt onto standard optimizers.

This page covers the composite reward, the three bonus families you will actually meet, and the decay property that hands control back to the task. For counting visits, read count-based exploration. For prediction-error curiosity, read the Intrinsic Curiosity Module. For the distillation trick that survives noise, read random network distillation.

The one idea you need first: epistemic versus aleatoric uncertainty. Epistemic uncertainty is ignorance you can cure by visiting (an unmapped room). Aleatoric uncertainty is randomness no visit cures (static on a screen). Good bonuses reward the first and ignore the second; bad ones pay the agent to stare at static.

Think of It Like This

How It Actually Works

The agent maximizes the composite reward, with $\beta \ge 0$ setting how much curiosity matters:

$r_{\text{total}}(s_t, a_t, s_{t+1}) = r_{\text{ext}}(s_t, a_t, s_{t+1}) + \beta \cdot r_{\text{int}}(s_t, a_t, s_{t+1})$

Big actor-critic systems often track two critics: an extrinsic head with a far horizon ($\gamma_{\text{ext}}$ near $0.999$) for task completion and an intrinsic head with a short one ($\gamma_{\text{int}}$ near $0.99$) for local novelty.

Three bonus families

Count-based novelty pays $r_{\text{int}}(s) = 1 / \sqrt{N(s)}$ for visit count $N(s)$: the first visit pays 1.0, the fourth pays 0.5, the ninth pays about 0.33. In continuous spaces exact counts are always 0 or 1, so methods like CTS density models compute pseudo-counts $\hat{N}(s)$ from a learned density instead. Details live in count-based exploration.

Prediction-error curiosity (ICM) trains a forward model $\hat{\phi}(s_{t+1})$ on features $\phi$ and pays the error: $r_{\text{int}} = (\eta / 2) \lVert \hat{\phi}(s_{t+1}) - \phi(s_{t+1}) \rVert_2^2$. Unmastered transitions surprise the model and pay well. The encoder is trained through an inverse model that predicts the action, which filters out noise the agent cannot control. Full treatment in the Intrinsic Curiosity Module.

Distillation novelty (RND) freezes a random target network $f$ and trains a predictor $\hat{f}_\theta$ to match it, paying $r_{\text{int}} = \lVert \hat{f}_\theta(s) - f(s) \rVert_2^2$. Familiar states distill to near zero error; novel ones do not. Because the target is deterministic, the error must decay with visits, which makes RND immune to the noisy-TV trap below. Full treatment in random network distillation.

Why curiosity hands control back

Intrinsic rewards are non-stationary by design: $\lim_{N(s) \to \infty} r_{\text{int}}(s) = 0$. As the map fills in, bonuses fade and $r_{\text{total}}$ converges to $r_{\text{ext}}$. The same knob that starts exploration ends it, with no schedule to tune: early roaming, late exploitation.

Worked example: crossing a six-state chain

States $s_0$ through $s_5$ in a line; only $s_5$ pays $r_{\text{ext}} = 10.0$ and ends the episode. Actions: 0 (left or stay), 1 (right). Settings: $\alpha = 0.5$, $\gamma = 0.9$, $\beta = 0.5$, count bonus $r_{\text{int}} = 1 / \sqrt{N(s')}$, all Q-values start at 0.

Stepping $s_0 \to s_1$ across visits:

• First visit, $N = 1$: $r_{\text{int}} = 1 / \sqrt{1} = 1.0$, so $r_{\text{total}} = 0.0 + 0.5 \times 1.0 = 0.5$
• Fourth visit, $N = 4$: $r_{\text{int}} = 0.5$, so $r_{\text{total}} = 0.25$
• Ninth visit, $N = 9$: $r_{\text{int}} \approx 0.3333$, so $r_{\text{total}} \approx 0.1667$

Curiosity decays monotonically: $1.0 \to 0.5 \to 0.3333$. On the very first visit the TD update is:

$\text{target} = 0.5 + 0.9 \times 0.0 = 0.5, \quad \delta = 0.5 - 0.0 = 0.5$

$Q(s_0, 1) \leftarrow 0.0 + 0.5 \times 0.5 = 0.25$

Plain Q-learning leaves $Q(s_0, 1) = 0.0$ and the agent never prefers moving right. The bonus lifts it to $0.25$, so the agent marches deeper on its next visit, and each fresh state pays the same trick forward until $s_5$ pays 10.0 for real.

Code

A pure-Python transcription of the worked example: bonus decay at visits 1, 4, 9 and the first TD update.

1import math2
3beta = 0.54alpha = 0.55gamma = 0.96
7
8def intrinsic_bonus(visits):9    """Count-based novelty: r_int = 1 / sqrt(N(s'))."""10    return 1.0 / math.sqrt(visits)11
12
13bonuses = [intrinsic_bonus(n) for n in (1, 4, 9)]14totals = [beta * b for b in bonuses]15q_updated = 0.0 + alpha * (totals[0] + gamma * 0.0 - 0.0)16
17print([round(b, 4) for b in bonuses])  # -> [1.0, 0.5, 0.3333]18print([round(t, 4) for t in totals])  # -> [0.5, 0.25, 0.1667]19print(round(q_updated, 4))  # -> 0.25

Watch Out For

The Quick Version

• Sparse rewards starve random exploration: a length-$T$ goal arrives with probability near $|\mathcal{A}|^{-T}$, so TD errors sit at zero.
• The composite reward $r_{\text{total}} = r_{\text{ext}} + \beta \cdot r_{\text{int}}$ pays the agent for novelty until the task pays for itself.
• Count, prediction-error, and distillation bonuses all decay with visits, handing control from exploration to exploitation with no schedule.
• The chain example decays $1.0 \to 0.5 \to 0.3333$, totals $0.5 \to 0.25 \to 0.1667$, and lifts $Q(s_0, 1)$ from $0.0$ to $0.25$ on one step.
• Good curiosity targets reducible ignorance, never irreducible noise: that distinction is the whole noisy-TV lesson.

What to Read Next

• Count-Based Exploration
• Curiosity-Driven Exploration (ICM)
• Random Network Distillation (RND)