VIME: Curiosity by Information Gain

VIME makes curiosity precise: an action is interesting when it changes what the agent believes about how the world works. Noise never changes beliefs for long, so the agent gets bored of static and chases what it can learn.

A belief update from mean 1.0 variance 0.25 to mean 1.2 variance 0.17 carries KL divergence 0.1161, which becomes the curiosity bonus

Why Does This Exist?

Sparse rewards starve agents: with no feedback for thousands of steps, random wandering never reaches the goal. Curiosity bonuses pay the agent to visit unfamiliar states, but the classic version pays for prediction error, the gap between predicted and actual next state. That works until the world contains randomness.

The failure has a name: the noisy TV problem. Put a flickering static screen in the room and prediction error stays huge forever, because static is unlearnable. The agent stares at noise, collecting curiosity pay for zero progress. The root confusion is between two uncertainties: aleatoric, the randomness baked into the world, and epistemic, the agent's own ignorance about learnable structure. Only the second deserves a bonus.

VIME, from Houthooft and colleagues, pays for information gain instead: how much a transition changes the agent's beliefs about the world's dynamics. Static cannot compress into a better model, so beliefs stop moving and the bonus dies. Learnable surprises keep paying.

This page covers the belief update, the KL bonus, and the fast approximation, with one hand-computed number. The prediction-error sibling is curiosity-driven exploration, the broader motive is exploration and intrinsic motivation, and the count-based alternative is random network distillation.

The one idea you need first

A Bayesian dynamics model treats its own weights $\theta$ as uncertain, holding a distribution over them rather than one value. The KL divergence $D_{\mathrm{KL}}$ measures how far that distribution moves after new evidence. Big move, big news; no move, nothing learned.

Think of It Like This

A physicist and a static-filled television

A physicist tests a new material near absolute zero and the resistance drops unexpectedly. One measurement squeezes a wide spread of candidate equations down to a narrow band. Her picture of the world changed, so the experiment was worth running. That squeeze is information gain.

Next door, a television shows untuned static. A surprise-chaser gasps at every frame: predicted grey, got black, huge error, forever. The physicist glances over, notes her equations about the screen settled after one second of viewing, and leaves. No belief moves, no reason to stay.

Where the analogy stops: the physicist reasons in symbols, while VIME tracks beliefs over thousands of network weights with factorized Gaussians, measuring the squeeze with a second-order approximation instead of exact inference.

How It Actually Works

The agent keeps beliefs $q(\theta; \phi_t)$ over dynamics weights: a factorized Gaussian with a mean and variance per weight, since the exact posterior is intractable. After acting and observing the next state, beliefs update to $\phi_{t+1}$ by fitting the new transition. The intrinsic reward is the size of that move, scaled by $\eta > 0$ :

$r_t^{\mathrm{int}} = \eta \, D_{\mathrm{KL}}\!\left(q(\theta; \phi_{t+1}) \,\|\, q(\theta; \phi_t)\right)$

The policy then optimizes external plus intrinsic reward together. Refitting the whole network per step would be far too slow, so each step takes one likelihood gradient and scores the KL with a second-order expansion around the current beliefs. For diagonal Gaussians that expansion has a closed form, which is what makes the bonus computable online.

Worked example

One weight, prior mean $1.0$ with variance $0.25$ (precision $4.0$ ). A transition supplies evidence at $1.6$ with precision $2.0$ .

Posterior variance: $1 / (4.0 + 2.0) = 0.1667$ . Posterior mean: $(4.0 \times 1.0 + 2.0 \times 1.6) / 6.0 = 7.2 / 6.0 = 1.2000$ .

The Gaussian KL from new to old beliefs has variance ratio $0.6667$ , mean term $0.1600$ , minus $1$ , plus log ratio $0.4055$ . Summed and halved: $0.5 \times 0.2322 = 0.1161$ . With $\eta = 1.0$ the curiosity bonus is $0.1161$ . The second-order Fisher shortcut gives $0.1078$ , close enough to steer by at a fraction of the cost.

Code

The belief update and both KL numbers in plain Python, with type hints, matching the hand computation:

1import math2
3mu_old, var_old = 1.0, 0.254obs, obs_precision, eta = 1.6, 2.0, 1.05
6post_var: float = 1.0 / (1.0 / var_old + obs_precision)7post_mu: float = post_var * (mu_old / var_old + obs_precision * obs)8kl: float = 0.5 * (9    post_var / var_old10    + (mu_old - post_mu) ** 2 / var_old11    - 1.012    + math.log(var_old / post_var)13)14fisher: float = 0.5 * (15    (post_mu - mu_old) ** 2 / var_old16    + (post_var - var_old) ** 2 / (2.0 * var_old**2)17)18
19print(f"posterior [{post_mu:.4f}, {post_var:.4f}]")  # -> posterior [1.2000, 0.1667]20print(f"exact KL {kl:.4f}")  # -> exact KL 0.116121print(f"fisher approx {fisher:.4f}")  # -> fisher approx 0.107822print(f"bonus {eta * kl:.4f}")  # -> bonus 0.1161

Watch Out For

Letting the bonus drown the real reward

The intrinsic scale $\eta$ and the raw KL magnitudes swing wildly across training: early beliefs move a lot, so bonuses explode just when the agent should be learning the task. The symptom is an agent that explores forever, revisiting frontier states while the actual goal sits solved-but-unvisited in the logs.

Fix it by normalizing: divide bonuses by a running standard deviation so curiosity and task reward stay in the same league, and anneal $\eta$ downward as competence rises.

Running the Bayesian machinery on raw pixels

A Bayesian network keeps a mean and a variance per weight, doubling parameters, and the per-step gradients are noisy on high-dimensional images. The symptom is curiosity bonuses that jitter wildly or collapse early, plus wall-clock training several times slower than the underlying policy optimizer.

Fix it by compressing first: run VIME over a compact latent embedding rather than pixels, update the full dynamics model on replayed mini-batches, and score each rollout step with the cheap Fisher form.

The Quick Version

VIME defines curiosity as information gain: the KL move in dynamics beliefs caused by a transition, not raw prediction error.
Unlearnable noise leaves beliefs stationary, so the bonus falls to zero and the noisy TV trap closes.
Beliefs are factorized Gaussians updated by variational inference, with a second-order Fisher form for online scoring.
In the worked update, beliefs move from [1.0, 0.25] to [1.2, 0.1667], giving exact KL 0.1161, Fisher 0.1078, and bonus 0.1161.
Normalize the bonus against task reward and keep the machinery off raw pixels, or curiosity will cost more than it earns.