Curiosity-Driven Exploration (ICM)

Reward the agent for surprise. When its model of the world predicts badly, pay it to go back and learn that corner until the surprise runs out.

An inverse model learns features that predict the action while a forward model earns 0.0250 of curiosity reward from the miss between 0.60 and 0.80

Why Does This Exist?

Goals that pay once per thousand steps defeat random action. Epsilon-greedy twitching almost never chains the whole sequence by luck, so training flatlines far from the goal. Exploration and intrinsic motivation answers with homemade rewards, and this page is the flagship version.

The naive homemade reward is pixel surprise: predict the next frame and get paid for the miss. That agent walks up to the nearest flicker and stares. Leaves, ripples, and static are forever unpredictable, so the reward never runs out and the agent never leaves. Count-based exploration has the same weakness on raw pixels, and random network distillation is the sibling fix worth comparing.

The Intrinsic Curiosity Module fixes the target. Predict in a small feature space trained to keep only what your actions steer, and the flicker stops paying because your actions never steered it.

Think of It Like This

A detective who ignores the curtains

A detective enters a strange house. Curtains flutter at random and a faulty set fizzes in the corner. A rookie logs every flutter as a clue and never leaves the room.

A good detective asks one question: did I cause that? Doors that swing on a handle turn matter. Curtains that flutter either way do not. She forms a guess before each push, pockets a thrill when the guess is wrong, and moves on once the mechanism is learned and the thrill dies.

Where it stops: she arrives with a lifetime of priors about doors. The module starts blank and must learn, from gradients alone, which pixels answer to actions.

How It Actually Works

States $s_t$ and $s_{t+1}$ pass through an encoder $\phi$ into compact features. Two heads read those features.

The inverse head guesses the action from the pair: $\hat{a}_t = g(\phi(s_t), \phi(s_{t+1}))$ . Its loss is mean squared error for continuous moves, cross-entropy for discrete ones. To guess the action well, $\phi$ must keep features the action moves and drop the rest. Static carries zero information about your move, so gradients scrub it out. That scrubbing is the whole trick.

The forward head guesses the next features: $\hat{\phi}(s_{t+1}) = f(\phi(s_t), a_t)$ . Its miss is the curiosity pay:

r_t^i = \frac{\eta}{2} \lVert \hat{\phi}(s_{t+1}) - \phi(s_{t+1}) \rVert_2^2

Here $\eta$ scales curiosity and the norm measures surprise. Policy earns $r_t = r_t^e + r_t^i$ , outside plus inside. Training weights forward loss against inverse loss with $\beta$ near $0.20$ , leaning on the inverse side that builds the clean space.

Worked example

Two feature dims, one action, $\eta = 1.0$ , $\beta = 0.20$ . True next features $[0.80, 0.40]$ , predicted $[0.60, 0.30]$ , true action $1.00$ , predicted $0.90$ .

Error vector: $[0.60 - 0.80, 0.30 - 0.40] = [-0.20, -0.10]$ . Squared length: $0.04 + 0.01 = 0.0500$ . Reward: $0.5 \times 0.0500 = 0.0250$ .

Inverse loss: $0.5 \times (0.90 - 1.00)^2 = 0.5 \times 0.01 = 0.0050$ . Forward loss: $0.5 \times 0.0500 = 0.0250$ . Combined: $0.20 \times 0.0250 + 0.80 \times 0.0050 = 0.0050 + 0.0040 = 0.0090$ . One number, $0.0090$ , trains the module while $0.0250$ trains the policy.

Code

Curiosity pay plus both losses, matching the hand numbers:

1err: list[float] = [0.60 - 0.80, 0.30 - 0.40]2sq: float = err[0] ** 2 + err[1] ** 23reward: float = 0.5 * sq4inv: float = 0.5 * (0.90 - 1.00) ** 25fwd: float = 0.5 * sq6total: float = 0.20 * fwd + 0.80 * inv7print(f"{sq:.4f} {reward:.4f}")  # -> 0.0500 0.02508print(f"{inv:.4f} {fwd:.4f} {total:.4f}")  # -> 0.0050 0.0250 0.0090

Both lines match the worked section. Revisit the same corner and the miss shrinks, so the pay decays on its own.

Watch Out For

Curiosity that never clocks out

Symptom: the agent keeps wandering novel corners after it already knows the goal path, and outside scores plateau. A fixed $\eta$ keeps paying for every unmemorized wall long after discovery should end.

Fix: anneal $\eta$ toward zero across training and normalize the inside reward by its running spread. Shift the budget from discovery early to exploitation late, or curiosity becomes vagrancy.

The Quick Version

The inverse head predicts the action from feature pairs, which forces features to keep only what actions steer.
The forward head predicts next features and its squared miss pays the policy, here $0.0250$ from a $0.0500$ miss.
One training step totals $0.0090$ at $\beta = 0.20$ , leaning on the inverse side.
Flicker stops paying because it never helped predict the action.
Anneal curiosity strength toward zero or the agent wanders past a solved goal.