Random Network Distillation (RND)

A student network tries to copy a frozen random teacher on every state it visits. Large copy errors mean the state is new, so the error itself becomes the curiosity reward.

A predictor network learns to copy a fixed random target, so the copy error starts at 2.25 on a new state and falls to 0.0045 once the state is familiar

Why Does This Exist?

Some environments pay almost nothing for thousands of steps. The classic case is Montezuma's Revenge, where an agent must climb ladders, grab a key, and open doors before the first point arrives. Random actions almost never stumble through that whole chain, so an agent trained on outside rewards alone stands still forever.

The fix is intrinsic motivation: pay the agent a synthetic bonus for visiting new states. But the two earlier options both break. Counting visits works in small gridworlds and collapses in pixel worlds, where the exact same image almost never repeats. Predicting the next frame, as the curiosity module does, breaks differently: in front of a flickering TV screen the next frame is unknowable, so the error stays huge and the agent stares at static forever.

Random Network Distillation, introduced by Burda and colleagues at OpenAI in 2018, keeps the curiosity idea and drops the future prediction. It asks only whether the present state looks familiar, by checking how well a trained student copies a frozen random teacher on it. That check is deterministic, so noise cannot fool it, and it scales to raw pixels. This page covers the RND mechanism itself. For the broader landscape read intrinsic motivation, and for the counting alternative read count-based exploration.

The one idea you need first

An intrinsic reward is a bonus the agent computes for itself, added to the reward the environment hands out. If $r^{ext}$ is the game score and $r^{int}$ is the novelty bonus, the agent trains on $r = r^{ext} + \alpha r^{int}$ , where $\alpha$ sets how much curiosity matters. RND is one concrete recipe for $r^{int}$ .

Think of It Like This

The apprentice and the professor's fixed cipher

A professor invents a strange but completely consistent cipher and never changes it. Every page of text always maps to the same symbols. An apprentice is told to reproduce the professor's ciphering for every manuscript in the library.

On a manuscript the apprentice has never seen, the first guess is far off, and that large error is exciting: here is something unread. After copying the same page fifty times, the apprentice reproduces it almost exactly, the error drops near zero, and boredom sets in. Off to the unread wings of the library.

Now put a page of pure ink splatter on the desk, the noisy TV of this story. The professor's cipher is still a fixed rule, so after a few passes the apprentice masters that page too and walks away. Static never holds attention, because the task was copying a fixed mapping, not predicting the next splatter.

Where the analogy stops: the apprentice files pages separately, while the predictor network shares weights across all states. Leave an old state unvisited for millions of frames and the shared weights drift, so stale states can regain a small bonus through forgetting.

How It Actually Works

RND runs two networks on the current observation $s$ , which is the agent's present input such as a game frame.

The target network $f(s)$ is randomly initialized and then frozen. It maps $s$ to a $k$ -dimensional vector, called an embedding, meaning a short list of numbers that stands in for the state. Its weights never change.
The predictor network $\hat{f}(s)$ starts random too but keeps training, with the sole job of outputting the same embedding the target produces.

The novelty bonus is the squared copy error:

r^{int} = \lVert \hat{f}(s) - f(s) \rVert_2^2

In plain words: subtract the two vectors element by element, square each difference, and add them up. The predictor trains on visited states to minimize this same quantity, so on a fresh state the error is large and on a well visited state it has been trained down near zero.

Why noise cannot trap it

A next-frame predictor tries to guess $s_{t+1}$ from $(s_t, a_t)$ . When the next frame contains true randomness, the best possible guess still misses by the variance of that randomness, so the error never falls and the bonus never fades. RND never predicts the future. Its target $f(s)$ is a deterministic function of the present state: same input, same output, every time. A predictor with enough capacity can learn that fixed mapping exactly, even for a frame full of static, so the bonus on noisy but familiar states still decays to zero.

Two value heads in practice

Curiosity bonuses shift as the predictor learns, which makes them non-stationary, a moving target unlike the fixed game score. Implementations therefore train two separate value estimates inside PPO: one for the outside return that resets at episode end, and one for the curiosity return that does not reset when the agent dies. Without that second detail, agents learn to die on purpose, respawn at the start, and farm novelty in the early rooms again.

Worked example

Embedding size $k = 3$ . The agent enters a new room in state $s$ .

The frozen target outputs:

f(s) = [1.20,\ -0.40,\ 0.80]

The untrained predictor outputs:

\hat{f}(s) = [0.20,\ 0.10,\ -0.20]

Subtract element by element: $[0.20 - 1.20,\ 0.10 - (-0.40),\ -0.20 - 0.80] = [-1.00,\ 0.50,\ -1.00]$ . Square and add: $1.00 + 0.25 + 1.00 = 2.2500$ . That is the first-visit bonus, and the diagram draws exactly this.

After training on this room, the predictor outputs $[1.15,\ -0.38,\ 0.76]$ . The error is $[-0.05,\ 0.02,\ -0.04]$ , and the bonus is $0.0025 + 0.0004 + 0.0016 = 0.0045$ . The drop is $(2.2500 - 0.0045) / 2.2500 \approx 99.80$ percent. The room is now boring, and the policy follows its remaining error gradient toward unexplored rooms.

Code

This snippet replays the worked example with plain arithmetic, no libraries. Check the printed numbers against the hand computation above.

1def intrinsic_reward(pred: list[float], target: list[float]) -> float:2    """Squared copy error between predictor and target embeddings."""3    return sum((p - t) ** 2 for p, t in zip(pred, target))4
5
6if __name__ == "__main__":7    target = [1.20, -0.40, 0.80]8    fresh = [0.20, 0.10, -0.20]9    trained = [1.15, -0.38, 0.76]10
11    r_fresh = intrinsic_reward(fresh, target)12    r_trained = intrinsic_reward(trained, target)13    print(f"fresh bonus: {r_fresh:.4f}")14    print(f"trained bonus: {r_trained:.4f}")15    print(f"decay: {(1.0 - r_trained / r_fresh) * 100:.2f}%")

1# -> expected output:2fresh bonus: 2.25003trained bonus: 0.00454decay: 99.80%

Watch Out For

Raw pixels saturate the target and wild bonuses drown the game score

The symptom: curiosity starts strong, then the policy ignores the real task entirely, or the bonus barely varies between rooms. The cause is scale. Feed raw pixel values in the range 0 to 255 into the networks and the nonlinearities saturate, so embeddings stop discriminating between states. And unscaled novelty rewards can exceed game rewards by orders of magnitude early in training.

The fix has two parts, both from the original paper. Normalize observations with a running mean and standard deviation, clipped to a sane range, before they reach either network. Then divide the intrinsic reward by a running standard deviation of past intrinsic returns, so curiosity keeps a steady magnitude while the predictor learns.

Forgetting brings dead rooms back to life

The symptom: late in training the agent revisits long cleared rooms as if they were new. The cause is the shared predictor weights. Training on recent rooms shifts weights that also encoded old rooms, so copy error on stale states creeps back up without any real novelty.

The fix: expect some resurgence and do not threshold the bonus at zero. Keep the non-episodic curiosity value head so the agent values genuine future novelty over refarmed rooms, and keep training the predictor on a mix of recent and older states rather than only the latest rollout.

The Quick Version

RND pays a novelty bonus equal to the squared error between a trainable predictor and a frozen random target on the current state.
Because the target is a deterministic function of the present, RND is immune to the noisy TV trap that breaks next-frame prediction.
The worked trace falls from 2.2500 on first visit to 0.0045 after training, a decay of about 99.80 percent.
Observation normalization plus intrinsic reward scaling are required, or saturation and reward imbalance wreck the run.
Shared predictor weights forget old rooms slowly, so stale states can regain a small bonus late in training.