Noisy Nets for Exploration

Why Does This Exist?

Epsilon greedy explores by rolling dice each step. With chance epsilon it picks a uniform random action, else the greedy action. That dithers. Left then right then left cancels progress, which hurts in mazes and sparse games where a distant goal needs a steady run of moves. It also treats mastered states like new ones, still blundering late in training, and its start, end, and decay need hand tuning per game.

Fortunato and coauthors introduced Noisy Nets in 2018 to move noise from actions to weights. One sampled perturbation stays fixed while the agent scores all actions in a state, so behavior stays steady across steps. The noise scales are learned, so exploration fades by itself where values turn certain. The base learner here is deep Q networks, and the natural sequel is Rainbow DQN, which ships Noisy Nets as its exploration piece. The older baseline is epsilon greedy.

This page covers the NoisyLinear split, factorised sampling that keeps cost low, and the gradient step that quiets sigma. Architecture detail for the value stream lives with dueling networks.

Think of It Like This

How It Actually Works

A standard layer maps input $x$ in $\mathbb{R}^p$ to output $y$ in $\mathbb{R}^q$ with weight $W$ and bias $b$ as $y = W x + b$. A NoisyLinear layer keeps a mean plus a scale times noise for each parameter:

Here $\mu^w$ and $\mu^b$ are learnable means, $\sigma^w$ and $\sigma^b$ are learnable scales, $\varepsilon^w$ and $\varepsilon^b$ are zero mean noise draws, and $\odot$ is element wise product. Sample eps once per step, hold the realized weights while scoring every action, then resample for the next step or batch.

Factorised sampling keeps cost low

Independent noise needs $p \times q + q$ draws. Factorised noise needs $p + q$. Draw $\varepsilon_{in}$ in $\mathbb{R}^p$ and $\varepsilon_{out}$ in $\mathbb{R}^q$ from a standard Gaussian, pass each through $f(z) = \mathrm{sgn}(z) \sqrt{|z|}$, then form weight noise by outer product and bias noise from the output side:

Worked example with real numbers

Take one neuron with $p = 2$ inputs and $q = 1$ output. Input $x = [2.0, -1.0]$. Means $\mu^w = [0.5, -0.2]$ and $\mu^b = 0.1$. Scales $\sigma^w = [0.4, 0.3]$ and $\sigma^b = 0.2$. Draw $\varepsilon_{in} = [1.0, -4.0]$ and $\varepsilon_{out} = [1.0]$.

Apply $f$. We get $f(1.0) = 1.0$, $f(-4.0) = -2.0$, and $f(1.0) = 1.0$ on the output side. So $\varepsilon^w = [1.0, -2.0]$ and $\varepsilon^b = 1.0$.

Realized weights are $w_0 = 0.5 + 0.4 \times 1.0 = 0.90$, $w_1 = -0.2 + 0.3 \times (-2.0) = -0.80$, and $b = 0.1 + 0.2 \times 1.0 = 0.30$. Forward output is:

The clean mean output is $0.5 \times 2.0 + (-0.2) \times (-1.0) + 0.1 = 1.30$. Noise shifted it by plus 1.60.

Set Bellman target $y^* = 1.50$ with step $\alpha = 0.1$. Error $\delta = 2.90 - 1.50 = 1.40$. Loss is $0.5 \times 1.40^2 = 0.98$. Gradients on means are $\delta x = [2.80, -1.40]$ with bias grad 1.40. Gradients on scales multiply by noise to $[2.80, 2.80]$ with bias grad 1.40. The update gives $\mu^w = [0.22, -0.06]$, $\sigma^w = [0.12, 0.02]$, $\mu^b = -0.04$, and $\sigma^b = 0.06$. Overshoot pushed all three sigma values down, which is self annealing in one step.

Code

This snippet repeats the forward pass and the sigma update from the worked example.

1import math2
3
4def f(z: float) -> float:5    return math.copysign(math.sqrt(abs(z)), z)6
7
8def noisy_forward(x0: float, x1: float) -> float:9    w0: float = 0.5 + 0.4 * f(1.0) * f(1.0)10    w1: float = -0.2 + 0.3 * f(1.0) * f(-4.0)11    b: float = 0.1 + 0.2 * f(1.0)12    return w0 * x0 + w1 * x1 + b13
14
15y: float = noisy_forward(2.0, -1.0)16print(f"{y:.4f}")  # -> 2.900017print(f"{0.5 * (y - 1.5) ** 2:.4f}")  # -> 0.980018print(f"{0.4 - 0.1 * 2.8:.2f}")  # -> 0.12

Watch Out For

The Quick Version

• Noisy Nets move exploration from random actions to learnable weight noise held steady across action scores.
• A NoisyLinear weight equals mean plus scale times noise, with both mean and scale learned by gradient descent.
• Factorised sampling builds $p \times q$ perturbations from $p + q$ draws through $f(z) = \mathrm{sgn}(z)\sqrt{|z|}$.
• The worked neuron outputs 2.90 noisy versus 1.30 clean, takes loss 0.98 against target 1.50, and shrinks sigma from 0.40 to 0.12.
• Overshoot lowers sigma and undershoot can sustain it, so mastered states quiet by themselves.
• Resample per step or batch, never per action score inside one state.

What to Read Next

• Deep Q-Networks
• Rainbow DQN: Six Upgrades in One Agent
• Dueling Network Architectures