RNN vs LSTM Memory

What you are looking at

The upper panel is a recurrent cell unrolled along time: one circle per timestep, all of them the same cell reusing the same weights. Under each one are up to three bars — the hidden state $h$ on the left, the cell state $c$ in the middle, and the forget gate $f$ on the right, as the stages introduce them. The Input Tokens row beneath the controls is the sequence being fed in, and every value in it is editable.

The lower panel, which arrives at the third stage, is the point of the lab. It plots how much the final state still depends on each earlier step. The axis is $\log_{10}$, so exponential decay is a straight line and a twelve-decade collapse still fits on screen. The dashed horizontal line is one part in a thousand — below it, the earliest input has no influence an optimiser could act on.

Where the number in the lower panel comes from

Backpropagation through time is the chain rule applied along the sequence. The gradient arriving at step $k$ has been multiplied by one Jacobian per step between $k$ and the end, and for a single-unit cell that Jacobian is a single number.

For the RNN:

$\frac{\partial h_t}{\partial h_{t-1}} = \bigl(1 - h_t^2\bigr) \cdot W_h$

Two factors, and they fail in opposite directions. $\tanh'(z) = 1 - h^2$ is at most 1 and heads to 0 as the unit saturates, so it can only ever shrink the signal. $W_h$ is unbounded, so it is the one that could grow it.

Which suggests an obvious fix, and the obvious fix does not work.

Why a bigger recurrent weight does not help

Move Recurrent Weight up from its default of 0.8 and watch Earliest Influence over a ten-step sequence:

It peaks at $W_h = 1$ and falls away on both sides. The reason is in the formula: a larger weight drives $h$ toward $\pm 1$, and at $h = 0.99$ the factor $1 - h^2$ is 0.02. You have bought a bigger $W_h$ by destroying $\tanh'$. The two halves of the product fight, and 0.283 is the best this architecture manages over ten steps at any setting.

A bucket chain with leaky buckets

Passing water down a line of people, each one spills a fraction. Ten people in, most of the water is gone, and there is no arrangement of spilling that gets it all the way.

You could try throwing harder — that is $W_h$. But throw hard enough and the buckets overflow, and an overflowing bucket does not pass on more, it passes on whatever it can hold and loses the rest. That ceiling is $\tanh$, and it is why throwing harder stops helping.

What actually works is not throwing at all. Put the water on a conveyor belt and let each person decide whether to take some off or add some. Nobody has to pass it on, so nobody spills it. That is the cell state.

The gate

Switch to the LSTM at the fourth stage and both curves stay on screen, so you are reading a comparison rather than remembering one.

The LSTM keeps a second channel, the cell state, and updates it by addition:

$c_t = f_t \cdot c_{t-1} + i_t \cdot g_t$

Differentiate that with respect to $c_{t-1}$ and you get the whole reason the architecture exists:

$\frac{\partial c_t}{\partial c_{t-1}} = f_t$

The forget gate, and nothing else. No activation derivative, so nothing capped at 1 by a squashing function and nothing that shrinks because the unit happens to be saturated. And $f$ is learned — the network can decide to hold this path open, which is not a decision an RNN gets to make.

This lab holds the input and output gates at $\sigma(10) \approx 0.99995$ with zero weights so the scene isolates the forget gate; both values are printed in the metrics rather than left silent.

An LSTM does not preserve memory by existing

The forget gate bias defaults to 1, which is the standard initialisation and looks like it ought to be enough. It is not. $\sigma(1) = 0.731$, so the per-step factor is 0.731 and over nine steps that is $0.731^9 = 0.0596$ — measured in this lab as slightly worse than the RNN at its own default of 9.6e-2.

Push Forget Gate Bias to 3 and the gate sits at 0.953, giving 0.646 over the same ten steps. That is the LSTM winning, and it wins because the gate is nearly open, not because it is an LSTM.

The positive bias initialisation is a head start, not a solution: it puts the gate on the open side of 0.5 so that gradient can flow at all in the first few epochs, which is what lets training discover where the gate should actually be. Read the reference card's memory-horizon row and then compare bias 1 against bias 3 — the architecture is the same in both, and one of them remembers.

When it fails, and what it fails into

Drag Forget Gate Bias to −3. The gate closes to 0.047, the forget-gate bars shrink to almost nothing, and the memory horizon collapses to 3 of 10 steps — the earliest token's influence on the final state is 1.2e-12, twelve orders of magnitude down.

It is worth being precise about what has happened, because the intuitive description is wrong. With $f \approx 0$ the update becomes $c_t = i_t \cdot g_t$: the cell state depends on the current input only, with no term carrying the past at all. That is not an LSTM behaving like an RNN — an RNN still routes the past through $W_h$ and manages 9.6e-2 over those same ten steps. A shut forget gate is worse than having no gate, because it actively erases what it is holding.