Skip to content
AI360Xpert
Core ML
Visual explainer

Long Short-Term Memory

LSTMs solve the short memory of vanilla RNNs by adding a separate cell state highway edited by three gates.

A vanilla RNN processes sequences step-by-step, but its gradients vanish over time, causing it to quickly forget early information.
A vanilla RNN processes sequences step-by-step, but its gradients vanish over time, causing it to quickly forget early information.

A vanilla RNN recomputes its hidden state entirely at every step. Because of repeated squashing functions, gradients decay exponentially, causing the network to forget long-range context.

The Memory Highway

An LSTM solves this by adding a separate cell state highway that carries memory forward without passing through a squashing function.
An LSTM solves this by adding a separate cell state highway that carries memory forward without passing through a squashing function.

Instead of just a hidden state, LSTMs add a second memory line called the cell state. It runs straight through the cell, allowing information to survive unchanged for many steps.

The Forget Gate

The forget gate decides what to erase from the cell state, scaling it down selectively without adding anything new.
The forget gate decides what to erase from the cell state, scaling it down selectively without adding anything new.

The first step is deciding what to throw away. The forget gate looks at the previous hidden state and the current input, outputting a number between 0 and 1 to selectively scale down the cell state.

The Input Gate

The input gate decides what new information to add, blending a gated candidate value into the cell state highway.
The input gate decides what new information to add, blending a gated candidate value into the cell state highway.

Next, the cell decides what new information to store. A candidate memory is generated, and the input gate scales it to decide exactly how much gets added to the cell state.

The Output Gate

The output gate filters the cell state to produce the hidden state, which is passed to the next step and the final prediction.
The output gate filters the cell state to produce the hidden state, which is passed to the next step and the final prediction.

Finally, the cell extracts the hidden state from the updated cell state. The output gate decides which parts of the cell state should be made visible to the rest of the network.

Where It Breaks

Because LSTMs must process tokens sequentially one by one, they are inherently slow to train and struggle to scale compared to Transformers.
Because LSTMs must process tokens sequentially one by one, they are inherently slow to train and struggle to scale compared to Transformers.

LSTMs process data strictly sequentially — step 3 cannot start until step 2 finishes. This bottleneck makes them hard to parallelize and slow to train on massive datasets.

The Quick Version

  • Vanilla RNNs quickly forget early tokens.
  • LSTMs add a persistent cell state highway.
  • Forget gates erase irrelevant old memory.
  • Input gates selectively add new context.
  • Output gates filter memory for predictions.
  • Sequential execution limits their scaling.

What to Read Next