Visual explainer

Vanilla RNN

Process a sequence one step at a time by passing a hidden state forward, though it struggles to remember long-range context.

A standard feedforward network processes each input independently, so it forgets the previous words in a sentence instantly.

A fixed-size network treats a sentence as a bag of independent words, discarding the order and dropping all context.

State that steps forward

By passing a hidden state forward, the network remembers what it saw earlier in the sequence.

Process the sequence one token at a time. The cell computes a running summary — the hidden state — and carries it into the next step.

The exact same weights

Crucially, it is the exact same network cell and weights reused at every time step.

It is one cell, running in a loop. The weights and the bias are identical across every single time step.

Infinite sequences

Because the weights are reused, the network can process sequences of any length.

This weight reuse allows the network to handle an input sequence of any length without altering its architecture.

Where It Breaks

During backpropagation, gradients shrink as they pass backwards through the cells, making it impossible to learn long-range dependencies.

Backpropagation through time pushes gradients backwards. Because of repeated multiplications, the error signal decays to zero, causing vanishing gradients.

The Quick Version

Sequences require memory.
A hidden state carries context.
One cell is reused every step.
Variable lengths are handled.
Gradients vanish over time.

Related Concepts

Vanilla RNNReuse the same small network at every time step, and let a hidden state carry forward whatever happened earlier so order in the sequence actually matters.