Sequence-to-Sequence

Standard feedforward neural networks and synchronous recurrent models assume that input and output sequences have identical dimensions ($T_x = T_y$) and monotonic point-to-point alignments. However, in foundational natural language applications such as machine translation, text summarization, and dialogue generation, sequence lengths inherently diverge.

Translating the three-word English sentence "the cat sat" ($T_x = 3$) into French requires four words: "le chat est assis" ($T_y = 4$). Rigid slot-to-slot classification collapses because:

• Length Asymmetry: The target sequence contains an extra token that possesses no synchronous corresponding slot in the input stream.
• Syntactic Reordering: Word orders vary across languages (e.g. adjectives following nouns in Romance languages or verbs moving to clause boundaries in German).

To handle variable-length, non-monotonic mapping, the Sequence-to-Sequence (Seq2Seq) framework decouples input ingestion from output generation through an intermediate memory buffer.

Fold Words Into State

The Encoder component of a Seq2Seq network ingests the input sequence $(x_1, x_2, \dots, x_{T_x})$ sequentially. At each time step $t$, the recurrent cell updates its internal hidden state vector $h_t$ by combining the current input embedding with the previous state:

$h_t = \tanh\left(W_{xh} x_t + W_{hh} h_{t-1} + b_h\right)$

Starting from an initial zero state $h_0 = [0, 0]$:

1. Step 1: Reading "the" ($x_1$) updates state to $h_1 = [0.20, 0.45]$, encoding initial determiner syntax.
2. Step 2: Reading "cat" ($x_2$) combines with $h_1$ to produce $h_2 = [0.55, 0.30]$, incorporating the subject noun.
3. Step 3: Reading "sat" ($x_3$) combines with $h_2$ to reach the terminal state $h_3 = [0.80, 0.30]$.

Each step compresses fresh token information into the running hidden state. Because the recurrent weights $(W_{xh}, W_{hh})$ are shared across all positions, the encoder can process sequences of arbitrary length.

Two Numbers Hold All

When the encoder reads the final input token at step $T_x$, its terminal hidden state is emitted as the context vector $c$:

$c = h_{T_x} = [0.80, 0.30]$

This context vector serves as the sole bridge between the encoder and the decoder. In classical Seq2Seq models (Sutskever et al., Cho et al.), $c$ is a fixed-dimensional vector (e.g. $d = 512$ or $1024$ numbers).

The context vector represents an extreme informational bottleneck: it must encapsulate the semantic meaning, grammatical mood, tense, and relational dependencies of the entire source sentence. Once emitted, $c$ directly initializes the decoder's starting hidden state:

$s_0 = c$

The decoder cannot query the original tokens again; whatever was lost during the encoder's recurrent compression is gone permanently.

Write One, Reread One

The Decoder operates as an autoregressive language model conditioned on the context vector $c$. It generates the output sequence $(y_1, y_2, \dots, y_{T_y})$ one token at a time:

$P(Y \mid X) = \prod_{t=1}^{T_y} P\left(y_t \mid y_{<t}, c\right)$

The generation loop unrolls step by step:

1. Step 1: Initialized with $s_0 = c$, the decoder receives a special start-of-sequence token $\langle\text{SOS}\rangle$. It computes state $s_1$ and projects through a softmax vocabulary head to emit $y_1 = \text{"le"}$.
2. Step 2: The emitted token "le" is fed back as the input for step 2. Combined with state $s_1$, the cell updates to $s_2$ and emits $y_2 = \text{"chat"}$.
3. Step 3: "chat" feeds back to produce $s_3$, emitting $y_3 = \text{"est"}$.
4. Step 4: "est" feeds back to produce $s_4$, emitting $y_4 = \text{"assis"}$.
5. Step 5: "assis" feeds back into step 5, where the softmax head emits the special end-of-sequence token $\langle\text{EOS}\rangle$, signaling the model to halt generation.

By feeding its own previous prediction back into the input slot, the decoder maintains grammatical consistency across arbitrary output lengths.

Eyes Beat Bottlenecks

While the fixed context vector works adequately for short sentences ($< 10$ words), it deteriorates rapidly on longer sequences.

In a recurrent encoder, each successive step partially overwrites earlier activations. By the time an RNN reaches word 6 or 20, the gradient signal and informational trace of the initial tokens decay exponentially through repeated Jacobian products ($0.25^6 \approx 0.0002$). In practice, only $2$ of $6$ early concepts survive into the final vector $c$.

To eliminate this bottleneck, Bahdanau et al. introduced the Cross-Attention Mechanism:

• Instead of discarding intermediate encoder hidden states $(h_1, h_2, \dots, h_{T_x})$, the decoder retains direct access to all of them.
• At each decoding step $t$, the decoder compares its current query state $s_t$ against each encoder state $h_i$ to compute dynamic alignment weights $\alpha_{t,i}$.
• When translating the subject noun, the decoder sends a focused attention beam directly to $h_1$, retrieving its representation with $85\%$ signal fidelity.

With attention, memory capacity scales dynamically with input length, rescuing $5$ of $6$ concepts and removing the fixed-vector constraint.

Guide, Then Let Go

Training a Seq2Seq model introduces a subtle operational dilemma between training stability and deployment realism:

1. Training: Teacher Forcing

During training, the decoder is guided using teacher forcing: at step $t$, the input provided to the decoder is the true ground-truth target token $y_{t-1}^*$, regardless of what token the model actually predicted. Teacher forcing stabilizes training gradients and prevents an early erroneous prediction from derailing the entire sequence during backpropagation.

2. Inference: Free-Running (Flying Solo)

At test time in production, ground-truth answers are unavailable. The decoder must feed its own generated predictions $\hat{y}_{t-1}$ back into itself.

This discrepancy causes exposure bias: the model was trained exclusively on pristine ground-truth prefixes, making it fragile when exposed to its own mistakes. If the model makes an early error at step 1 (emitting "un" instead of "le"), it encounters a state trajectory unseen during training, leading to compounding hallucination cascades. Engineering pipelines mitigate exposure bias through scheduled sampling—gradually replacing ground-truth tokens with model predictions during late training epochs.

Where It Breaks

The fundamental theoretical limit of vanilla Seq2Seq is the Fixed-Capacity Collapse Cliff:

As input sentence length increases from $10$ to $50$ tokens:

• Attention-Augmented Seq2Seq: Translation quality (measured by BLEU score) remains steady and robust ($34.0 \to 31.5$), because the dynamic attention matrix scales with sentence length $\mathcal{O}(T_x \cdot T_y)$.
• Plain Fixed-Vector Seq2Seq: BLEU score falls off a cliff—dropping from $32.5$ at length 10 down to $19.5$ at length 30, and collapsing to $7.0$ at length 50.

Because a fixed-size vector $c \in \mathbb{R}^d$ possesses finite informational Shannon capacity, attempting to compress an entire paragraph into a single static array forces the network to overwrite initial tokens with the sentence tail. In real-world production translation and summarization, vanilla fixed-vector Seq2Seq has been entirely superseded by attention-based architectures and the Transformer.

The Quick Version

• Sequence-to-sequence translation decouples input ingestion from output generation to handle length mismatches ($T_x \ne T_y$).
• The recurrent encoder sequentially folds input tokens into a running hidden state: $h_t = \tanh(W x_t + U h_{t-1})$.
• A single fixed context vector $c = h_{T_x}$ serves as the information bottleneck bridge seeding the decoder ($s_0 = c$).
• The decoder unrolls autoregressively, feeding each generated token back in to predict the next word until $\langle\text{EOS}\rangle$.
• Fixed vectors suffer exponential recurrent decay, dropping early tokens on sequences longer than 10 words.
• Dynamic cross-attention solves the bottleneck by providing direct query-key memory access to all encoder states.
• Teacher forcing stabilizes training, but scheduled sampling is required to prevent inference exposure bias.