Skip to content
AI360Xpert
Core ML
Visual explainer

Sequence-to-Sequence

Sequence-to-sequence models map an input sequence to an output sequence of a different length by compressing the entire input into a single fixed-size context vector.

Input and output sequences rarely have the same length, making simple one-to-one mapping impossible.
Input and output sequences rarely have the same length, making simple one-to-one mapping impossible.

Traditional neural networks map one input directly to one output. But tasks like machine translation or text summarization involve sequences of entirely different lengths. A three-word English sentence might translate into a four-word French sentence, completely breaking any simple one-to-one alignment. The model needs a way to decouple the input length from the output length.

The Encoder

The encoder reads the entire input sequence step-by-step, updating its internal state with each new token.
The encoder reads the entire input sequence step-by-step, updating its internal state with each new token.

To handle mismatched lengths, the architecture splits the job into two independent halves. The first half is the encoder, which reads the input sequence one token at a time. It does not generate any output during this phase; its only job is to aggressively build up a dense internal representation of what the input sentence actually means.

The Context Vector

The final encoder state becomes the context vector—a fixed-size bottleneck containing the entire input's meaning.
The final encoder state becomes the context vector—a fixed-size bottleneck containing the entire input's meaning.

When the encoder finishes reading the last token, its final hidden state becomes the context vector. This single, fixed-size mathematical array is the only bridge between the two halves of the network. It must capture the full semantic meaning, grammar, and nuance of the entire input sequence, acting as a complete summary of everything read so far.

The Decoder

The decoder unpacks the context vector to generate the output sequence one token at a time.
The decoder unpacks the context vector to generate the output sequence one token at a time.

The second half of the network, the decoder, takes this context vector as its initial state and begins unpacking it. It generates the translated output sequence strictly step-by-step. By feeding its own previously generated tokens back into itself at each step, it continues producing text until it confidently outputs a special stop token signifying the translation is complete.

Where It Breaks

The fixed-size context vector cannot hold infinite information, meaning long sequences are inevitably truncated or forgotten.
The fixed-size context vector cannot hold infinite information, meaning long sequences are inevitably truncated or forgotten.

Because the context vector is mathematically constrained to a fixed size, it acts as a severe information bottleneck. If the input is a fifty-word paragraph, the encoder is forced to compress it into exactly the same amount of memory space as a three-word sentence. This causes early words and complex long-range dependencies to be overwritten and forgotten entirely.

The Quick Version

  • Translation requires a model to decouple mismatched sequence lengths.
  • The encoder reads the entire input sequence to build context.
  • It compresses all meaning into one fixed-size context vector.
  • The decoder unfolds this single vector into the target output.
  • The bottleneck causes severe forgetting on very long input sequences.

What to Read Next