Gen AI

Encoder-Decoder ASR

Encoder-decoder models listen to the entire audio clip first, then use cross-attention to translate the sounds into text one word at a time, looking back at the audio as needed.

The encoder builds a dense representation of the whole audio clip; the decoder then generates text token-by-token, using cross-attention to look back at the most relevant audio frames.

Why Does This Exist?

Early deep learning approaches to speech recognition, like CTC loss (ctc-loss), processed audio strictly left-to-right. They were fast, but they were myopic. If a speaker mumbled a word, a CTC model had to guess what it was using only the audio from that exact fraction of a second.

Humans don't listen like that. If you hear a mumbled word, you often figure out what it was based on the words that come after it. You use future context to decode past sounds.

Encoder-decoder architectures (often called sequence-to-sequence models) bring this ability to machines. By reading the entire audio clip before making a single guess, they revolutionized ASR accuracy. The most famous early example was Listen, Attend and Spell (LAS), and today, models like OpenAI's Whisper are built on this exact foundation.

Think of It Like This

Imagine a professional translator working on a recorded speech.

They don't try to translate the speech word-by-word as the recording plays. Instead, they listen to the whole recording to grasp the full context (the encoder). Then, they write down the translation one sentence at a time (the decoder).

As they write, they might rewind and re-listen to specific parts of the tape to ensure they got the phrasing right (this is cross-attention). They are using their understanding of the whole speech to inform every individual word they write.

How It Actually Works

An encoder-decoder ASR model has two distinct halves, connected by an attention mechanism.

1. The Encoder (Listen)

The encoder takes in a sequence of acoustic frames (usually mel spectrograms) and processes them into a sequence of high-dimensional vectors.

Because the encoder is typically a bidirectional RNN or a Transformer, every vector in its output contains information about the entire audio clip. The vector for frame 50 knows what happened at frame 10 and frame 90. This solves the myopia problem.

2. The Decoder (Spell)

The decoder is an autoregressive language model. Its job is to output text, one token at a time. To decide what the next word should be, it looks at two things:

The text it has already generated.
The acoustic representations produced by the encoder.

3. Cross-Attention (Attend)

The magic happens in the cross-attention layer linking the two halves.

When the decoder is trying to generate the third word, it doesn't just blindly pull from the encoder. It uses cross-attention to ask: "Given that I just wrote 'we are', which audio frames should I focus on to figure out the next word?" The attention mechanism calculates a weight for every single audio frame, effectively placing a spotlight on the segment of the audio that contains the next word.

Watch Out For

The latency wall. Encoder-decoder models are strictly offline. Because the encoder needs the entire sequence to build its representations, and the decoder needs the encoder's output to start translating, the model cannot emit a single word until the speaker stops talking.

If you try to use an encoder-decoder model for live closed-captioning, the text won't appear until the end of the sentence. For streaming use cases, you must use an RNN Transducer (rnn-transducer) instead.

The Quick Version

The Problem: Processing audio strictly left-to-right prevents models from using future context to decipher unclear sounds.
The Fix: An encoder reads the entire audio clip to build a context-rich representation. A decoder then generates the transcript token-by-token.
The Engine: Cross-attention allows the decoder to dynamically focus on specific parts of the audio as it writes.
The Cost: It cannot stream. You have to wait for the audio to finish before transcription begins.