wav2vec 2.0
The 2020 Facebook AI paper that successfully applied self-supervised learning to audio, allowing speech recognition models to be trained with vastly less human-transcribed data.
Paper: wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Authors: Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael Auli · 2020
Read the paper
The Problem
Training high-quality Automatic Speech Recognition (ASR) systems historically required thousands of hours of audio meticulously transcribed by humans. This meant ASR was only good for a few high-resource languages (like English or Mandarin). The NLP world had solved this data bottleneck using self-supervised learning (like BERT), where the model learns by masking words and predicting them. However, you can't easily "mask" a continuous audio wave because audio isn't naturally broken into discrete "words" or "tokens."
The Idea
Facebook AI solved this by creating a model that learns discrete speech units during training. wav2vec 2.0 takes raw audio, passes it through a CNN to extract features, and then masks (hides) some of those features. The model must predict what belongs in the masked gaps. To make this possible, they use a "quantizer" that forces the continuous audio into a finite set of discrete speech units. The model learns to identify these hidden units, effectively learning the phonetic structure of human speech without ever seeing a written transcript.
How It Works
The architecture has three main components:
- Feature Encoder: A CNN that converts the raw audio waveform into continuous feature vectors (latent representations).
- Quantization Module: A module that maps these continuous features into discrete speech units (acting like a codebook of phonemes).
- Transformer Context Network: It takes the continuous features (with some portions masked out) and tries to predict the correct discrete speech unit for the masked portion using contrastive loss (distinguishing the true unit from distractor units).
After pre-training on thousands of hours of unlabeled audio, the model can be fine-tuned on a tiny amount of transcribed audio (as little as 10 minutes) to achieve state-of-the-art ASR.
Why It Mattered
wav2vec 2.0 revolutionized speech processing. It proved that self-supervised learning works for audio, drastically lowering the barrier to entry for low-resource languages. It allowed companies to build highly accurate speech recognition systems for hundreds of languages using unlabelled radio or YouTube audio.
What Came After
This self-supervised paradigm became the standard for audio models. It was followed by HuBERT, and later formed the audio backbone for massive multimodal models (like SeamlessM4T) that can translate between dozens of languages in real-time.