Skip to content
AI360Xpert
Paper Breakdowns
Paper breakdown

Whisper

The 2022 OpenAI paper that achieved human-level robustness in speech recognition by training on a massive 680,000-hour dataset of noisy, weakly supervised web audio.

Paper: Robust Speech Recognition via Weak Supervision

Authors: Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever · 2022

Read the paper
Unlike self-supervised models, Whisper relies on sheer scale. It was trained on 680,000 hours of noisy audio paired with imperfect transcripts scraped from the internet.
Unlike self-supervised models, Whisper relies on sheer scale. It was trained on 680,000 hours of noisy audio paired with imperfect transcripts scraped from the internet.

The Problem

While self-supervised models like wav2vec 2.0 were highly data-efficient, they still struggled with "robustness." If a person had a thick accent, or there was background noise, or they were speaking far from the microphone, standard ASR systems failed. Historically, ASR models were trained on perfectly clean, studio-recorded datasets (like LibriSpeech), making them brittle when deployed in the messy real world.

The Idea

OpenAI took the opposite approach to wav2vec 2.0. Instead of clever self-supervised masking on clean audio, they opted for massive, "weakly supervised" scale. They scraped 680,000 hours of multilingual audio from the web, paired with their associated transcripts (like YouTube closed captions). These transcripts were often flawed (AI generated or user-typed with typos). However, they hypothesized that the sheer volume and diversity of the noisy data would force the model to learn incredibly robust representations that ignore background noise and accents.

How It Works

The architecture is deliberately simple to allow for maximum scaling:

  1. Architecture: A standard Transformer Encoder-Decoder (the same architecture used in early LLMs like T5).
  2. Input: The audio is converted to a log-Mel spectrogram and fed into the encoder.
  3. Multitask Output: The decoder is prompted with special tokens that tell it what task to perform. It can seamlessly switch between English transcription, translating another language into English, or predicting timestamps for subtitles.

Because the training data included coughing, music, wind, and overlapping speakers, Whisper learned to focus purely on the human speech.

Why It Mattered

Whisper effectively solved offline speech recognition. It achieved human-level accuracy across dozens of languages and accents without needing to be fine-tuned for specific environments. OpenAI open-sourced the model weights, and it instantly became the backbone for nearly all transcription software, podcast analysis, and video captioning tools worldwide.

What Came After

Whisper v2 and v3 were subsequently released with better accuracy and language support. The open-source community heavily optimized the model (e.g., Whisper.cpp, Faster-Whisper), allowing it to run in real-time on standard laptops and smartphones.