Gen AI

Automatic Speech Recognition (ASR)

When you speak to Siri, your phone records a physical sound wave. Automatic Speech Recognition is the AI process that listens to that messy, continuous wave and types out the exact text you said.

Why Does This Exist?

For decades, getting a computer to understand human speech was one of the hardest problems in computer science. Humans speak with different accents, at different speeds, in noisy rooms, and we constantly slur our words together.

If you say "I'm going to," it often sounds like "Imma gunna." An English speaker knows exactly what that means based on context, but a computer just sees a squiggly line of audio. Automatic Speech Recognition (ASR) is the foundational technology behind Speech-to-Text. It bridges the gap between raw physics (acoustic sound waves) and structured human language (discrete text). ASR is what powers Siri, Alexa, YouTube closed captions, and medical dictation software.

Think of It Like This

The Court Stenographer

Imagine a court stenographer sitting in a noisy courtroom.

The lawyers are arguing, someone drops a book, and the air conditioner is humming. This is the raw audio waveform.
The stenographer listens to this chaotic mix of sounds, ignores the air conditioner (feature extraction), figures out what English words map to the specific sounds the lawyers are making (the acoustic model), and types those words onto a clean sheet of paper (text output).

ASR is simply a digital version of this highly trained stenographer.

How It Actually Works

Modern ASR systems (often called End-to-End systems) typically follow a three-step pipeline: Feature Extraction, Acoustic Modeling, and Decoding.

1. Feature Extraction (Spectrograms)

Neural networks are notoriously bad at processing raw, 1D audio waveforms directly. If you record audio at 16,000 Hertz, a 10-second audio clip is a sequence of 160,000 numbers. That sequence is far too long for a neural network to process efficiently.

Instead, we convert the audio into a Mel-Spectrogram. Using a mathematical tool called a Fourier Transform, we chop the audio into tiny 25-millisecond windows. For each window, we calculate how much energy is in the low frequencies (bass) versus high frequencies (treble). This converts the 1D audio wave into a 2D image (Time $\times$ Frequency). Now, we can just use standard Image AI (like CNNs or Vision Transformers) to "look" at the audio!

2. The Acoustic Model

The Mel-Spectrogram is passed into an Acoustic Model. Historically, this was a combination of CNNs and Recurrent Neural Networks (RNNs). Today, it is almost exclusively a Transformer. The model analyzes the "image" of the audio and tries to predict what letter (or phoneme) is being spoken at every single time slice. For example, if it sees a burst of high-frequency noise, it might predict an "S" or "Sh" sound.

3. The Alignment Problem

Here is the core difficulty of ASR: people speak at different speeds. If I say "Hello" very slowly, the "LL" sound might stretch across 20 time slices of the spectrogram. If I say it quickly, it might only take 2 time slices. How does the network know that 20 "L" predictions in a row should be collapsed into just two "L"s in the final text? Modern models solve this using either CTC (Connectionist Temporal Classification) or a Sequence-to-Sequence Attention mechanism. These algorithms figure out how to perfectly collapse and align the continuous audio with the discrete text.

Show Me the Code

This pseudocode shows the basic data flow of a modern ASR pipeline.

1import torch2
3def asr_pipeline(raw_waveform, asr_model, feature_extractor, tokenizer):4    """5    Converts a raw 1D audio waveform into text.6    """7    # 1. Convert raw audio into a 2D Mel-Spectrogram8    # Shape changes from (160000,) to (80_Mel_Bands, 1000_Time_Frames)9    spectrogram = feature_extractor(raw_waveform)10    11    # 2. Pass the "image" of the audio through the neural network12    # The model outputs a probability distribution over the vocabulary for EVERY time frame13    # Shape: (1000_Time_Frames, Vocabulary_Size)14    logits = asr_model(spectrogram)15    16    # 3. Decoding and Alignment17    # Find the most likely character at each time step18    predicted_token_ids = torch.argmax(logits, dim=-1)19    20    # Example raw output: [H, H, e, e, e, l, l, l, l, l, o, o, <blank>, <blank>, W, o...]21    22    # 4. Collapse the repetitions and decode into final English text23    final_text = tokenizer.decode_and_collapse(predicted_token_ids)24    25    return final_text

Watch Out For

The Cocktail Party Problem

ASR models are trained to transcribe a single speaker. If you feed an ASR model audio of three people talking over each other at a loud party, the model will catastrophically fail, attempting to merge the words of all three people into a single, nonsensical sentence. Solving this requires a separate AI step called "Speaker Diarization" (figuring out who is speaking when) or "Source Separation" (isolating different voices into separate audio tracks) before passing the audio to the ASR model.

The Quick Version

Automatic Speech Recognition (ASR) is the process of converting raw acoustic audio into text.
Because raw audio waves are too long and messy, they are first mathematically converted into a 2D image of frequencies called a Mel-Spectrogram.
A neural network (like a Transformer) "looks" at this spectrogram and predicts the spoken characters over time.
The most difficult part of ASR is dealing with speech speed (alignment), which requires specialized algorithms to collapse long, drawn-out sounds into single characters.
ASR forms the foundation of all voice-activated assistants, automated transcription, and real-time translation systems.