Text-to-Speech (TTS)
Converting text to speech is a two-step process: first, the AI acts like a composer, writing the 'sheet music' (a spectrogram) of how the words should sound. Then, a second AI acts like an instrument, turning that sheet music into actual physical sound waves.
Why Does This Exist?
Early computer voices (like Stephen Hawking's famous voice synthesizer) used Concatenative TTS. Humans recorded thousands of hours of audio, chopped them up into individual syllables ("ah", "beh", "cuh"), and the computer simply glued those audio files together. It sounded robotic because humans don't talk like that; the pitch and emotion of a syllable change entirely based on the word coming after it.
Deep learning revolutionized this by treating TTS as a generative AI problem. Instead of gluing audio files together, Neural TTS generates the sound wave from scratch. Models like Google's Tacotron (2017) finally crossed the uncanny valley, producing speech so natural that humans often couldn't tell they were talking to a machine.
Think of It Like This
The Composer and the Orchestra
If you want to create a beautiful symphony from a story, you need two experts.
- The Acoustic Model (The Composer): Reads the script. They know that a certain scene is sad, so they write down specific notes, tempos, and frequencies on a piece of sheet music. However, sheet music makes no sound.
- The Vocoder (The Orchestra): Looks at the sheet music. They know exactly how to vibrate a violin string or blow into a trumpet to translate those written notes into physical sound waves that hit your ear.
Modern TTS splits the work exactly like this.
How It Actually Works
The classical deep learning TTS pipeline (which dominated the industry until the recent rise of LLM-based TTS) is a two-stage process.
1. The Text Front-End
English is a chaotic language. "Read" rhymes with "Seed" in present tense, but rhymes with "Bed" in past tense. Before hitting the neural network, the raw text is passed through a normalizer. It converts "$100" to "one hundred dollars". It then converts the English characters into Phonemes (the International Phonetic Alphabet), which ensures the AI knows exactly how a word is pronounced regardless of its spelling.
2. The Acoustic Model (e.g., Tacotron 2)
The phoneme sequence is fed into the Acoustic Model (usually an Encoder-Decoder RNN or Transformer). Its job is to convert the sequence of phonemes into a Mel-Spectrogram. A Mel-Spectrogram is a 2D image showing how the audio frequencies should change over time. Because people speak at different speeds, the model uses an Attention Mechanism to figure out how many pixels of the spectrogram it should dedicate to the "S" sound versus the "A" sound.
3. The Vocoder (e.g., WaveNet or HiFi-GAN)
The Acoustic Model output is just an image (a matrix of numbers representing frequencies). You can't play an image through a speaker. We need to convert the Spectrogram back into a 1D physical sound wave (tens of thousands of samples per second). This is the job of the Vocoder.
- WaveNet was the original breakthrough vocoder. It used an autoregressive CNN to predict the physical sound wave one single sample at a time. It sounded flawless, but it was incredibly slow to generate.
- HiFi-GAN replaced it by using Generative Adversarial Networks (GANs). It learned to generate the entire waveform simultaneously, allowing real-time, instantaneous speech generation on consumer hardware.
Show Me the Code
This pseudocode shows the classic two-step pipeline used in almost all production TTS systems before 2023.
import torch
def generate_tts(text_string, acoustic_model, vocoder, text_normalizer): """ Standard Two-Stage TTS Pipeline """ # 1. Normalize and phonemize the text # "I have $5" -> "AY HH AE V F AY V D AA L ER Z" phonemes = text_normalizer(text_string) # 2. Generate the "Sheet Music" (Mel-Spectrogram) # The acoustic model figures out the pitch, duration, and tone. # Output shape: (Batch, Mel_Bands, Time_Frames) mel_spectrogram = acoustic_model(phonemes) # 3. Generate the physical sound wave (Vocoder) # The vocoder turns the frequency image into actual audio samples. # Output shape: (Batch, 1, Audio_Samples) e.g., 44,100 numbers per second audio_waveform = vocoder(mel_spectrogram) return audio_waveformWatch Out For
The Prosody Problem
Because the Acoustic Model generates the spectrogram autoregressively (or semi-autoregressively), it has to guess the emotion (prosody) of the sentence. If a sentence ends with a question mark, the model knows to raise its pitch at the end. But what if it's a sarcastic sentence? What if the text is "I didn't steal the money," but the intended emphasis is "I didn't steal the money"? Standard TTS models tend to average out all emotions, resulting in a very pleasant, but very flat, "news anchor" reading style.
The Quick Version
- Early TTS systems glued pre-recorded syllables together, resulting in robotic speech.
- Deep Learning TTS treats speech generation as a two-stage generative process.
- Stage 1 uses an Acoustic Model (like Tacotron) to translate text characters into a Mel-Spectrogram (an image of the audio frequencies).
- Stage 2 uses a Vocoder (like WaveNet or HiFi-GAN) to translate that Spectrogram image into a raw, 1D physical audio waveform that can be played through a speaker.
- While this architecture produces incredibly natural-sounding speech, it requires hours of clean studio recording from a single speaker to train properly.
What to Read Next
- Read Speech Synthesis to see how modern LLM-based TTS systems bypassed this two-stage pipeline entirely to achieve instant voice cloning from a 3-second clip.
- Read Automatic Speech Recognition to review how the exact reverse of this pipeline works (converting raw audio back into text).