Variational Autoencoders (VAE)
Introduced the Variational Autoencoder, a generative model that learns a continuous, structured latent space through the mathematical 'reparameterization trick'.
The Problem
Standard Autoencoders can compress an image into a small 'latent vector' and reconstruct it. However, they are useless for generating new images. Because they just memorize a mapping, their latent space is jagged and disjointed. If you pick a random point in a standard autoencoder's latent space and decode it, you just get garbage noise.
The Idea
The authors wanted to force the latent space to be continuous and structured, so that every point in the space decodes to a valid, realistic output. To do this, instead of an encoder outputting a single point, it outputs a probability distribution (a mean and a variance).
How It Works
- Probabilistic Encoder: The input (e.g., a face) is passed through an encoder, which outputs two vectors: a vector of means and a vector of standard deviations.
- The Reparameterization Trick: To get the final latent vector to pass to the decoder, the model samples from this distribution. However, you can't backpropagate gradients through a random sample. The authors introduced a brilliant trick:
z = mean + (standard_deviation * epsilon), where epsilon is a random number. Now, the randomness is just an input, and gradients can flow cleanly through the mean and deviation parameters. - KL Divergence: A regularization term is added to the loss function that mathematically forces these distributions to resemble a standard normal distribution (a bell curve). This ensures the latent space stays tightly packed and smooth.
- Decoder: The sampled vector is decoded back into an image.
Why It Mattered
VAEs provided a rigorous mathematical framework for deep generative modeling. Unlike GANs, they are stable to train and allow for precise manipulation of the latent space (e.g., finding the 'vector' for a smile, and adding it to a non-smiling face).
What Came After
While VAEs tend to produce slightly blurrier images than GANs, their stable latent spaces made them incredibly useful. Today, they are a critical component of Latent Diffusion Models (like Stable Diffusion), which use a VAE to compress images into a dense latent space before applying the diffusion process.