Score-Based SDEs

The Problem

By 2020, there were two distinct, highly successful approaches to generating data by reversing noise: Denoising Diffusion Probabilistic Models (DDPMs), which used a discrete Markov chain of noise steps, and Score Matching with Langevin Dynamics (SMLD), which used multiple discrete noise scales. Both worked, but they felt like two sides of the same coin. The math was discrete, messy, and tied to arbitrary "step sizes." Researchers needed a unified theoretical framework.

The Idea

The authors (who pioneered the SMLD approach) showed that as you make the discrete noise steps infinitely small, both DDPM and SMLD converge into a single continuous-time process governed by a Stochastic Differential Equation (SDE). This meant that destroying an image with noise is just a forward SDE, and generating an image is just solving the reverse-time SDE. The neural network's only job is to estimate the "score function" (the gradient of the data distribution) required to run the reverse SDE.

How It Works

The framework models data distribution as a continuous flow over time $t \in [0, 1]$.

1. Forward SDE: An equation describing how data flows into pure noise over continuous time. This equation requires no training.
2. Reverse SDE: A theorem by Anderson (1982) states that for every forward SDE, there is a corresponding reverse-time SDE. To calculate it, you only need one thing: the "score function" $\nabla_x \log p_t(x)$—which simply points in the direction of higher data probability.
3. Score Estimation: A neural network (like a U-Net) is trained to estimate this score function at all time steps.
4. Generation: Once trained, you sample pure noise and use any off-the-shelf numerical SDE solver (like Euler-Maruyama) or ODE solver to solve the reverse equation back to $t=0$, yielding a generated image.

Why It Mattered

This paper provided the rigorous mathematical foundation for the entire diffusion field. By moving from discrete steps to continuous differential equations, it unlocked decades of existing mathematics. Researchers no longer had to invent custom sampling algorithms; they could just plug their trained network into established, highly optimized differential equation solvers.

What Came After

The continuous-time SDE/ODE perspective became the standard way researchers analyze and improve diffusion models. It led directly to Probability Flow ODEs, which enabled exact likelihood calculations, and laid the groundwork for Consistency Models and Flow Matching, which eventually allowed image generation in just 1 or 2 steps.