Deep Learning Diffusion Models: Foundations and Classic Models
Author: ResearchWriter_基础与经典 Generated: 2026-07-02
Chapter 1: Mathematical Foundations and Core Concepts of Diffusion Models
1.1 Introduction and Historical Context
Diffusion models are one of the most important technical paradigms in generative AI today, and their theoretical roots span statistical physics, stochastic processes, and variational inference. In 2015, Sohl-Dickstein et al.1 first drew inspiration from nonequilibrium statistical mechanics and proposed a generative framework that progressively destroys data structure through a forward diffusion process and then restores it through a reverse diffusion process. Then Song & Ermon (2019)2 revisited the problem from the perspective of score matching and proposed the Noise Conditional Score Network (NCSN); Ho et al. (2020)3 developed the Denoising Diffusion Probabilistic Model (DDPM), which brought diffusion models into practical use through variational inference and a simplified training objective. In 2021, Song et al.4 proposed a unified framework based on stochastic differential equations (SDE), showing that NCSN and DDPM are discretizations of two classes of continuous-time diffusion processes—VE-SDE and VP-SDE—and that both can be described in the same language of “score + SDE.” In 2022, the EDM work by Karras et al.5 systematically elucidated the design space of diffusion models, unifying the choices of preconditioning, loss weighting, and samplers. This chapter starts from the mathematical foundations and systematically builds the core conceptual framework of diffusion models.
graph LR
A[2015 Sohl-Dickstein
diffusion probabilistic models] --> B[2019 Song & Ermon
NCSN/score matching]
A --> C[2020 Ho et al.
DDPM]
B --> D[2021 Song et al.
Score SDE unified framework]
C --> D
D --> E[2022 Karras et al.
EDM design space]
E --> F[Modern diffusion models
Stable Diffusion, etc.]
1.2 The Forward Diffusion Process
The core idea of a diffusion model is to define a forward process—also called the diffusion process—that adds Gaussian noise to a data sample step by step through a series of Markov chain steps, until the data distribution has completely degenerated into a standard Gaussian. Given a data sample $\mathbf{x}_0 \sim q(\mathbf{x}_0)$, the forward process is defined as follows:
$$ q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathcal{N}\left(\mathbf{x}_t \mid \sqrt{1-\beta_t} \mathbf{x}_{t-1}, \beta_t \mathbf{I}\right) $$| Symbol | Meaning |
|---|---|
| $\mathbf{x}_0$ | Original data sample, of dimension $d$ |
| $\mathbf{x}_t$ | Noised sample at step $t$, $t \in \{1, \dots, T\}$ |
| $\beta_t$ | Variance schedule parameter at step $t$, $\beta_t \in (0, 1)$ |
| $T$ | Total number of diffusion steps, typically $T = 1000$ |
| $\mathbf{I}$ | Identity matrix with the same dimensionality as the data |
| $\mathcal{N}(\mu, \Sigma)$ | Gaussian distribution with mean $\mu$ and covariance $\Sigma$ |
Because each step is a conditional Gaussian, the joint distribution of the forward process is:
$$ q(\mathbf{x}_{1:T} \mid \mathbf{x}_0) = \prod_{t=1}^{T} q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) $$| Symbol | Meaning |
|---|---|
| $\mathbf{x}_{1:T}$ | All intermediate variables from $t=1$ to $T$, $\mathbf{x}_1, \dots, \mathbf{x}_T$ |
Reparameterization trick: Sampling step by step along the Markov chain is inefficient. Thanks to the additivity of Gaussians, you can sample $\mathbf{x}_t$ at any time $t$ directly from $\mathbf{x}_0$:
$$ q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}\left(\mathbf{x}_t \mid \sqrt{\bar{\alpha}_t} \mathbf{x}_0, (1 - \bar{\alpha}_t) \mathbf{I}\right) $$where $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$. Equivalently:
$$ \mathbf{x}_t = \sqrt{\bar{\alpha}_t} \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t} \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) $$| Symbol | Meaning |
|---|---|
| $\alpha_t$ | $1 - \beta_t$, the fraction of signal retained at step $t$ |
| $\bar{\alpha}_t$ | $\prod_{s=1}^{t} \alpha_s$, the cumulative signal-retention fraction |
| $\boldsymbol{\epsilon}$ | A noise vector sampled from a standard Gaussian |
When $T$ is large enough that $\bar{\alpha}_T \approx 0$, we have $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$—it has degenerated into standard Gaussian noise. In other words, the forward process smoothly transforms any arbitrarily complex data distribution $q(\mathbf{x}_0)$ into a simple prior $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{0}, \mathbf{I})$.
1.3 The Reverse Denoising Process
If we could reverse this forward process, we could start from pure noise $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and progressively denoise it into a real data sample. In principle, when $\beta_t$ is small enough, the true reverse conditional $q(\mathbf{x}_{t-1} \mid \mathbf{x}_t)$ is well approximated by a Gaussian1. But Bayes’ rule, $q(\mathbf{x}_{t-1} \mid \mathbf{x}_t) = q(\mathbf{x}_t \mid \mathbf{x}_{t-1})\, q(\mathbf{x}_{t-1}) / q(\mathbf{x}_t)$, shows that it depends on the marginals $q(\mathbf{x}_{t-1})$ and $q(\mathbf{x}_t)$ induced by the whole data distribution (equivalently, you’d need to know the score at every time step), so it can’t be computed directly. Diffusion models therefore train a neural network to parameterize the reverse process:
$$ p_{\theta}(\mathbf{x}_{t-1} \mid \mathbf{x}_t) = \mathcal{N}\left(\mathbf{x}_{t-1} \mid \boldsymbol{\mu}_{\theta}(\mathbf{x}_t, t), \Sigma_{\theta}(\mathbf{x}_t, t)\right) $$| Symbol | Meaning |
|---|---|
| $p_{\theta}$ | The reverse-process distribution defined by parameters $\theta$ |
| $\boldsymbol{\mu}_{\theta}(\mathbf{x}_t, t)$ | Mean function parameterized by the neural network |
| $\Sigma_{\theta}(\mathbf{x}_t, t)$ | Covariance matrix (fixed or learned) |
| $\theta$ | Parameters of the neural network |
The joint distribution of the reverse process starts from the prior $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{0}, \mathbf{I})$:
$$ p_{\theta}(\mathbf{x}_{0:T}) = p(\mathbf{x}_T) \prod_{t=1}^{T} p_{\theta}(\mathbf{x}_{t-1} \mid \mathbf{x}_t) $$To generate a sample, you start from standard Gaussian noise $\mathbf{x}_T$ and repeatedly apply the parameterized reverse transition kernel until you reach $\mathbf{x}_0$. This reverse process can be viewed as a series of denoising autoencoders that share parameters and are conditioned on the time step $t$, usually implemented with a U-Net or Transformer architecture.
Intuition: The forward process is like slowly stirring a drop of ink into water, messing things up just a little at each step; the reverse process learns how to “pull back” a little at each step. Precisely because the change at each step is small can a single reverse transition be approximated by a simple Gaussian.
graph LR
subgraph Forward process
X0["x0
data"] --> |add noise| X1["x1
lightly noised"]
X1 --> |...| XT["xT
pure noise"]
end
subgraph Reverse process
XT --> |denoise| XT1["xT-1"]
XT1 --> |...| X0_out["x0
generated sample"]
end
style X0 fill:#e1f5e1
style X0_out fill:#e1f5e1
style XT fill:#f5e1e1
1.4 Variational Inference and the Evidence Lower Bound (ELBO)
The training objective of a diffusion model is to maximize the data log-likelihood $\log p_{\theta}(\mathbf{x}_0)$. Computing this likelihood directly requires integrating out all the intermediate variables $\mathbf{x}_{1:T}$, which is infeasible in high-dimensional spaces. So we use variational inference and maximize the log-likelihood indirectly by optimizing the evidence lower bound (ELBO).
Here we treat the forward process $q(\mathbf{x}_{1:T} \mid \mathbf{x}_0)$ as the variational distribution over the latent variables $\mathbf{x}_{1:T}$ (similar to the encoder in a VAE, except that it’s fixed and has no learnable parameters), and $p_{\theta}(\mathbf{x}_{0:T})$ as the generative model. Multiply and divide $q(\mathbf{x}_{1:T} \mid \mathbf{x}_0)$ inside the integral of $\log p_{\theta}(\mathbf{x}_0) = \log \int p_{\theta}(\mathbf{x}_{0:T}) \, \mathrm{d}\mathbf{x}_{1:T}$, then apply Jensen’s inequality ($\log$ is concave) to get:
$$ \log p_{\theta}(\mathbf{x}_0) \geq \mathbb{E}_{q(\mathbf{x}_{1:T} \mid \mathbf{x}_0)} \left[ \log \frac{p_{\theta}(\mathbf{x}_{0:T})}{q(\mathbf{x}_{1:T} \mid \mathbf{x}_0)} \right] = -L_{\text{VLB}} $$| Symbol | Meaning |
|---|---|
| $L_{\text{VLB}}$ | The variational lower bound (negative ELBO); the training objective is to minimize it |
| $\mathbb{E}_{q}[\cdot]$ | Expectation with respect to distribution $q$ |
To break it into terms we can compute step by step, the key move is to use the Markov property and Bayes’ rule to rewrite each forward transition in a “reverse” form conditioned on $\mathbf{x}_0$:
$$ q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = q(\mathbf{x}_t \mid \mathbf{x}_{t-1}, \mathbf{x}_0) = \frac{q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0) \, q(\mathbf{x}_t \mid \mathbf{x}_0)}{q(\mathbf{x}_{t-1} \mid \mathbf{x}_0)} $$Once you substitute this in, neighboring $q(\mathbf{x}_t \mid \mathbf{x}_0)$ terms cancel like a telescope, and the lower bound decomposes exactly into a sum of KL divergences plus one reconstruction term3:
$$ L_{\text{VLB}} = \mathbb{E}_{q} \left[ \underbrace{D_{\text{KL}}(q(\mathbf{x}_T \mid \mathbf{x}_0) \| p(\mathbf{x}_T))}_{L_T} + \sum_{t=2}^{T} \underbrace{D_{\text{KL}}(q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0) \| p_{\theta}(\mathbf{x}_{t-1} \mid \mathbf{x}_t))}_{L_{t-1}} - \underbrace{\log p_{\theta}(\mathbf{x}_0 \mid \mathbf{x}_1)}_{L_0} \right] $$| Symbol | Meaning |
|---|---|
| $D_{\text{KL}}(q \| p)$ | KL divergence, measuring the difference between distributions $q$ and $p$ |
| $q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0)$ | The reverse conditional given $\mathbf{x}_0$ (analytically computable, tractable) |
| $L_T$ | Prior-matching term, with no learnable parameters; because both $q(\mathbf{x}_T \mid \mathbf{x}_0)$ and $p(\mathbf{x}_T)$ are close to a standard Gaussian, this term is near 0 and can be ignored during training |
| $L_{t-1}$ | The per-step denoising-matching term, the core training objective |
| $L_0$ | Data-reconstruction term (for discrete data, the decoder log-likelihood) |
Since $q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0)$ is the posterior of two Gaussians, it has the closed form:
$$ q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0) = \mathcal{N}\left(\mathbf{x}_{t-1} \mid \tilde{\boldsymbol{\mu}}_t(\mathbf{x}_t, \mathbf{x}_0), \tilde{\beta}_t \mathbf{I}\right) $$$$ \tilde{\boldsymbol{\mu}}_t = \frac{\sqrt{\bar{\alpha}_{t-1}} \beta_t}{1 - \bar{\alpha}_t} \mathbf{x}_0 + \frac{\sqrt{\alpha_t}(1 - \bar{\alpha}_{t-1})}{1 - \bar{\alpha}_t} \mathbf{x}_t $$$$ \tilde{\beta}_t = \frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t} \beta_t $$| Symbol | Meaning |
|---|---|
| $\tilde{\boldsymbol{\mu}}_t$ | True posterior mean given $\mathbf{x}_0$ and $\mathbf{x}_t$ |
| $\tilde{\beta}_t$ | True posterior variance given $\mathbf{x}_0$ |
Note that although the true $q(\mathbf{x}_{t-1} \mid \mathbf{x}_t)$ is intractable, the posterior conditioned additionally on $\mathbf{x}_0$ is a closed-form Gaussian—which is why, during training, you can supervise the reverse process against a “known answer.” When the reverse-process variance $\Sigma_{\theta}$ is fixed to $\tilde{\beta}_t \mathbf{I}$ (or $\beta_t \mathbf{I}$), the KL divergence between two Gaussians with identical covariance reduces to the squared difference of their means (plus a constant independent of $\theta$), which leads to DDPM’s simplified training objective (see Chapter 2).
1.5 Score Matching Theory
1.5.1 Explicit Score Matching (ESM)
The score is defined as the gradient of the log probability density with respect to the data:
$$ \mathbf{s}(\mathbf{x}) = \nabla_{\mathbf{x}} \log p(\mathbf{x}) $$| Symbol | Meaning |
|---|---|
| $\mathbf{s}(\mathbf{x})$ | The score function, with the same dimensionality as $\mathbf{x}$ |
| $\nabla_{\mathbf{x}}$ | The gradient operator with respect to $\mathbf{x}$ |
For an unnormalized model $p_{\theta}(\mathbf{x}) = \frac{1}{Z_{\theta}} \exp(-E_{\theta}(\mathbf{x}))$ parameterized by an energy function $E_{\theta}(\mathbf{x})$, the score is $\nabla_{\mathbf{x}} \log p_{\theta}(\mathbf{x}) = -\nabla_{\mathbf{x}} E_{\theta}(\mathbf{x})$, which doesn’t depend on the intractable normalizing constant $Z_{\theta}$.
Explicit score matching (ESM)6 directly minimizes the Fisher divergence between the model score and the data score:
$$ L_{\text{ESM}}(\theta) = \frac{1}{2} \mathbb{E}_{p_{\text{data}}(\mathbf{x})} \left[ \left\| \nabla_{\mathbf{x}} \log p_{\text{data}}(\mathbf{x}) - \nabla_{\mathbf{x}} \log p_{\theta}(\mathbf{x}) \right\|_2^2 \right] $$| Symbol | Meaning |
|---|---|
| $p_{\text{data}}$ | The true data distribution |
| $p_{\theta}$ | The model distribution (with parameters $\theta$) |
| $\|\cdot\|_2$ | The L2 norm |
However, the score of the true data distribution, $\nabla_{\mathbf{x}} \log p_{\text{data}}(\mathbf{x})$, is unknown, which makes ESM impractical.
1.5.2 Hyvärinen’s Implicit Score Matching (ISM)
Hyvärinen’s (2005)6 breakthrough was to eliminate the data score from the objective using integration by parts. Using the identity:
$$ \mathbb{E}_{p_{\text{data}}}\left[ \nabla_{\mathbf{x}} \log p_{\text{data}}(\mathbf{x})^{\top} \mathbf{s}_{\theta}(\mathbf{x}) \right] = -\mathbb{E}_{p_{\text{data}}}\left[ \nabla_{\mathbf{x}} \cdot \mathbf{s}_{\theta}(\mathbf{x}) \right] $$(Derivation sketch: the left side $= \int \nabla_{\mathbf{x}} p_{\text{data}}(\mathbf{x})^{\top} \mathbf{s}_{\theta}(\mathbf{x}) \, \mathrm{d}\mathbf{x}$; integrate by parts, assuming $p_{\text{data}}(\mathbf{x}) \mathbf{s}_{\theta}(\mathbf{x})$ goes to 0 at infinity.) Expand the square in the Fisher divergence, replace the cross term with the identity above, and note that the squared data-score term is independent of $\theta$. You can then rewrite the objective as:
$$ L_{\text{ISM}}(\theta) = \mathbb{E}_{p_{\text{data}}(\mathbf{x})} \left[ \text{tr}(\nabla_{\mathbf{x}} \mathbf{s}_{\theta}(\mathbf{x})) + \frac{1}{2} \|\mathbf{s}_{\theta}(\mathbf{x})\|_2^2 \right] + C $$| Symbol | Meaning |
|---|---|
| $\text{tr}(\cdot)$ | The matrix trace |
| $\nabla_{\mathbf{x}} \mathbf{s}_{\theta}(\mathbf{x})$ | The Jacobian of the model score (the Hessian of the model log-density when $\mathbf{s}_{\theta} = \nabla_{\mathbf{x}} \log p_{\theta}$) |
| $C$ | A constant independent of $\theta$, ignorable during optimization |
ISM’s fatal flaw is that it requires computing the trace of the Jacobian $\text{tr}(\nabla_{\mathbf{x}} \mathbf{s}_{\theta}(\mathbf{x}))$. For high-dimensional data such as images ($d \sim 10^5$), that takes $d$ backpropagation passes and is prohibitively expensive.
1.5.3 Denoising Score Matching (DSM)
Vincent (2011)7 proposed denoising score matching (DSM), which sidesteps the Hessian computation by adding Gaussian noise to the data. The idea is to learn the score of the noised data distribution $q_{\sigma}(\tilde{\mathbf{x}}) = \int p_{\text{data}}(\mathbf{x}) \mathcal{N}(\tilde{\mathbf{x}} \mid \mathbf{x}, \sigma^2 \mathbf{I}) \mathrm{d}\mathbf{x}$, with the explicit objective:
$$ L_{\text{DSM}}(\theta) = \frac{1}{2} \mathbb{E}_{p_{\text{data}}(\mathbf{x})} \mathbb{E}_{\tilde{\mathbf{x}} \sim \mathcal{N}(\mathbf{x}, \sigma^2 \mathbf{I})} \left[ \left\| \mathbf{s}_{\theta}(\tilde{\mathbf{x}}) - \nabla_{\tilde{\mathbf{x}}} \log q_{\sigma}(\tilde{\mathbf{x}} \mid \mathbf{x}) \right\|_2^2 \right] $$Since $\nabla_{\tilde{\mathbf{x}}} \log q_{\sigma}(\tilde{\mathbf{x}} \mid \mathbf{x}) = -\frac{\tilde{\mathbf{x}} - \mathbf{x}}{\sigma^2}$, the objective simplifies to:
$$ L_{\text{DSM}}(\theta) = \frac{1}{2} \mathbb{E}_{\mathbf{x}, \boldsymbol{\epsilon}} \left[ \left\| \mathbf{s}_{\theta}(\mathbf{x} + \sigma \boldsymbol{\epsilon}) + \frac{\boldsymbol{\epsilon}}{\sigma} \right\|_2^2 \right] $$| Symbol | Meaning |
|---|---|
| $\sigma$ | The noise level (standard deviation) |
| $\tilde{\mathbf{x}}$ | A noised data sample |
| $q_{\sigma}$ | The noised marginal distribution |
| $\boldsymbol{\epsilon}$ | Standard Gaussian noise, $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ |
DSM turns score learning into a regression problem: the network predicts the noise direction in a noised sample, with no Hessian to compute. This is the training foundation of modern diffusion models.
Why does regressing the “conditional score” learn the “marginal score”? Vincent proved that $L_{\text{DSM}}$ differs from the ESM objective on the noised distribution $q_{\sigma}$ only by a constant independent of $\theta$, so the optimum satisfies $\mathbf{s}_{\theta^*}(\tilde{\mathbf{x}}) = \nabla_{\tilde{\mathbf{x}}} \log q_{\sigma}(\tilde{\mathbf{x}})$. Intuitively, the same $\tilde{\mathbf{x}}$ can come from many different clean samples $\mathbf{x}$, and squared-error regression automatically averages those “known answers”—and that average is exactly the marginal score. Note that DSM learns the score of the noised distribution $q_{\sigma}$, which approaches the score of $p_{\text{data}}$ only when $\sigma$ is small; but when $\sigma$ is small, the training signal covers only regions near the data—which is the motivation for using multiple noise scales at once (see Section 2.3).
1.5.4 Sliced Score Matching (SSM)
Song et al. (2019)8 proposed sliced score matching (SSM), which estimates the trace using random projections. The idea is to project the high-dimensional score onto random directions:
$$ L_{\text{SSM}}(\theta) = \mathbb{E}_{p_{\text{data}}} \mathbb{E}_{\mathbf{v} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})} \left[ \mathbf{v}^{\top} \nabla_{\mathbf{x}} \mathbf{s}_{\theta}(\mathbf{x}) \mathbf{v} + \frac{1}{2} \left( \mathbf{v}^{\top} \mathbf{s}_{\theta}(\mathbf{x}) \right)^2 \right] $$| Symbol | Meaning |
|---|---|
| $\mathbf{v}$ | A random projection vector sampled from a standard Gaussian |
| $\mathbf{v}^{\top} \nabla_{\mathbf{x}} \mathbf{s}_{\theta} \mathbf{v}$ | The Jacobian–vector product (JVP/HVP) dotted with $\mathbf{v}$; with automatic differentiation this takes roughly one extra backpropagation pass |
SSM uses the idea of the Hutchinson trace estimator ($\mathbb{E}_{\mathbf{v}}[\mathbf{v}^{\top} \mathbf{A} \mathbf{v}] = \text{tr}(\mathbf{A})$ when $\mathbb{E}[\mathbf{v}\mathbf{v}^{\top}] = \mathbf{I}$), replacing the exact trace computation that takes $d$ backpropagation passes with an unbiased random estimate that takes about one pass per projection vector. For image-scale diffusion models, though, DSM remains the most common practical choice because it’s more stable and easier to implement.
1.6 Tweedie’s Formula and the Denoising–Score Equivalence
Tweedie’s formula9 establishes a deep connection between the MMSE (minimum mean squared error) denoiser and the score function. For the Gaussian noise model $q_{\sigma}(\tilde{\mathbf{x}}) = \int p_{\text{data}}(\mathbf{x}) \mathcal{N}(\tilde{\mathbf{x}} \mid \mathbf{x}, \sigma^2 \mathbf{I}) \mathrm{d}\mathbf{x}$, we have:
$$ \nabla_{\tilde{\mathbf{x}}} \log q_{\sigma}(\tilde{\mathbf{x}}) = \frac{\mathbb{E}[\mathbf{x} \mid \tilde{\mathbf{x}}] - \tilde{\mathbf{x}}}{\sigma^2} $$| Symbol | Meaning |
|---|---|
| $\mathbb{E}[\mathbf{x} \mid \tilde{\mathbf{x}}]$ | The posterior expectation of the clean sample $\mathbf{x}$ given the noised sample $\tilde{\mathbf{x}}$ (the optimal denoiser output) |
This means training a denoiser $D_{\theta}(\tilde{\mathbf{x}}) \approx \mathbb{E}[\mathbf{x} \mid \tilde{\mathbf{x}}]$ is equivalent to learning the score function. You can recover the score from the denoiser:
$$ \mathbf{s}_{\theta}(\tilde{\mathbf{x}}) = \frac{D_{\theta}(\tilde{\mathbf{x}}) - \tilde{\mathbf{x}}}{\sigma^2} $$This equivalence is the key bridge that unifies diffusion-model theory: DDPM’s $\boldsymbol{\epsilon}$-prediction (predicting noise) and EDM’s $D_{\theta}$-prediction (predicting denoised data) are both parameterizations of score estimation (for the exact conversion, see Section 1.9).
Intuition: The score $\nabla_{\tilde{\mathbf{x}}} \log q_{\sigma}(\tilde{\mathbf{x}})$ points “toward higher probability”; Tweedie’s formula tells us that this direction is the one from the current noised sample $\tilde{\mathbf{x}}$ toward the “most likely clean sample” $\mathbb{E}[\mathbf{x} \mid \tilde{\mathbf{x}}]$, divided by the noise variance.
1.7 The SDE/ODE Unified Framework
1.7.1 The Forward SDE
Song et al. (2021)4 extended diffusion processes from discrete to continuous time, describing them with a stochastic differential equation (SDE). The general form of the forward SDE is:
$$ \mathrm{d}\mathbf{x} = \mathbf{f}(\mathbf{x}, t) \mathrm{d}t + g(t) \mathrm{d}\mathbf{w} $$| Symbol | Meaning |
|---|---|
| $\mathbf{f}(\mathbf{x}, t)$ | The drift coefficient, which sets the deterministic trend |
| $g(t)$ | The diffusion coefficient, which sets the intensity of the random noise |
| $\mathbf{w}$ | The standard Wiener process (Brownian motion) |
| $t$ | A continuous time variable, usually $t \in [0, T]$ |
The marginal distribution $p_t(\mathbf{x})$ of this SDE evolves smoothly from the data distribution $p_0$ at $t=0$ to a prior (such as a standard Gaussian) at $t=T$. Different choices of $\mathbf{f}$ and $g$ give different diffusion models:
- VP-SDE (variance preserving): The continuous limit of DDPM (the discrete $\beta_i$ and continuous $\beta(t)$ satisfy $\beta_i \approx \beta(i/T)/T$; let $T \to \infty$), with $\mathbf{f}(\mathbf{x}, t) = -\frac{1}{2} \beta(t) \mathbf{x}$ and $g(t) = \sqrt{\beta(t)}$; if the data variance is 1, the total variance of $\mathbf{x}_t$ stays at 1 throughout.
- VE-SDE (variance exploding): The continuous limit of NCSN, with $\mathbf{f}(\mathbf{x}, t) = \mathbf{0}$ and $g(t) = \sqrt{\frac{\mathrm{d} \sigma^2(t)}{\mathrm{d}t}}$—the signal is unchanged while the noise variance explodes.
| Symbol | Meaning |
|---|---|
| $\beta(t)$ | The continuous-time variance schedule function |
| $\sigma(t)$ | The continuous-time noise-level function |
| $p_t(\mathbf{x})$ | The marginal distribution at time $t$ |
When the drift $\mathbf{f}$ is linear in $\mathbf{x}$ (as both VP and VE are), the conditional distribution given $\mathbf{x}_0$—the perturbation kernel—has a unified closed form:
$$ p_{0t}(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}\left(\mathbf{x}_t \mid s(t) \mathbf{x}_0, \sigma^2(t) \mathbf{I}\right) $$where $s(t)$ and $\sigma(t)$ are schedule functions. Different choices recover all the classic noise schedules:
| Model | $s(t)$ | $\sigma(t)$ | Property |
|---|---|---|---|
| DDPM/VP-SDE | $e^{-\frac{1}{2}\int_0^t \beta(s)\mathrm{d}s}$ | $\sqrt{1 - s(t)^2}$ | Variance preserving |
| NCSN/VE-SDE | $1$ | $\sigma_{\min}\left(\frac{\sigma_{\max}}{\sigma_{\min}}\right)^{t}$, $t \in [0, 1]$ | Variance exploding |
| EDM | $1$ | $t$ (after reparameterization) | Unified design space |
| Rectified Flow / Flow Matching (OT path)10 | $1-t$ | $t$ | Linear interpolation (note: DDIM uses DDPM’s VP schedule and does not belong in this row) |
1.7.2 The Reverse SDE
Anderson’s (1982)11 time-reversal theorem states that, under mild regularity conditions, every forward SDE has a corresponding reverse SDE of the form:
$$ \mathrm{d}\mathbf{x} = \left[ \mathbf{f}(\mathbf{x}, t) - g(t)^2 \nabla_{\mathbf{x}} \log p_t(\mathbf{x}) \right] \mathrm{d}t + g(t) \mathrm{d}\bar{\mathbf{w}} $$| Symbol | Meaning |
|---|---|
| $\nabla_{\mathbf{x}} \log p_t(\mathbf{x})$ | The score of the marginal distribution at time $t$ (the score we need to learn) |
| $\mathrm{d}\bar{\mathbf{w}}$ | The reverse-time Wiener process |
Here time runs backward from $T$ to $0$, and $\mathrm{d}t$ is a negative infinitesimal time step. Compared with the forward SDE, the reverse SDE has just one extra term, $-g(t)^2 \nabla_{\mathbf{x}} \log p_t(\mathbf{x})$: it pushes samples “back” toward high-density regions along the score, counteracting the spreading caused by forward diffusion.
Key insight: The reverse SDE depends only on the coefficients of the forward SDE and the time-dependent score function. So training a time-conditioned network $\mathbf{s}_{\theta}(\mathbf{x}, t) \approx \nabla_{\mathbf{x}} \log p_t(\mathbf{x})$ is enough to enable sampling.
1.7.3 The Probability Flow ODE
Song et al. (2021) further showed that there exists a deterministic process with exactly the same marginal distributions as the forward/reverse SDE—the probability flow ODE:
$$ \mathrm{d}\mathbf{x} = \left[ \mathbf{f}(\mathbf{x}, t) - \frac{1}{2} g(t)^2 \nabla_{\mathbf{x}} \log p_t(\mathbf{x}) \right] \mathrm{d}t $$| Symbol | Meaning |
|---|---|
| Right-hand side | No noise term; deterministic evolution |
Why the same marginals? The marginal density $p_t$ of the forward SDE satisfies the Fokker–Planck equation. When $g$ depends only on $t$, the identity $\Delta_{\mathbf{x}} p_t = \nabla_{\mathbf{x}} \cdot \left( p_t \nabla_{\mathbf{x}} \log p_t \right)$ lets us rewrite the diffusion term in “transport” form as well:
$$ \frac{\partial p_t}{\partial t} = -\nabla_{\mathbf{x}} \cdot \left( \mathbf{f} \, p_t \right) + \frac{1}{2} g(t)^2 \Delta_{\mathbf{x}} p_t = -\nabla_{\mathbf{x}} \cdot \left[ \left( \mathbf{f} - \frac{1}{2} g(t)^2 \nabla_{\mathbf{x}} \log p_t \right) p_t \right] $$The right side is exactly the continuity equation satisfied by a deterministic flow with velocity field $\mathbf{f} - \frac{1}{2} g(t)^2 \nabla_{\mathbf{x}} \log p_t$, so an ODE evolving along this velocity field has exactly the same $p_t$ as the original SDE.
The probability flow ODE and the reverse SDE share the same marginals $p_t$, but the ODE removes the stochasticity. That means:
- Starting from the same initial noise $\mathbf{x}_T$, the ODE trajectory maps deterministically to a unique $\mathbf{x}_0$ (and vice versa, which lets you “encode” an image back into noise);
- The ODE allows exact likelihood computation: using the instantaneous change-of-variables formula from continuous normalizing flows, $\frac{\mathrm{d} \log p_t(\mathbf{x}_t)}{\mathrm{d}t} = -\nabla_{\mathbf{x}} \cdot \tilde{\mathbf{f}}(\mathbf{x}_t, t)$ (where $\tilde{\mathbf{f}}$ is the ODE drift), integrating along the trajectory gives $\log p_0(\mathbf{x}_0)$;
- DDIM’s deterministic sampling ($\eta = 0$) can be seen as a first-order discretization of the probability flow ODE (for higher-order ODE solvers, see Chapter 5).
On the stochastic side, Song et al. also proposed Predictor-Corrector (PC) sampling: at each step, first make a “prediction” with a numerical solver for the reverse SDE, then “correct” it with several score-based Langevin MCMC steps to reduce discretization error.
graph TD
A["Forward SDE
dx = f dt + g dw"] --> B["Reverse SDE
dx = (f - g²∇log p)dt + g dw̄"]
A --> C["Probability Flow ODE
dx = (f - ½g²∇log p)dt"]
B --> D["Stochastic sampling
DDPM/PC sampling"]
C --> E["Deterministic sampling
DDIM/Heun/Flow"]
style A fill:#f5e1e1
style B fill:#e1e1f5
style C fill:#e1f5e1
1.8 A Unified View of Noise Schedules and Their Equivalence
All modern diffusion models can be unified under the following perturbation kernel framework:
$$ \mathbf{x}_t = \alpha_t \mathbf{x}_0 + \sigma_t \boldsymbol{\epsilon} $$where $\alpha_t$ and $\sigma_t$ control the magnitudes of the signal and the noise, respectively.
Notation note: This section follows the notation of the continuous-time literature (e.g., VDM, EDM), where $\alpha_t$ denotes the signal coefficient, corresponding to $\sqrt{\bar{\alpha}_t}$ in DDPM notation and not the same quantity as the $\alpha_t = 1 - \beta_t$ in Section 1.2; $\sigma_t$ corresponds to $\sqrt{1 - \bar{\alpha}_t}$. It’s also equivalent to $s(t)$ and $\sigma(t)$ in Section 1.7.1.
The signal-to-noise ratio (SNR), $\text{SNR}(t) = \alpha_t^2 / \sigma_t^2$, gives a unified way to understand the equivalence of different noise schedules5:
| Schedule type | Equivalence | Source |
|---|---|---|
| VP (DDPM) | $\alpha_t^2 + \sigma_t^2 = 1$ | Ho et al. 2020 |
| VE (NCSN) | $\alpha_t = 1$, $\sigma_t$ increasing | Song & Ermon 2019 |
| EDM | $\alpha_t = 1$, $\sigma_t = t$ (reparameterized) | Karras et al. 2022 |
| Cosine | $\alpha_t \approx \cos\left(\frac{\pi t}{2}\right)$, $\sigma_t \approx \sin\left(\frac{\pi t}{2}\right)$ (ignoring the small offset $s$; belongs to VP) | Nichol & Dhariwal 202112 |
| Flow/OT | $\alpha_t = 1-t$, $\sigma_t = t$ | Lipman et al. 202310; Liu et al. 2023 (Rectified Flow)13 |
Key result: Kingma et al. (2021), in VDM14, proved that in the continuous-time limit the ELBO can be written in a form that depends only on the SNR:
$$ \mathcal{L}_{\infty}(\mathbf{x}_0) = -\frac{1}{2} \mathbb{E}_{\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})} \int_{0}^{1} \text{SNR}'(t) \left\| \mathbf{x}_0 - \hat{\mathbf{x}}_{\theta}(\mathbf{x}_t; t) \right\|_2^2 \mathrm{d}t $$| Symbol | Meaning |
|---|---|
| $\text{SNR}'(t)$ | The derivative of SNR with respect to time (SNR decreases monotonically with $t$, so this is negative and the overall loss is positive) |
| $\hat{\mathbf{x}}_{\theta}(\mathbf{x}_t; t)$ | The network’s prediction of the clean data |
After the change of variables $\lambda = \log \text{SNR}$, you can see that this integral depends only on the two endpoints, $\text{SNR}(0)$ and $\text{SNR}(1)$, not on how the transition from high to low SNR happens in between. In other words, the optimal solution is essentially the same for different schedules; the differences show up mainly in how different noise levels are weighted (or sampled) during training and in the resulting gradient variance. That’s why modern practice (EDM, for example) tends to design the training distribution and loss weights directly in terms of the noise level $\sigma$ (or $\log \text{SNR}$) rather than the time $t$, yielding simpler math and better numerical stability.
graph LR
subgraph "Unified framework"
X0["x0"] --> |"α(t)"| PLUS["+"]
NOISE["ε ~ N(0,I)"] --> |"σ(t)"| PLUS
PLUS --> XT["xt"]
end
style X0 fill:#e1f5e1
style XT fill:#f5e1e1
1.9 Four Parameterizations of the Network Prediction Target: Noise, Clean Data, v, and Score
Under the unified perturbation kernel $\mathbf{x}_t = \alpha_t \mathbf{x}_0 + \sigma_t \boldsymbol{\epsilon}$, given $\mathbf{x}_t$ and $t$, knowing either the “noise” or the “clean data” lets you compute the other. So the network can predict any of the following quantities, which are all interconvertible:
| Parameterization | What the network learns | Representative work |
|---|---|---|
| $\boldsymbol{\epsilon}$-prediction | The added noise $\boldsymbol{\epsilon}$ | DDPM3 |
| $\mathbf{x}_0$-prediction | Clean data $\mathbf{x}_0$ | EDM’s denoiser $D_{\theta}$5 |
| $\mathbf{v}$-prediction | $\mathbf{v} = \alpha_t \boldsymbol{\epsilon} - \sigma_t \mathbf{x}_0$ | Salimans & Ho 202215, the 768-v model in SD 2.x |
| Score prediction | $\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t)$; for a single sample the regression target is $-\boldsymbol{\epsilon}/\sigma_t$ | NCSN2, Score SDE4 |
Conversion formulas ($\hat{\cdot}$ denotes the network’s prediction):
$$ \hat{\mathbf{x}}_0 = \frac{\mathbf{x}_t - \sigma_t \hat{\boldsymbol{\epsilon}}}{\alpha_t}, \qquad \mathbf{s}_{\theta}(\mathbf{x}_t, t) = -\frac{\hat{\boldsymbol{\epsilon}}}{\sigma_t} = \frac{\alpha_t \hat{\mathbf{x}}_0 - \mathbf{x}_t}{\sigma_t^2} $$The second equality is the generalization of Tweedie’s formula from Section 1.6 to the case $\alpha_t \neq 1$. For the VP schedule ($\alpha_t^2 + \sigma_t^2 = 1$), the conversion between $\mathbf{v}$ and the other two is:
$$ \hat{\mathbf{x}}_0 = \alpha_t \mathbf{x}_t - \sigma_t \hat{\mathbf{v}}, \qquad \hat{\boldsymbol{\epsilon}} = \sigma_t \mathbf{x}_t + \alpha_t \hat{\mathbf{v}} $$Different parameterizations = different loss weighting. Substituting the conversions above gives:
$$ \left\| \boldsymbol{\epsilon} - \hat{\boldsymbol{\epsilon}} \right\|_2^2 = \text{SNR}(t) \left\| \mathbf{x}_0 - \hat{\mathbf{x}}_0 \right\|_2^2, \qquad \left\| \mathbf{v} - \hat{\mathbf{v}} \right\|_2^2 = \left( 1 + \text{SNR}(t) \right) \left\| \mathbf{x}_0 - \hat{\mathbf{x}}_0 \right\|_2^2 $$(The second equation requires the VP schedule.) In other words, the three parameterizations have the same optimum; they differ only in the weights implicitly assigned to different noise levels and in numerical stability:
- $\boldsymbol{\epsilon}$-prediction: At low noise the task is easy and the weight is high; but at very high noise ($\alpha_t \to 0$), recovering $\hat{\mathbf{x}}_0$ from $\hat{\boldsymbol{\epsilon}}$ requires dividing by $\alpha_t$, which amplifies errors. If the terminal SNR is exactly 0, $\boldsymbol{\epsilon}$-prediction is meaningless at the last step ($\mathbf{x}_T$ is just noise).
- $\mathbf{x}_0$-prediction: Stable at high noise, but at low noise the network must reproduce the input’s details exactly, so the effective regression weight is small.
- $\mathbf{v}$-prediction: Stable at both ends, and the choice used in progressive distillation and several large models; with $\sigma_{\text{data}} = 1$, EDM’s preconditioning also makes the inner network learn $\mathbf{v}$ (up to a sign).
- The velocity prediction of Flow Matching / Rectified Flow, $\mathbf{u} = \boldsymbol{\epsilon} - \mathbf{x}_0$ (corresponding to $\alpha_t = 1-t$ and $\sigma_t = t$), belongs to the same family; see Chapter 6 for details.
1.10 Chapter Summary
This chapter systematically established the mathematical foundations of diffusion models:
- The forward process progressively degrades data into noise via a Markov chain or SDE;
- The reverse process learns to denoise via a parameterized neural network;
- Variational inference provides a lower-bound optimization framework for maximum likelihood;
- Score matching provides score estimation methods that bypass the normalizing constant, with DSM the most practical form;
- Tweedie’s formula reveals the equivalence between denoisers and score functions;
- The SDE/ODE unified framework extends discrete models to continuous time and explains the deep connections among DDPM, NCSN, and DDIM;
- Noise-schedule equivalence shows that all mainstream methods are essentially different parameterizations of one design space: the continuous-time ELBO depends only on the SNR endpoints, and the schedule mainly affects loss weighting;
- The prediction targets ($\boldsymbol{\epsilon}$, $\mathbf{x}_0$, $\mathbf{v}$, score) are interconvertible, differing in implicit loss weighting and numerical stability.
These foundations provide the necessary theoretical preparation for Chapter 2’s analysis of specific classic models.
Chapter 2: Classic Diffusion Models: DDPM, DDIM, NCSN, and EDM
2.1 DDPM: Denoising Diffusion Probabilistic Model (Ho et al., 2020)
Denoising Diffusion Probabilistic Models (DDPM)3 is the landmark work that moved diffusion models from theory to large-scale image generation. Its core contribution was to simplify the complex variational lower bound into an intuitive mean squared error on noise prediction, and to show image quality comparable to or better than the GANs of the time.
2.1.1 Full Derivation and the Simplified Training Objective
DDPM fixes the reverse-process variance to $\Sigma_{\theta} = \sigma_t^2 \mathbf{I}$, with $\sigma_t^2 = \beta_t$ or $\tilde{\beta}_t$. In that case the KL divergence at step $t-1$, $L_{t-1}$, depends entirely on a mean squared error:
$$ L_{t-1} = \mathbb{E}_q \left[ \frac{1}{2\sigma_t^2} \left\| \tilde{\boldsymbol{\mu}}_t(\mathbf{x}_t, \mathbf{x}_0) - \boldsymbol{\mu}_{\theta}(\mathbf{x}_t, t) \right\|_2^2 \right] + C $$where $C$ is a constant independent of $\theta$. Note that the $\sigma_t^2$ here is the reverse-process variance, a different quantity from the $\sigma_t$ in the perturbation kernel of Section 1.8.
Ho et al. found that reparameterizing the reverse mean $\boldsymbol{\mu}_{\theta}$ as a noise predictor $\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t)$ brings a huge simplification. Since $\mathbf{x}_t = \sqrt{\bar{\alpha}_t} \mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t} \boldsymbol{\epsilon}$, you can solve for $\mathbf{x}_0 = \frac{\mathbf{x}_t - \sqrt{1-\bar{\alpha}_t} \boldsymbol{\epsilon}}{\sqrt{\bar{\alpha}_t}}$. Substituting into the posterior mean $\tilde{\boldsymbol{\mu}}_t$:
$$ \tilde{\boldsymbol{\mu}}_t = \frac{1}{\sqrt{\alpha_t}} \left( \mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \boldsymbol{\epsilon} \right) $$So parameterize the model’s prediction as:
$$ \boldsymbol{\mu}_{\theta}(\mathbf{x}_t, t) = \frac{1}{\sqrt{\alpha_t}} \left( \mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t) \right) $$| Symbol | Meaning |
|---|---|
| $\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t)$ | The noise predicted by the network, the $\boldsymbol{\epsilon}$-prediction parameterization |
| Loss | The MSE between the true noise $\boldsymbol{\epsilon}$ and the predicted noise $\boldsymbol{\epsilon}_{\theta}$ |
Substituting it back into $L_{t-1}$ turns the mean difference into a noise difference: $\tilde{\boldsymbol{\mu}}_t - \boldsymbol{\mu}_{\theta} = \frac{\beta_t}{\sqrt{\alpha_t} \sqrt{1 - \bar{\alpha}_t}} \left( \boldsymbol{\epsilon}_{\theta} - \boldsymbol{\epsilon} \right)$, so what follows strictly from the VLB is a weighted noise regression:
$$ L_{t-1} = \mathbb{E}_{\mathbf{x}_0, \boldsymbol{\epsilon}} \left[ \frac{\beta_t^2}{2 \sigma_t^2 \alpha_t (1 - \bar{\alpha}_t)} \left\| \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t) \right\|_2^2 \right] + C $$Ho et al. found that simply dropping the weight worked even better, giving the simplified objective:
$$ L_{\text{simple}} = \mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}} \left[ \| \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_{\theta}(\sqrt{\bar{\alpha}_t} \mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t} \boldsymbol{\epsilon}, t) \|_2^2 \right] $$| Symbol | Meaning |
|---|---|
| $t$ | A time step sampled uniformly from $\{1, \dots, T\}$ |
| $\mathbf{x}_0$ | Clean data sampled from the training set |
| $\boldsymbol{\epsilon}$ | Standard Gaussian noise |
| $L_{\text{simple}}$ | The simplified loss (DDPM’s standard training objective) |
Dropping the weight relatively downweights the small-$t$ terms (where noise is small and denoising is easy), letting the network spend more of its capacity on the harder high-noise denoising, which improves sample quality; the cost is that $L_{\text{simple}}$ is no longer a strict likelihood lower bound and the log-likelihood metric gets slightly worse.
Key insight: Since $\mathbf{s}_{\theta}(\mathbf{x}_t, t) = -\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t) / \sqrt{1 - \bar{\alpha}_t}$, DDPM training, under a specific weighting of noise levels, is equivalent to a weighted sum of denoising score matching (DSM) over $T$ discrete noise levels. $\boldsymbol{\epsilon}$-prediction turns score estimation into an intuitive noise-regression task through reparameterization.
2.1.2 Training Algorithm and Code Implementation
DDPM training algorithm:
| |
| Parameter | Description | Typical value |
|---|---|---|
| $T$ | Total diffusion steps | 1000 |
| $\beta_1$ | Initial variance | $10^{-4}$ |
| $\beta_T$ | Final variance | 0.02 |
| Schedule | How the variance grows | Linear |
| Network | Denoising network architecture | U-Net with Time Embeddings |
2.1.3 Sampling Algorithm
DDPM sampling starts from $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and iterates:
$$ \mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left( \mathbf{x}_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t) \right) + \sigma_t \mathbf{z} $$where $1 - \alpha_t = \beta_t$; $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ (for $t > 1$; at $t = 1$ take $\mathbf{z} = \mathbf{0}$); and $\sigma_t^2$ is $\beta_t$ or $\tilde{\beta}_t$ (Ho et al. found the two perform similarly; the code below uses $\tilde{\beta}_t$).
| Symbol | Meaning |
|---|---|
| $\mathbf{z}$ | Extra random noise injected during sampling |
| $\sigma_t$ | The standard deviation of the sampling noise |
| |
DDPM key characteristics:
- Strengths: Stable training (a single regression loss, no adversarial training), high generation quality, and strong theoretical interpretability;
- Weaknesses: Sampling requires $T$ network forward passes (1000 NFEs), so it’s slow;
- Use cases: High-quality image generation and as a base model for later acceleration methods;
- Key parameters: $T$, the $\beta$ schedule, and the network architecture (usually U-Net + self-attention);
- Caveats: Requires large batches and long training times; performance is sensitive to the $\beta$ schedule.
2.1.4 Improved DDPM: Cosine Schedule and Learned Variance (Nichol & Dhariwal, 2021)
Nichol and Dhariwal12 improved two weaknesses of DDPM, and later models such as ADM and GLIDE widely adopted these improvements.
1) Cosine noise schedule. The linear schedule destroys information too quickly in its second half: $\bar{\alpha}_t$ approaches 0 early on, so a large fraction of the final time steps are nearly pure noise and contribute little to training (especially noticeable on low-resolution images). The cosine schedule defines $\bar{\alpha}_t$ directly so that it decreases roughly linearly in the middle and changes gently at both ends:
$$ \bar{\alpha}_t = \frac{f(t)}{f(0)}, \qquad f(t) = \cos^2\left( \frac{t/T + s}{1 + s} \cdot \frac{\pi}{2} \right) $$| Symbol | Meaning |
|---|---|
| $s$ | A small offset (set to $0.008$) to keep $\beta_t$ from getting too small near $t = 0$ |
| $\beta_t$ | Inferred from $\beta_t = 1 - \bar{\alpha}_t / \bar{\alpha}_{t-1}$ and clipped to at most $0.999$ to avoid singularities as $t \to T$ |
2) Learned reverse variance. DDPM fixes the variance to $\beta_t$ or $\tilde{\beta}_t$. Ho et al. noted that these two correspond to the optimal variances in two extreme cases ($\mathbf{x}_0$ drawn from a standard Gaussian / $\mathbf{x}_0$ deterministic), and can be seen as upper and lower bounds on the true reverse variance. Improved DDPM has the network additionally output a vector $\mathbf{v}$ and interpolates elementwise in log space between the two:
$$ \Sigma_{\theta}(\mathbf{x}_t, t) = \exp\left( \mathbf{v} \log \beta_t + (1 - \mathbf{v}) \log \tilde{\beta}_t \right) $$Because $L_{\text{simple}}$ doesn’t depend on the variance, training uses a hybrid objective:
$$ L_{\text{hybrid}} = L_{\text{simple}} + \lambda L_{\text{VLB}}, \qquad \lambda = 0.001 $$where the $L_{\text{VLB}}$ term is used only to learn the variance (with a stop-gradient on the mean).
| Improvement | Effect |
|---|---|
| Cosine schedule | Better sample quality and likelihood on low-resolution data |
| Learned variance + $L_{\text{hybrid}}$ | Markedly better log-likelihood; less quality loss when reducing the number of sampling steps (e.g., to 50) |
Note: the interpolation coefficient $\mathbf{v}$ output by the network here shares a name with the $\mathbf{v}$-prediction target in Section 1.9, but the two are unrelated.
2.2 DDIM: Denoising Diffusion Implicit Model (Song et al., 2021)
DDPM’s main bottleneck is slow sampling. Song et al. (2021)16 proposed Denoising Diffusion Implicit Models (DDIM), whose core idea is to define a family of non-Markovian forward processes that share DDPM’s marginal distributions $q(\mathbf{x}_t \mid \mathbf{x}_0)$ but allow more flexible reverse sampling.
2.2.1 Non-Markovian Forward Processes
DDIM directly defines a family of “reverse” conditionals given $\mathbf{x}_0$ (the forward transition obtained by inverting Bayes’ rule, $q_{\sigma}(\mathbf{x}_t \mid \mathbf{x}_{t-1}, \mathbf{x}_0)$, depends on both $\mathbf{x}_{t-1}$ and $\mathbf{x}_0$, so the whole forward process is non-Markovian):
$$ q_{\sigma}(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0) = \mathcal{N}\left(\mathbf{x}_{t-1} \mid \sqrt{\bar{\alpha}_{t-1}} \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_{t-1} - \sigma_t^2} \cdot \frac{\mathbf{x}_t - \sqrt{\bar{\alpha}_t} \mathbf{x}_0}{\sqrt{1 - \bar{\alpha}_t}}, \sigma_t^2 \mathbf{I}\right) $$| Symbol | Meaning |
|---|---|
| $\sigma_t$ | A hyperparameter controlling the stochasticity of reverse sampling, $0 \le \sigma_t \le \sqrt{1 - \bar{\alpha}_{t-1}}$ |
| Mean, first term | $\sqrt{\bar{\alpha}_{t-1}} \mathbf{x}_0$: scales the clean signal to the magnitude it should have at step $t-1$ |
| Mean, second term | $\frac{\mathbf{x}_t - \sqrt{\bar{\alpha}_t} \mathbf{x}_0}{\sqrt{1 - \bar{\alpha}_t}}$ is exactly the noise $\boldsymbol{\epsilon}$ contained in $\mathbf{x}_t$, i.e., “the direction pointing toward $\mathbf{x}_t$” |
| Variance $\sigma_t^2 \mathbf{I}$ | The only source of randomness |
You can verify that for any $\sigma_t$ in the range above, the marginals $q_{\sigma}(\mathbf{x}_t \mid \mathbf{x}_0)$ of this family are identical to DDPM’s, namely $\mathcal{N}\left(\sqrt{\bar{\alpha}_t} \mathbf{x}_0, (1 - \bar{\alpha}_t) \mathbf{I}\right)$. Because $L_{\text{simple}}$ depends only on this marginal, the same trained $\boldsymbol{\epsilon}_{\theta}$ can be reused in any process in the family without retraining. In particular, when you take
$$ \sigma_t = \sqrt{\frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t}} \sqrt{1 - \frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}}} \quad \left( \text{i.e., } \sigma_t^2 = \tilde{\beta}_t \right) $$the forward process becomes Markovian again, and is exactly DDPM.
2.2.2 Deterministic Sampling and Acceleration
In practice a scalar $\eta \ge 0$ controls the stochasticity uniformly: $\sigma_t(\eta) = \eta \sqrt{\frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t}} \sqrt{1 - \frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}}}$. Replacing the true $\boldsymbol{\epsilon}$ with the network’s prediction (and thus getting an estimate $\hat{\mathbf{x}}_0$ of $\mathbf{x}_0$), the general DDIM update is:
$$ \mathbf{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \, \hat{\mathbf{x}}_0 + \sqrt{1 - \bar{\alpha}_{t-1} - \sigma_t^2} \, \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t) + \sigma_t \mathbf{z}, \qquad \hat{\mathbf{x}}_0 = \frac{\mathbf{x}_t - \sqrt{1 - \bar{\alpha}_t} \, \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t)}{\sqrt{\bar{\alpha}_t}} $$At $\eta = 1$ you recover DDPM’s stochastic sampling; at $\eta = 0$ you get fully deterministic sampling:
$$ \mathbf{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \underbrace{\left(\frac{\mathbf{x}_t - \sqrt{1-\bar{\alpha}_t} \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t)}{\sqrt{\bar{\alpha}_t}}\right)}_{\text{predicted } \mathbf{x}_0} + \underbrace{\sqrt{1 - \bar{\alpha}_{t-1}} \boldsymbol{\epsilon}_{\theta}(\mathbf{x}_t, t)}_{\text{noise direction}} $$| Symbol | Meaning |
|---|---|
| Term in parentheses | The estimate of $\mathbf{x}_0$ inferred from $\mathbf{x}_t$ and the predicted noise |
| Second term | Re-noising to the level of step $t-1$ using $\sqrt{1-\bar{\alpha}_{t-1}}$ |
DDIM’s key advantages:
- Training invariance: Uses exactly the same training objective as DDPM, with no retraining needed;
- Few-step sampling: Because training depends only on the marginal $q(\mathbf{x}_t \mid \mathbf{x}_0)$, you can sample on just a subsequence $\tau_1 < \dots < \tau_S$ ($S \ll T$, e.g., 50 steps or even 10), replacing $t-1$ in the formula with the previous time step in the subsequence; with few steps, $\eta = 0$ clearly outperforms stochastic sampling;
- Invertibility: The deterministic mapping provides an approximately invertible encoding from data to the latent $\mathbf{x}_T$, useful for interpolation and editing;
- Latent-space operations: You can perform semantic operations in the latent space $\mathbf{x}_T$ and then decode back to image space.
| Feature | Description |
|---|---|
| Use cases | Applications needing fast inference, image editing, and deterministic encoding |
| Key parameters | $\eta \in [0, 1]$ (stochasticity control) and the subsampling step count $S$ |
| Caveats | With too few steps, deterministic sampling can introduce artifacts; $S \geq 20$ usually works well |
| |
2.3 NCSN / NCSNv2: Noise Conditional Score Network (Song & Ermon, 2019–2020)
Noise Conditional Score Networks (NCSN)2 are another important branch in the history of diffusion models, building a generative model directly from the perspective of score matching.
2.3.1 Multi-Scale Noise Score Estimation
The core of NCSN is training a noise-conditioned network $\mathbf{s}_{\theta}(\mathbf{x}, \sigma)$ to estimate the score at multiple noise levels $\{\sigma_i\}_{i=1}^L$ simultaneously:
$$ \forall \sigma \in \{\sigma_1, \dots, \sigma_L\}: \quad \mathbf{s}_{\theta}(\mathbf{x}, \sigma) \approx \nabla_{\mathbf{x}} \log q_{\sigma}(\mathbf{x}) $$| Symbol | Meaning |
|---|---|
| $L$ | The number of noise levels: the original NCSN uses $L = 10$ ($\sigma_1 = 1$, $\sigma_L = 0.01$, with pixel values normalized to $[0, 1]$); NCSNv2 uses hundreds to thousands on high-resolution data (e.g., 232 for CIFAR-10 and 2311 for FFHQ $256 \times 256$) |
| $\sigma_i$ | The $i$-th noise level, usually in a geometric series $\sigma_1 > \sigma_2 > \dots > \sigma_L$ |
| $q_{\sigma}(\mathbf{x})$ | $\int p_{\text{data}}(\tilde{\mathbf{x}}) \mathcal{N}(\mathbf{x} \mid \tilde{\mathbf{x}}, \sigma^2 \mathbf{I}) \mathrm{d}\tilde{\mathbf{x}}$ |
The training objective is the sum of denoising score matching losses at each noise level:
$$ L_{\text{NCSN}}(\theta) = \frac{1}{L} \sum_{i=1}^{L} \lambda(\sigma_i) \mathbb{E}_{\mathbf{x}_0, \boldsymbol{\epsilon}} \left[ \left\| \mathbf{s}_{\theta}(\mathbf{x}_0 + \sigma_i \boldsymbol{\epsilon}, \sigma_i) + \frac{\boldsymbol{\epsilon}}{\sigma_i} \right\|_2^2 \right] $$| Symbol | Meaning |
|---|---|
| $\lambda(\sigma_i)$ | A noise-level-dependent loss weight, usually $\lambda(\sigma) = \sigma^2$: because the norm of the optimal score is roughly proportional to $1/\sigma$, multiplying by $\sigma^2$ puts the losses at different scales at roughly the same magnitude |
| Target | The network output $\mathbf{s}_{\theta}$ should predict $-\boldsymbol{\epsilon}/\sigma_i$ (the score) |
2.3.2 Annealed Langevin Sampling
First, recall Langevin dynamics: once you know the score, you can sample from a distribution $p(\mathbf{x})$:
$$ \mathbf{x}_{k+1} = \mathbf{x}_k + \delta \, \nabla_{\mathbf{x}} \log p(\mathbf{x}_k) + \sqrt{2\delta} \, \mathbf{z}_k, \qquad \mathbf{z}_k \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) $$| Symbol | Meaning |
|---|---|
| $\delta$ | Step size; as $\delta \to 0$ and the number of iterations $K \to \infty$, the distribution of $\mathbf{x}_K$ converges to $p(\mathbf{x})$ |
| Second term | “Climbing” along the score toward high-density regions |
| Third term | Injects noise to keep all samples from collapsing onto the same mode |
Running Langevin sampling directly on the data distribution has two problems: the data concentrate near a low-dimensional manifold, so in most regions the density is extremely low and the score estimate is inaccurate; and the modes of a multimodal distribution are separated by low-density regions, so the chain rarely switches between them. NCSN’s solution is annealed Langevin dynamics: first sample at large noise (where the distribution is well smoothed and the modes merge), then lower the noise level stage by stage, initializing each stage’s Langevin MCMC from the previous stage’s result:
| |
| Symbol | Meaning |
|---|---|
| $\alpha_i$ | The Langevin step size at the $i$-th noise level |
| $T$ | The number of steps per noise level, usually 100–200 |
| Final return | The refined sample at the smallest $\sigma_L$ (near 0) |
Why this step size: $\alpha_i = \epsilon \sigma_i^2 / \sigma_L^2$ keeps the signal-to-noise ratio of the update direction consistent across noise levels: at large noise the score is small, so use a large step; at small noise the score is large, so use a small step.
2.3.3 Improvements in NCSNv2
NCSNv2 (Song & Ermon, 2020)17 introduced several key techniques to improve performance and stability:
- Better noise-scale selection: The initial noise $\sigma_1$ is chosen from the maximum pairwise distance in the training data to ensure sample diversity;
- Geometric noise levels: The $\sigma_i$ are strictly geometric, $\sigma_1/\sigma_2 = \dots = \sigma_{L-1}/\sigma_L > 1$;
- Simplified noise conditioning: Parameterize $\mathbf{s}_{\theta}(\mathbf{x}, \sigma) = \mathbf{s}_{\theta}(\mathbf{x}) / \sigma$, reducing the complexity of the conditioning input;
- EMA for stable training: Use an exponential moving average to smooth the model weights and reduce fluctuations in sample quality;
- Step count and step size: Choose the Langevin step count $T$ and step size $\epsilon$ at each noise level from theory rather than empirical tuning;
- Architecture changes: Because noise information is now injected through output scaling, the network no longer needs conditional instance normalization (CondInstanceNorm++), instead using normalization layers without noise conditioning, and a deeper, wider RefineNet to support higher resolutions.
| NCSN vs NCSNv2 | NCSN | NCSNv2 |
|---|---|---|
| CIFAR-10 FID | 25.32 | 10.87 (with denoising) |
| Maximum resolution | $32 \times 32$ (original paper) | $256 \times 256$ |
| Noise conditioning | Explicit conditioning | Parameterized $\mathbf{s}_{\theta}(\mathbf{x})/\sigma$ |
| Training stability | Fair | EMA + architectural improvements |
| Step-size schedule | Empirical choice | Theory-driven |
| |
NCSN key characteristics:
- Strengths: Learns the score directly, with a simple theory; multi-scale noising and annealed sampling ease the problems of inaccurate scores in low-density regions and slow mixing between modes;
- Weaknesses: Requires many Langevin steps (usually $L \times T > 1000$ NFE) and is extremely slow; score estimates are still inaccurate in low-density regions where data are scarce;
- Use cases: Score estimation, density estimation, and anomaly detection;
- Key parameters: The number of noise levels $L$, the geometric ratio, and the step size $\epsilon$;
- Caveats: The noise range needs careful design; Langevin dynamics can be unstable in low-density regions.
2.4 EDM: Elucidating the Design Space of Diffusion Models (Karras et al., 2022)
Elucidating the Design Space of Diffusion-based Generative Models (EDM)5 is a systematic engineering work in the diffusion-model field. Rather than proposing a new architecture, it unifies, clarifies, and optimizes all the key design choices of diffusion models, establishing a modular design space.
2.4.1 Unified Framework and $\sigma$-Parameterization
EDM’s core is to unify all diffusion models in a continuous framework that takes the noise level $\sigma$ as the independent variable. Data perturbation is expressed as:
$$ \mathbf{x} = \mathbf{y} + \sigma \boldsymbol{\epsilon}, \quad \mathbf{y} \sim p_{\text{data}}, \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) $$| Symbol | Meaning |
|---|---|
| $\mathbf{y}$ | Clean data (the signal) |
| $\mathbf{x}$ | The noised observation |
| $\sigma$ | The continuous noise level, $\sigma \in [\sigma_{\min}, \sigma_{\max}]$ |
EDM trains a denoiser $D_{\theta}(\mathbf{x}; \sigma)$ to predict the clean data $\mathbf{y}$:
$$ \mathcal{L} = \mathbb{E}_{\sigma, \mathbf{y}, \boldsymbol{\epsilon}} \left[ \lambda(\sigma) \| D_{\theta}(\mathbf{y} + \sigma \boldsymbol{\epsilon}; \sigma) - \mathbf{y} \|_2^2 \right] $$| Symbol | Meaning |
|---|---|
| $D_{\theta}(\mathbf{x}; \sigma)$ | The denoising network, taking noised data and the noise level as input |
| $\lambda(\sigma)$ | A noise-dependent loss weight |
2.4.2 Preconditioning
EDM’s most important theoretical contribution is its preconditioning scheme, which uses $\sigma$-dependent scaling functions to keep the inputs and outputs of the network $F_{\theta}$ at unit variance across noise levels:
$$ D_{\theta}(\mathbf{x}; \sigma) = c_{\text{skip}}(\sigma) \mathbf{x} + c_{\text{out}}(\sigma) F_{\theta}(c_{\text{in}}(\sigma) \mathbf{x}; c_{\text{noise}}(\sigma)) $$| Symbol | Meaning | EDM choice |
|---|---|---|
| $c_{\text{skip}}(\sigma)$ | Skip-connection scaling | $\sigma_{\text{data}}^2 / (\sigma^2 + \sigma_{\text{data}}^2)$ |
| $c_{\text{out}}(\sigma)$ | Network output scaling | $\sigma \cdot \sigma_{\text{data}} / \sqrt{\sigma^2 + \sigma_{\text{data}}^2}$ |
| $c_{\text{in}}(\sigma)$ | Network input scaling | $1 / \sqrt{\sigma^2 + \sigma_{\text{data}}^2}$ |
| $c_{\text{noise}}(\sigma)$ | Noise-conditioning mapping | $\frac{1}{4} \ln(\sigma)$ |
| $\sigma_{\text{data}}$ | Data standard deviation | 0.5 (after image normalization) |
Design rationale for the preconditioning:
- $c_{\text{in}}$ keeps the magnitude of the input to $F_{\theta}$ around 1, independent of $\sigma$;
- $c_{\text{out}}$ ensures the error in $F_{\theta}$’s output isn’t amplified;
- $c_{\text{skip}}$ makes the network predict different targets at different noise levels: at low noise it mostly predicts the residual (noise), and at high noise it mostly predicts the signal (the data mean).
2.4.3 Loss Weighting and Training Noise Distribution
Substituting the preconditioning into the loss gives $\lambda(\sigma) \left\| D_{\theta} - \mathbf{y} \right\|_2^2 = \lambda(\sigma) \, c_{\text{out}}(\sigma)^2 \left\| F_{\theta} - \frac{\mathbf{y} - c_{\text{skip}}(\sigma) \mathbf{x}}{c_{\text{out}}(\sigma)} \right\|_2^2$, so for the inner network $F_{\theta}$ the effective loss weight is $\lambda(\sigma) \, c_{\text{out}}(\sigma)^2$. To make the effective weight equal 1 at every noise level, choose:
$$ \lambda(\sigma) = \frac{1}{c_{\text{out}}(\sigma)^2} = \frac{\sigma^2 + \sigma_{\text{data}}^2}{(\sigma \cdot \sigma_{\text{data}})^2} $$| Symbol | Meaning |
|---|---|
| $\lambda(\sigma)$ | The loss weight, balancing different noise levels |
During training, the noise level is sampled from a log-normal distribution:
$$ \ln(\sigma) \sim \mathcal{N}(P_{\text{mean}}, P_{\text{std}}^2) $$| Parameter | Meaning | Typical value |
|---|---|---|
| $P_{\text{mean}}$ | Log-mean | -1.2 |
| $P_{\text{std}}$ | Log-standard-deviation | 1.2 |
| $\sigma_{\min}$ | Minimum noise | 0.002 |
| $\sigma_{\max}$ | Maximum noise | 80.0 |
Key insight: At extremely low noise levels, distinguishing tiny amounts of noise is both difficult and unnecessary; at extremely high noise levels, the target is always close to the dataset mean. EDM therefore focuses training on intermediate noise levels, which matter most for forming perceptible content.
2.4.4 Samplers: Heun’s Second-Order Method and Stochastic Churn
EDM proposes a second-order deterministic sampler (based on Heun’s method) and a stochastic sampler (with a churn mechanism):
Deterministic sampler (Heun’s 2nd order):
$$ \mathbf{k}_1 = f(\mathbf{x}_i, \sigma(t_i)), \quad \mathbf{k}_2 = f(\mathbf{x}_i + \Delta t \cdot \mathbf{k}_1, \sigma(t_{i+1})) $$$$ \mathbf{x}_{i+1} = \mathbf{x}_i + \frac{1}{2}(\mathbf{k}_1 + \mathbf{k}_2) \Delta t $$| Symbol | Meaning |
|---|---|
| $f(\mathbf{x}, \sigma)$ | The derivative of the probability flow ODE, $\mathrm{d}\mathbf{x}/\mathrm{d}\sigma$, computed by the denoiser |
| $\Delta t$ | $t_{i+1} - t_i$; EDM takes $\sigma(t) = t$, so this is just the change in noise level, $\sigma_{i+1} - \sigma_i$ (a negative value) |
With $s(t) = 1$ and $\sigma(t) = t$, the probability flow ODE reduces to $\mathrm{d}\mathbf{x}/\mathrm{d}\sigma = -\sigma \nabla_{\mathbf{x}} \log p(\mathbf{x}; \sigma)$; then using Tweedie’s formula (Section 1.6) to replace the score with the denoiser gives:
$$ f(\mathbf{x}, \sigma) = -\sigma \nabla_{\mathbf{x}} \log p(\mathbf{x}; \sigma) = \frac{\mathbf{x} - D_{\theta}(\mathbf{x}; \sigma)}{\sigma} $$Intuitively, $f$ points “away from the denoised result”; since $\Delta t < 0$, each step actually moves toward the denoised result $D_{\theta}$. Heun’s method first tries an Euler step, then applies a trapezoidal correction using the slope at the endpoint, costing 2 network evaluations per step; the last step ($\sigma_{i+1} = 0$) reduces to a single Euler step.
Noise schedule (EDM discretization):
$$ \sigma_i = \left(\sigma_{\max}^{1/\rho} + \frac{i}{N-1}(\sigma_{\min}^{1/\rho} - \sigma_{\max}^{1/\rho})\right)^{\rho} $$| Symbol | Meaning | Typical value |
|---|---|---|
| $N$ | Number of sampling steps | 18–40 (CIFAR-10 to ImageNet-64, corresponding to 35–79 NFEs) |
| $\rho$ | Schedule shape parameter; larger values concentrate more steps at the low-noise end | 7 |
| $\sigma_i$ | The noise level at step $i$, with $\sigma_0 = \sigma_{\max}$, $\sigma_{N-1} = \sigma_{\min}$, plus an added $\sigma_N = 0$ | — |
Stochastic churn: EDM’s stochastic sampler first adds noise (churn) at each step: it raises the noise level from $\sigma_i$ to $\hat{\sigma}_i = (1 + \gamma_i) \sigma_i$, i.e., it adds fresh noise with variance $\hat{\sigma}_i^2 - \sigma_i^2$; then it integrates deterministically with one Heun step from $\hat{\sigma}_i$ to $\sigma_{i+1}$. This “noise-then-denoise” round trip is conceptually similar to the Langevin correction in Predictor-Corrector, and can correct errors accumulated in earlier steps. Key parameters:
| Parameter | Meaning | EDM value on ImageNet-64 |
|---|---|---|
| $S_{\text{churn}}$ | Total churn strength, with $\gamma_i = \min\left( S_{\text{churn}}/N, \sqrt{2} - 1 \right)$ at each step | 40 |
| $S_{\text{min}}$ | Lower bound: only add noise when $\sigma_i \in [S_{\text{min}}, S_{\text{max}}]$ | 0.05 |
| $S_{\text{max}}$ | Upper bound of the noise-injection interval | 50 |
| $S_{\text{noise}}$ | Amplification factor for the new noise’s standard deviation (slightly above 1, compensating for the denoiser’s oversmoothing) | 1.003 |
The optimal values of these parameters vary with the dataset and model; EDM determines them by grid search.
EDM key characteristics:
- Strengths: A modular design space and stable training; at publication it reached SOTA: class-conditional CIFAR-10 FID 1.79 (unconditional 1.97) with only 35 NFEs;
- Weaknesses: The preconditioning parameters need tuning for the data distribution;
- Use cases: Modern image generation, structure generation (e.g., proteins), and a baseline for much later work;
- Key parameters: $\sigma_{\text{data}}$, $P_{\text{mean}}$/$P_{\text{std}}$, the sampling step count $N$, and $\rho$;
- Caveats: When $\sigma_{\text{data}} = 1$, $F_{\theta}$’s regression target formally matches $\mathbf{v}$-prediction (up to a sign; see Section 1.9), adaptively interpolating between $\boldsymbol{\epsilon}$-prediction and $\mathbf{x}_0$-prediction as the noise level changes; the Heun sampler is tightly coupled to EDM’s $\sigma$ schedule.
| |
2.5 Noise-Schedule Equivalence and Relationships Among Models
The table below systematically compares where all the classic methods sit in the design space:
| Dimension | DDPM | DDIM | NCSN | NCSNv2 | EDM |
|---|---|---|---|---|---|
| Training objective | Variational lower bound / $\boldsymbol{\epsilon}$-MSE | Same as DDPM | Multi-scale DSM | Multi-scale DSM | Denoising MSE + preconditioning |
| Forward process | Markovian VP | Non-Markovian ($\eta$-parameterized) | Multi-scale VE | Multi-scale VE | Continuous $\sigma$ |
| Reverse sampling | Stochastic Markovian | Deterministic ($\eta = 0$, ODE) / stochastic ($\eta > 0$) | Annealed Langevin | Annealed Langevin | Heun 2nd + churn |
| Parameterization | $\boldsymbol{\epsilon}$-prediction | $\boldsymbol{\epsilon}$-prediction | Score $\mathbf{s}_{\theta}$ | $\mathbf{s}_{\theta}(\mathbf{x})/\sigma$ | Denoiser $D_{\theta}$ |
| Core innovation | Simplified VLB | Non-Markovian acceleration | Multi-noise score | Stability improvements | Design-space unification |
| Typical NFE | 1000 | 10–50 | 1000+ | 1000+ | 35–79 |
| CIFAR-10 FID | 3.17 | 4.67 (50 steps) / 4.04 (1000 steps) | 25.32 | 10.87 | 1.79 (class-conditional; unconditional 1.97) |
The FIDs in the table are all taken from the original papers, and the evaluation settings aren’t fully consistent (for example, DDIM uses $\eta = 0$ and shares the same model as DDPM), so they’re only good for rough comparison.
graph TB
subgraph "Diffusion model family"
A["Score matching
Hyvärinen 2005"] --> B["NCSN 2019
multi-noise score"]
A --> C["DDPM 2020
variational inference + ε-prediction"]
B --> D["Score SDE 2021
continuous unified framework"]
C --> D
C --> E["DDIM 2020/2021
non-Markovian/ODE"]
D --> F["EDM 2022
design space elucidated"]
E --> F
B --> G["NCSNv2 2020
improved training and architecture"]
G --> D
end
style A fill:#fff3e1
style D fill:#e1f5ff
style F fill:#e1ffe1
Mathematical relationships among the noise schedules: All schedules can be mapped to the $\alpha_t$-$\sigma_t$ representation. For example:
- DDPM’s $\beta_t$ schedule has the VP property $\alpha_t^2 + \sigma_t^2 = 1$;
- NCSN’s geometric series of $\sigma_i$ has the VE property $\alpha_t = 1$;
- DDIM keeps DDPM’s VP schedule ($\alpha_t = \sqrt{\bar{\alpha}_t}$, $\sigma_t = \sqrt{1 - \bar{\alpha}_t}$) and changes only the sampling process; the linear path $\alpha_t = 1 - t$, $\sigma_t = t$ corresponds to Flow Matching / Rectified Flow;
- EDM’s $\sigma$-parameterization unifies these into a continuous noise level: any $\mathbf{x}_t = \alpha_t \mathbf{x}_0 + \sigma_t \boldsymbol{\epsilon}$ can be written as $\mathbf{x}_t / \alpha_t = \mathbf{x}_0 + (\sigma_t / \alpha_t) \boldsymbol{\epsilon}$, i.e., a VE form with noise level $\sigma = \sigma_t / \alpha_t = 1/\sqrt{\text{SNR}}$.
With EDM’s preconditioning, the network $F_{\theta}$ under different schedules actually learns the same information, presented through different input/output scalings. This also explains why EDM’s sampler can be applied directly to previously trained VP and VE models and substantially improve them—the sampler and the training process can be designed separately.
2.6 Chapter Summary
This chapter analyzed four classic diffusion models in depth:
- DDPM simplified diffusion models through variational inference and $\boldsymbol{\epsilon}$-prediction, establishing the foundation for stable training and high-quality generation; Improved DDPM added a cosine schedule and learned variance;
- DDIM constructed a family of non-Markovian processes with the same marginals as DDPM, yielding deterministic, step-skipping sampling without retraining and achieving a 10–50× speedup;
- NCSN/NCSNv2 learn scores at multiple noise levels directly from the score-matching perspective, using annealed Langevin sampling to ease the problems of low-density regions and multimodal mixing, but at high computational cost;
- EDM unified the design space of diffusion models and, through systematic optimization of preconditioning, loss weighting, and samplers, reached state-of-the-art FID and efficiency.
These models form the foundation of modern diffusion models such as Stable Diffusion, DALL·E 2/3, and Sora. Understanding the mathematical relationships among them—especially the equivalence of score matching, denoising, and variational inference—is a necessary prerequisite for the advanced topics that follow (conditional generation, flow matching, consistency models).
References
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. ICML. ↩︎ ↩︎
Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. NeurIPS. ↩︎ ↩︎ ↩︎
Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS. ↩︎ ↩︎ ↩︎ ↩︎
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-based generative modeling through stochastic differential equations. ICLR. ↩︎ ↩︎ ↩︎
Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. NeurIPS. ↩︎ ↩︎ ↩︎ ↩︎
Hyvärinen, A. (2005). Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6, 695-709. ↩︎ ↩︎
Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural Computation, 23(7), 1661-1674. ↩︎
Song, Y., Garg, S., Shi, J., & Ermon, S. (2019). Sliced score matching: A scalable approach to density and score estimation. UAI 2019 (PMLR 115, published in 2020). ↩︎
Efron, B. (2011). Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496), 1602-1614. ↩︎
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., & Le, M. (2023). Flow matching for generative modeling. ICLR. ↩︎ ↩︎
Anderson, B. D. O. (1982). Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3), 313-326. ↩︎
Nichol, A., & Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. ICML. ↩︎ ↩︎
Liu, X., Gong, C., & Liu, Q. (2023). Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR. ↩︎
Kingma, D. P., Salimans, T., Poole, B., & Ho, J. (2021). Variational diffusion models. NeurIPS. ↩︎
Salimans, T., & Ho, J. (2022). Progressive distillation for fast sampling of diffusion models. ICLR. ↩︎
Song, J., Meng, C., & Ermon, S. (2021). Denoising diffusion implicit models. ICLR. ↩︎
Song, Y., & Ermon, S. (2020). Improved techniques for training score-based generative models. NeurIPS. ↩︎
