Chapter 5: Sampling Algorithms and Acceleration Strategies
Sampling a diffusion model is, at bottom, a matter of numerically solving a reverse stochastic differential equation (SDE) or the corresponding probability flow ODE (see §1.7). The original DDPM needed hundreds to thousands of network forward evaluations (NFE, Number of Function Evaluations) to produce a high-quality sample, which severely limited its deployment in real-time settings. This chapter surveys the full family of sampling-acceleration techniques, moving from simple to advanced:
- Training-free numerical solvers: leave the model untouched and change only “how the equation is solved” or “which timesteps are taken,” compressing 1000 steps to 10–25;
- Distillation methods: train an additional student model to compress multi-step sampling into 1–4 steps;
- Architecture-level efficiency optimizations: token pruning, feature caching, and the like, reducing the cost of each step.
Note
Two common notational systems appear in this chapter; take care to tell them apart:
- Discrete DDPM notation (§5.1.1–§5.1.2, §5.1.8): $\alpha_t = 1 - \beta_t$ is the per-step retention rate, $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$ the cumulative retention rate, and $\sigma_t$ the standard deviation of the noise injected in one reverse step.
- Continuous-time notation (§5.1.3–§5.1.6, §5.3, consistent with §1.8): the forward perturbation is written $x_t = \alpha_t x_0 + \sigma_t \epsilon$, where $\alpha_t, \sigma_t$ are the signal coefficient and noise coefficient, respectively. For VP diffusion (DDPM), $\alpha_t = \sqrt{\bar{\alpha}_t}$ and $\sigma_t = \sqrt{1 - \bar{\alpha}_t}$.
The $\alpha_t$ and $\sigma_t$ in the two systems have different meanings, so check which section a formula comes from before reading it.
5.1 The sampler family: from stochastic to deterministic
5.1.1 DDPM: the founder of stochastic sampling
Denoising Diffusion Probabilistic Models (DDPM)1 define two Markov chains, one for forward noising and one for reverse denoising (derivation in §2.1). The forward process gradually adds Gaussian noise to the data $x_0$, and the marginal distribution at any time has a closed form:
$$q(x_t \mid x_0) = \mathcal{N}\left(x_t; \sqrt{\bar{\alpha}_t}\, x_0, (1 - \bar{\alpha}_t) \mathbf{I}\right)$$where $\bar{\alpha}_t = \prod_{s=1}^{t} (1 - \beta_s)$ and $\beta_t$ is the preset noise schedule.
| Symbol | Meaning |
|---|---|
| $x_t$ | Noisy sample at timestep $t$ |
| $x_0$ | Original data sample |
| $\bar{\alpha}_t$ | Cumulative signal retention rate |
| $\beta_t$ | Noise variance at timestep $t$ |
| $\mathbf{I}$ | Identity covariance matrix |
The reverse process trains a noise-prediction network $\epsilon_\theta(x_t, t)$ to estimate and remove the noise; the sampling update is:
$$x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}}\, \epsilon_\theta(x_t, t) \right) + \sigma_t z$$| Symbol | Meaning |
|---|---|
| $\alpha_t = 1 - \beta_t$ | Per-step signal retention rate |
| $\sigma_t$ | Reverse-process noise standard deviation, often taken as $\sqrt{\beta_t}$ or the posterior standard deviation $\sqrt{\tilde{\beta}_t}$ ($\tilde{\beta}_t = \frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t}\beta_t$), or learned by the network (Improved DDPM) |
| $z \sim \mathcal{N}(0, \mathbf{I})$ | Standard Gaussian random noise (omitted at the final step $t=1$) |
| $\epsilon_\theta(x_t, t)$ | Noise predicted by the neural network |
Intuition: the term in parentheses “subtracts the noise the network thinks is in excess,” giving an estimate of the mean of $x_{t-1}$; $\sigma_t z$ then injects a little fresh noise to keep sampling stochastic.
DDPM sampling is stochastic: every step injects random noise. Its characteristics:
- Sampling steps: usually the same $T = 1000$ steps as training; skipping steps directly noticeably degrades quality
- Generation quality: with enough steps, sample quality is stable and rich in detail
- Diversity: the random noise injected at each step guarantees sample diversity
- Use cases: offline generation tasks where quality matters most and speed is not a concern
5.1.2 DDIM: a milestone in deterministic sampling
Denoising Diffusion Implicit Models (DDIM)2 is a milestone in sampling acceleration. Its key observation: DDPM’s training objective depends only on the marginal distribution $q(x_t \mid x_0)$, so one can construct a family of non-Markovian forward processes that share the same marginal distribution (and hence the same trained network) but correspond to different reverse sampling processes (derivation in §2.2):
$$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\, \underbrace{\frac{x_t - \sqrt{1 - \bar{\alpha}_t}\, \epsilon_\theta(x_t, t)}{\sqrt{\bar{\alpha}_t}}}_{\text{predicted } x_0} + \underbrace{\sqrt{1 - \bar{\alpha}_{t-1} - \sigma_t^2} \cdot \epsilon_\theta(x_t, t)}_{\text{direction component toward } x_t} + \underbrace{\sigma_t z}_{\text{stochastic component}}$$When $\sigma_t = 0$ is set, sampling becomes fully deterministic — this is what is usually meant by DDIM sampling:
$$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\, \hat{x}_0 + \sqrt{1 - \bar{\alpha}_{t-1}}\, \epsilon_\theta(x_t, t)$$| Symbol | Meaning |
|---|---|
| $\hat{x}_0 = \frac{x_t - \sqrt{1 - \bar{\alpha}_t}\, \epsilon_\theta(x_t, t)}{\sqrt{\bar{\alpha}_t}}$ | Clean data inferred from the current noise prediction (i.e., an estimate of the posterior mean $\mathbb{E}[x_0 \mid x_t]$, corresponding to Tweedie’s formula in §1.6) |
| $\sigma_t$ | Randomness control parameter; $\sigma_t = 0$ gives deterministic DDIM |
| $\eta$ | DDIM’s generalized parameter, $\sigma_t = \eta \sqrt{\frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t}} \sqrt{1 - \frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}}}$ |
Intuition: each DDIM step does two things — first it “guesses” $\hat{x}_0$ in one shot, then it “mixes” the same noise direction $\epsilon_\theta$ back in at the noise level appropriate to time $t-1$. Since no fresh noise is injected, the whole trajectory is uniquely determined by the initial noise $x_T$.
DDIM’s core contributions:
- Faster sampling: one can sample on a subsequence of timesteps with $S \ll T$, dropping from 1000 steps to 50–100 while keeping acceptable quality
- Deterministic mapping: the same initial noise $x_T$ always produces the same sample, which helps interpolation, image editing, and inversion (DDIM Inversion)
- ODE interpretation: DDIM is a first-order discretization of the probability flow ODE. Concretely, let $\bar{x}_t = x_t / \sqrt{\bar{\alpha}_t}$ and $\bar{\sigma}_t = \sqrt{(1 - \bar{\alpha}_t)/\bar{\alpha}_t}$; the DDIM update is exactly one Euler step of the ODE $\frac{\mathrm{d}\bar{x}}{\mathrm{d}\bar{\sigma}} = \epsilon_\theta$: $\bar{x}_{t-1} = \bar{x}_t + (\bar{\sigma}_{t-1} - \bar{\sigma}_t)\, \epsilon_\theta(x_t, t)$. It is also equivalent to the first-order DPM-Solver below (note: it is not Euler’s method applied directly in the original time variable $t$)
- Key parameter: $\eta$ controls randomness; $\eta = 0$ is fully deterministic, while $\eta = 1$ gives $\sigma_t^2 = \tilde{\beta}_t$, recovering DDPM’s ancestral sampling
5.1.3 DPM-Solver: a higher-order ODE solver
DPM-Solver3 exploits the semi-linear structure of the diffusion ODE: the right-hand side consists of a “linear term in $x_t$” and a “nonlinear term given by the neural network.” The linear part can be solved exactly and analytically, and only the nonlinear part needs numerical approximation, so the error is far smaller than handing the whole equation to a generic black-box solver.
In continuous-time notation $x_t = \alpha_t x_0 + \sigma_t \epsilon$, the probability flow ODE expressed with the noise-prediction network is:
$$\frac{\mathrm{d}x_t}{\mathrm{d}t} = f(t)\, x_t + \frac{g^2(t)}{2\sigma_t}\, \epsilon_\theta(x_t, t)$$| Symbol | Meaning |
|---|---|
| $f(t) = \frac{\mathrm{d}\log \alpha_t}{\mathrm{d}t}$ | Drift coefficient (linear part) |
| $g^2(t) = \frac{\mathrm{d}\sigma_t^2}{\mathrm{d}t} - 2\frac{\mathrm{d}\log \alpha_t}{\mathrm{d}t}\sigma_t^2$ | Square of the diffusion coefficient |
| $\epsilon_\theta(x_t, t)$ | Output of the noise-prediction network |
| $\lambda_t = \log \frac{\alpha_t}{\sigma_t}$ | Half the log signal-to-noise ratio (half log-SNR), increasing monotonically as $t$ decreases |
This follows from the probability flow ODE in §1.7.3: substitute the score $\nabla_x \log p_t(x_t) \approx -\epsilon_\theta(x_t, t)/\sigma_t$ into $f x - \frac{1}{2} g^2 \nabla_x \log p_t$ (hence the plus sign in front of the nonlinear term).
The key step in DPM-Solver is changing variables to $\lambda$. Using variation of constants on the linear part gives the exact solution from time $s$ to time $t$ ($t < s$):
$$x_t = \frac{\alpha_t}{\alpha_s} x_s - \alpha_t \int_{\lambda_s}^{\lambda_t} e^{-\lambda}\, \hat{\epsilon}_\theta(\hat{x}_\lambda, \lambda)\, \mathrm{d}\lambda$$where $\hat{\epsilon}_\theta(\hat{x}_\lambda, \lambda)$ denotes the network output after switching the time variable to $\lambda$. The first term is exact propagation of the linear part; the second is an “exponentially weighted integral of the network,” and only it needs approximation. Taking a $k-1$ order Taylor expansion of the network output yields a $k$-th-order solver.
Write the sampling time sequence as $t_0 = T > t_1 > \dots > t_M \approx 0$, with step size $h_i = \lambda_{t_i} - \lambda_{t_{i-1}} > 0$.
First order (DPM-Solver-1): treat the network output as constant over the interval, and the integral can be computed exactly:
$$x_{t_i} = \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}} x_{t_{i-1}} - \sigma_{t_i} \left( e^{h_i} - 1 \right) \epsilon_\theta(x_{t_{i-1}}, t_{i-1})$$Substituting $\alpha_t = \sqrt{\bar{\alpha}_t}$ and $\sigma_t = \sqrt{1 - \bar{\alpha}_t}$, one can verify that it is identical to DDIM ($\eta = 0$).
Second order (DPM-Solver-2, single-step method): take an intermediate point $s_i$ within the interval (satisfying $\lambda_{s_i} = \lambda_{t_{i-1}} + r_1 h_i$, often with $r_1 = \frac{1}{2}$) and use two network calls to estimate the first derivative:
$$u_i = \frac{\alpha_{s_i}}{\alpha_{t_{i-1}}} x_{t_{i-1}} - \sigma_{s_i} \left( e^{r_1 h_i} - 1 \right) \epsilon_\theta(x_{t_{i-1}}, t_{i-1})$$$$x_{t_i} = \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}} x_{t_{i-1}} - \sigma_{t_i} \left( e^{h_i} - 1 \right) \epsilon_\theta(x_{t_{i-1}}, t_{i-1}) - \frac{\sigma_{t_i}}{2 r_1} \left( e^{h_i} - 1 \right) \left( \epsilon_\theta(u_i, s_i) - \epsilon_\theta(x_{t_{i-1}}, t_{i-1}) \right)$$The last term is the “higher-order correction”: it uses the difference between the network outputs at the intermediate point and the starting point to approximate the rate of change of the network output with $\lambda$.
Features of DPM-Solver:
- Convergence speed: the paper reports FID 4.70 at 10 NFE and FID 2.87 at 20 NFE on CIFAR-103
- Training-free: applies directly to pretrained models (both discrete-time and continuous-time), with no extra training
- Choice of order: first order is equivalent to DDIM; the original second- and third-order versions are single-step methods (taking extra intermediate points inside each interval), while the later DPM-Solver++ provides a multi-step method that reuses historical points
- Use cases: unconditional sampling and conditional sampling with small guidance scales
5.1.4 DPM-Solver++: a solver built for guided sampling
DPM-Solver++4 is specially optimized for guided sampling. With classifier-free guidance (CFG, see §3.3), large guidance scales amplify the numeric range and estimation error of the network output, making higher-order solvers unstable — sometimes even worse than first-order DDIM.
The core improvement in DPM-Solver++ is solving the ODE in data-prediction parameterization:
$$x_\theta(x_t, t) = \frac{x_t - \sigma_t\, \epsilon_\theta(x_t, t)}{\alpha_t}$$| Symbol | Meaning |
|---|---|
| $x_\theta(x_t, t)$ | Clean data $x_0$ predicted by the network |
| $\alpha_t$ | Signal coefficient |
| $\sigma_t$ | Noise coefficient |
Under data-prediction parameterization, the probability flow ODE becomes:
$$\frac{\mathrm{d}x_t}{\mathrm{d}t} = \left( f(t) + \frac{g^2(t)}{2\sigma_t^2} \right) x_t - \frac{\alpha_t\, g^2(t)}{2\sigma_t^2}\, x_\theta(x_t, t)$$Its exact solution and first-order update are respectively:
$$x_t = \frac{\sigma_t}{\sigma_s} x_s + \sigma_t \int_{\lambda_s}^{\lambda_t} e^{\lambda}\, \hat{x}_\theta(\hat{x}_\lambda, \lambda)\, \mathrm{d}\lambda, \qquad x_{t_i} = \frac{\sigma_{t_i}}{\sigma_{t_{i-1}}} x_{t_{i-1}} - \alpha_{t_i} \left( e^{-h_i} - 1 \right) x_\theta(x_{t_{i-1}}, t_{i-1})$$Tip
Under EDM’s setting of $\alpha_t = 1$ and $\sigma_t = t$ (§2.4), the ODE above reduces to $\frac{\mathrm{d}x}{\mathrm{d}t} = \frac{x - x_\theta(x, t)}{t}$, which is exactly the equation solved by the EDM sampler.
The most commonly used DPM-Solver++(2M) (second-order multi-step method) reuses the network output already computed at the previous step, so it needs only 1 NFE per step:
$$x_{t_i} = \frac{\sigma_{t_i}}{\sigma_{t_{i-1}}} x_{t_{i-1}} - \alpha_{t_i} \left( e^{-h_i} - 1 \right) D_i, \qquad D_i = \left( 1 + \frac{1}{2 r_i} \right) x_\theta(x_{t_{i-1}}, t_{i-1}) - \frac{1}{2 r_i}\, x_\theta(x_{t_{i-2}}, t_{i-2})$$where $r_i = h_{i-1} / h_i$ is the ratio of two adjacent step sizes.
Why is data prediction more stable? The data $x_0$ has a well-defined range (e.g., pixels lie in $[-1, 1]$), whereas noise predictions at large guidance scales can be very large in magnitude; extrapolating in data space is less likely to “shoot out of” the training distribution. In addition, the data-prediction form can be combined directly with dynamic thresholding (from Imagen5) for pixel-space models:
$$s = \max\left( \operatorname{percentile}\left( \lvert x_\theta \rvert, p \right), 1 \right), \qquad x_\theta \leftarrow \frac{\operatorname{clip}\left( x_\theta, -s, s \right)}{s}$$| Symbol | Meaning |
|---|---|
| $p$ | Percentile (often 99.5% in Imagen) |
| $s$ | Dynamic threshold computed per sample |
| $\operatorname{clip}(\cdot, -s, s)$ | Clip values to the interval $[-s, s]$ |
That is: if many pixels in the prediction fall outside $[-1, 1]$, first clip them to $[-s, s]$, then divide everything by $s$ to bring them back into $[-1, 1]$, alleviating oversaturation at large guidance scales. Latent-space models (such as Stable Diffusion) generally do not use thresholding.
Practical configurations for DPM-Solver++:
- Recommended order: second-order multi-step (2M) is the most common; third-order multi-step (3M) suits no guidance or small guidance scales
- Number of steps: the paper reports that 15–20 steps suffice for high-quality guided sampling (for both pixel-space and latent-space models)4
- Compatibility with CFG: specially optimized for stability at large guidance scales
- SDE variant: stochastic variants such as DPM-Solver++(2M) SDE are also common in practice; they inject a little noise at each step, giving richer detail at few steps but no longer deterministic results
5.1.5 UniPC: a unified predictor-corrector framework
UniPC (Unified Predictor-Corrector)6 brings the predictor-corrector idea from classical ODE numerical methods to diffusion sampling. It has two components:
- UniP (unified predictor): uses several historical points to predict the solution at the next step, supporting arbitrary order
- UniC (unified corrector): once the network output at the new point is available, it goes back and corrects the previous prediction, raising the accuracy order by one
In noise-prediction form, for example, the update of a $p$-th-order predictor has a unified analytic form:
$$x_{t_i} = \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}} x_{t_{i-1}} - \sigma_{t_i} \left( e^{h_i} - 1 \right) \epsilon_\theta(x_{t_{i-1}}, t_{i-1}) - \sigma_{t_i} B(h_i) \sum_{m=1}^{p-1} \frac{a_m}{r_m} D_m$$| Symbol | Meaning |
|---|---|
| $h_i$ | Log-SNR step size $\lambda_{t_i} - \lambda_{t_{i-1}}$ |
| $B(h_i)$ | Any nonzero function satisfying $B(h) = O(h)$, often $B(h) = h$ or $B(h) = e^{h} - 1$ |
| $D_m$ | Difference term $\epsilon_\theta(x_{s_m}, s_m) - \epsilon_\theta(x_{t_{i-1}}, t_{i-1})$, where $s_m$ is a historical time point |
| $r_m$ | Relative step size $(\lambda_{s_m} - \lambda_{t_{i-1}}) / h_i$ |
| $a_m$ | Coefficients determined by the order conditions (a small linear system) |
The first two terms are exactly DPM-Solver-1 (i.e., DDIM); the summation term is the higher-order correction using historical information.
The corrector (UniC) update is:
$$x_{t_i}^{c} = \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}} x_{t_{i-1}}^{c} - \sigma_{t_i} \left( e^{h_i} - 1 \right) \epsilon_\theta(x_{t_{i-1}}^{c}, t_{i-1}) - \sigma_{t_i} B(h_i) \sum_{m=1}^{p} \frac{a_m}{r_m} D_m$$Compared with the predictor, the sum has one extra term, the $p$-th: it uses the network output at the predicted point $x_{t_i}$ (taking $r_p = 1$). Since that network output would be computed anyway at the next step, correction requires no extra NFE.
Advantages of UniPC:
- Strong at very few steps: the paper reports FID 3.87 on CIFAR-10 (unconditional) and FID 7.51 on ImageNet 256×256 (conditional) with just 10 NFE6
- Plug-and-play: UniC can be attached after any existing solver to raise the accuracy order without extra network calls
- Arbitrary order: the analytic form holds uniformly for any order, with no need to re-derive formulas for each order
5.1.6 DEIS: the exponential integrator sampler
Diffusion Exponential Integrator Sampler (DEIS)7 likewise exploits the semi-linear structure of the diffusion ODE, using the well-established exponential integrator framework from numerical analysis. Its analysis identifies the discretization scheme as the single most important factor for few-step sampling quality; treating the linear part analytically and pairing it with $\epsilon$-parameterization (where the network output varies more gently over time) can substantially reduce discretization error.
Using the integrating-factor method, the solution of the ODE can be written as:
$$x_{t_i} = \Phi(t_i, t_{i-1})\, x_{t_{i-1}} + \int_{t_{i-1}}^{t_i} \Phi(t_i, \tau)\, \frac{g^2(\tau)}{2\sigma_\tau}\, \epsilon_\theta(x_\tau, \tau)\, \mathrm{d}\tau, \qquad \Phi(t, s) = \exp\left( \int_{s}^{t} f(u)\, \mathrm{d}u \right)$$| Symbol | Meaning |
|---|---|
| $\Phi(t, s)$ | Analytic propagation factor (transition function) of the linear part; for VP diffusion, $\Phi(t, s) = \alpha_t / \alpha_s$ |
| $\epsilon_\theta(x_\tau, \tau)$ | Nonlinear part (the network’s noise estimate) |
| $f(\tau), g(\tau)$ | Drift and diffusion coefficients |
(Note that during sampling $t_i < t_{i-1}$, so the integral runs “backward.”)
DEIS approximates the nonlinear integral with polynomial extrapolation (Adams–Bashforth-like multi-step methods) or Runge–Kutta methods to achieve higher-order sampling:
- Performance: FID 4.17 at 10 NFE and FID 3.37 at 15 NFE on CIFAR-107
- Algorithm variants: $t$AB / $t$RK, which fit polynomials in the original time $t$, and $\rho$AB / $\rho$RK, which fit in the reparameterized time $\rho = \sigma_t / \alpha_t$ (noise-to-signal ratio); the first-order method in $\rho$ space is equivalent to DDIM
5.1.7 Align Your Steps (AYS): optimizing the sampling schedule
The solver determines “how to take each step,” while the sampling schedule determines “which noise levels to land on.” Past schedules were mostly hand-designed (uniform steps, EDM’s $\rho = 7$ polynomial schedule, etc.). Align Your Steps (AYS)8 was the first to offer a principled optimization method: using stochastic calculus, it minimizes an upper bound on the KL divergence (KLUB) between the path distributions of the “discretized SDE” and the “true SDE,” yielding optimal timesteps for a given solver, model, and dataset:
$$\min_{\{t_i\}_{i=0}^{N}} \mathrm{KLUB}\left( q_{\text{true}} \,\Vert\, q_{\text{disc}} \right)$$| Symbol | Meaning |
|---|---|
| $\{t_i\}$ | Sequence of sampling timesteps to optimize |
| $N$ | Number of sampling steps |
| $\mathrm{KLUB}$ | KL-divergence upper bound |
| $q_{\text{true}}$ | Path distribution of the true reverse SDE |
| $q_{\text{disc}}$ | Path distribution after discretization (linearized approximation) at the given timesteps |
Intuition: the “information content” is not uniform across noise levels — in some regions the trajectory bends sharply and needs dense sampling, while in others it is nearly straight and can be crossed in big strides. AYS makes the steps follow the trajectory’s “degree of curvature.”
Features of AYS:
- On images, video, and 2D toy data, with several solvers, the optimized schedules beat hand-designed ones in almost every experiment, with the largest gains in few-step settings (around 10 steps)8
- Training-free and plug-and-play: it only replaces the timestep sequence, leaving the model and solver unchanged
5.1.8 Analytic-DPM: an analytic estimate of the optimal variance
Analytic-DPM9 (ICLR 2022 Outstanding Paper Award) addresses an often-overlooked question: how should the variance $\sigma_t^2$ in the reverse Gaussian transition $p(x_{t-1} \mid x_t) = \mathcal{N}(\mu_t(x_t), \sigma_t^2 \mathbf{I})$ be chosen? DDPM uses hand-crafted $\beta_t$ or $\tilde{\beta}_t$, while Improved DDPM lets the network learn it. Analytic-DPM proves that, in the sense of minimizing KL divergence, both the optimal variance and the corresponding optimal KL have analytic forms in terms of the score function.
Consider the DDIM family of forward processes $q_\lambda(x_{t-1} \mid x_t, x_0) = \mathcal{N}\left(\tilde{\mu}_t(x_t, x_0), \lambda_t^2 \mathbf{I}\right)$ ($\lambda_t^2 = \tilde{\beta}_t$ corresponds to DDPM, $\lambda_t = 0$ to DDIM); the optimal reverse variance is:
$$\sigma_t^{*2} = \lambda_t^2 + \left( \sqrt{\frac{1 - \bar{\alpha}_t}{\alpha_t}} - \sqrt{1 - \bar{\alpha}_{t-1} - \lambda_t^2} \right)^2 \left( 1 - \frac{\mathbb{E}_{q(x_t)}\left[ \left\lVert \epsilon_\theta(x_t, t) \right\rVert_2^2 \right]}{d} \right)$$| Symbol | Meaning |
|---|---|
| $\sigma_t^{*2}$ | Optimal reverse variance |
| $\lambda_t^2$ | Variance parameter of the chosen forward-process family (note: here $\lambda_t$ is not the log SNR) |
| $\alpha_t, \bar{\alpha}_t$ | Per-step / cumulative retention rate in discrete DDPM notation |
| $d$ | Data dimensionality |
| $\mathbb{E}\left[\lVert \epsilon_\theta \rVert_2^2\right] / d$ | Average energy of the noise prediction, estimated by Monte Carlo on a small number of samples |
Intuition: the more “confident” the network’s noise prediction (energy closer to $d$, meaning $x_0$ is estimated more accurately), the less extra variance needs to be compensated. The paper states this in terms of the score function $s_\theta = -\epsilon_\theta / \sqrt{1 - \bar{\alpha}_t}$; the two are equivalent.
By estimating the expectation in the formula above with Monte Carlo, Analytic-DPM achieves training-free inference improvements:
- Speedup: 20–80× faster than the original DDPM / DDIM at similar sample quality, with no retraining9
- Better log-likelihood: improves the likelihood of several diffusion models
- Variance clipping: derives upper and lower bounds on the optimal variance and clips the estimate to correct for bias in the score model itself
5.1.9 Sampler comparison summary
graph TD
A[Diffusion model samplers] --> B[Stochastic sampling]
A --> C[Deterministic sampling]
B --> D[DDPM
1000 steps, high quality]
C --> E[DDIM
50-100 steps, deterministic]
C --> F[Higher-order ODE solvers
10-20 steps]
F --> G[DPM-Solver
semi-linear structure]
F --> H[DEIS
exponential integrator]
F --> I[UniPC
predictor-corrector]
C --> J[Guidance optimization
DPM-Solver++]
C --> K[Schedule optimization
AYS]
B --> L[Analytic variance
Analytic-DPM]
style D fill:#ffcccc
style E fill:#ccffcc
style G fill:#ccccff
style H fill:#ccccff
style I fill:#ccccff
style J fill:#ffffcc
style K fill:#ffffcc
style L fill:#ffffcc
Note
Analytic-DPM optimizes the variance of the reverse stochastic process, so it sits on the stochastic-sampling side; AYS optimizes the schedule and applies to both stochastic and deterministic solvers.
5.2 The trade-off between sampling steps and quality: NFE vs FID
The central tension in sampling acceleration is the trade-off between the number of network calls (NFE) and sample quality (usually measured by FID, where lower is better). The table below summarizes representative results reported in the original papers:
| Sampler | NFE | CIFAR-10 FID | ImageNet 256 FID | Type |
|---|---|---|---|---|
| DDPM | 1000 | 3.17 | — | Stochastic |
| DDIM ($\eta = 0$) | 100 | 4.16 | — | Deterministic, first-order |
| DPM-Solver | 10 / 20 | 4.70 / 2.87 | — | Higher-order ODE (single-step) |
| DPM-Solver++ | 15–20 | — | — | Guided-sampling optimization (the paper’s main evaluation is guided-sampling quality) |
| UniPC | 10 | 3.87 | 7.51 (conditional) | Predictor-corrector |
| DEIS | 10 / 15 | 4.17 / 3.37 | — | Exponential integrator |
| Analytic-DPM | — | 20–80× speedup at similar quality | — | Analytic variance |
Warning
The numbers come from their respective papers, which used different pretrained models, training setups, and evaluation details; read them for trends, not for item-by-item comparison across methods. “—” means the paper did not report that setting or it is not included here. With CFG, each step must compute both the conditional and unconditional branches, so the actual number of network forward passes is about twice the number of steps.
Key observations:
- Diminishing returns: beyond roughly 20 NFE, the improvement in FID clearly slows
- Sweet spot for higher-order solvers: 10–20 steps is the most efficient range for most training-free solvers
- The challenge of guided sampling: CFG amplifies the error of higher-order methods; DPM-Solver++’s data-prediction and multi-step design was built precisely for this
- Training-free vs. training-based: training-free methods (DPM-Solver++, UniPC, etc.) can compress 1000 steps to 10–20; getting down to 1–4 steps usually requires extra training such as distillation (§5.3)
5.3 Distillation methods: from many steps to few
Training-free solvers typically degrade noticeably below 10 steps, because the curvature of the ODE trajectory cannot be followed accurately with very few straight segments. The idea behind distillation is: use a pretrained multi-step model (the teacher) as an “answer generator” and train a student model that can take large jumps. By “what the student imitates,” they fall roughly into three categories:
- Imitating the trajectory: the student reproduces in one step what the teacher does in several (Progressive Distillation, Consistency Distillation §6.2, Reflow §6.5);
- Imitating the distribution: no point-by-point alignment is required; only the overall distribution of the student’s samples must match the teacher’s (DMD and others, §5.3.4);
- Adversarial enhancement: add a discriminator to make up for blur and missing detail at few steps (ADD, LADD).
5.3.1 Progressive Distillation
Progressive Distillation10 uses a student-teacher framework, halving the number of sampling steps each round: the student’s one-step DDIM update must reproduce the teacher’s two-step DDIM update. Suppose the teacher samples on an $N$-step time grid, at current time $t$, and write $t' = t - \frac{0.5}{N}$ and $t'' = t - \frac{1}{N}$ (continuous time $t \in [0, 1]$, notation as in §5.1.3). First take two DDIM steps with the teacher $\hat{x}_\eta$:
$$z_{t'} = \alpha_{t'}\, \hat{x}_\eta(z_t) + \frac{\sigma_{t'}}{\sigma_t} \left( z_t - \alpha_t\, \hat{x}_\eta(z_t) \right), \qquad z_{t''} = \alpha_{t''}\, \hat{x}_\eta(z_{t'}) + \frac{\sigma_{t''}}{\sigma_{t'}} \left( z_{t'} - \alpha_{t'}\, \hat{x}_\eta(z_{t'}) \right)$$Then solve backward for the $x$-prediction target that lets the student go from $z_t$ to $z_{t''}$ in one step, and regress on it:
$$\tilde{x} = \frac{z_{t''} - \frac{\sigma_{t''}}{\sigma_t} z_t}{\alpha_{t''} - \frac{\sigma_{t''}}{\sigma_t} \alpha_t}, \qquad \mathcal{L}_{\text{PD}}(\theta) = \mathbb{E}\left[ w(\lambda_t) \left\lVert \tilde{x} - \hat{x}_\theta(z_t) \right\rVert_2^2 \right]$$| Symbol | Meaning |
|---|---|
| $\hat{x}_\theta$ | Student model (predicts clean data) |
| $\hat{x}_\eta$ | Teacher model (the student trained in the previous round, with frozen parameters) |
| $z_t$ | Sample obtained by forward-noising training data, $\alpha_t x + \sigma_t \epsilon$ |
| $\tilde{x}$ | Distillation target: makes the student’s one DDIM step land exactly on the teacher’s two-step result |
| $w(\lambda_t)$ | Loss weight depending on the log SNR (e.g., truncated-SNR weighting) |
After each round of training, the student becomes the teacher for the next round, and the step count halves as $N \to N/2 \to N/4 \to \dots$. Starting from a sampler with as many as 8192 steps, the paper distills down to 4 steps with almost no perceptual quality loss (4-step FID 3.0 on CIFAR-10), and the total time for the whole distillation pipeline does not exceed that of training the original model10.
The work also proposed parameterizations that are more stable at very few steps, the most influential being $v$-prediction: $v = \alpha_t \epsilon - \sigma_t x_0$, later widely used in models such as Stable Diffusion 2.x. The subsequent Guided Distillation11 distills CFG’s two forward passes, “conditional + unconditional,” into a single model and then applies progressive distillation, enabling few-step generation under guided sampling as well.
5.3.2 Adversarial Diffusion Distillation (ADD)
ADD12 combines GAN-style adversarial training with diffusion distillation and was the first method to achieve one-step real-time generation on a large-scale foundation model (SDXL-Turbo is based on it):
$$\mathcal{L}_{\text{ADD}} = \mathcal{L}_{\text{adv}}^{G}\left( \hat{x}_\theta(x_s, s), \phi \right) + \lambda\, \mathcal{L}_{\text{distill}}\left( \hat{x}_\theta(x_s, s), \psi \right)$$| Symbol | Meaning |
|---|---|
| $\hat{x}_\theta(x_s, s)$ | Sample generated by the student in one step from noisy input $x_s$ |
| $\phi$ | Trainable parameters of the discriminator (a frozen DINOv2 feature network + a trainable lightweight discriminator head) |
| $\psi$ | Frozen parameters of the teacher diffusion model |
| $\mathcal{L}_{\text{adv}}^{G}$ | Adversarial loss on the generator side |
| $\mathcal{L}_{\text{distill}}$ | Score-distillation loss: re-noise the student’s output to $t$, let the teacher denoise it, and pull the student’s output toward the teacher’s denoised result |
| $\lambda$ | Loss-balancing coefficient |
Key design choices in ADD:
- Adversarial loss: uses features from a pretrained ViT (DINOv2) for discrimination, forcing the one-step output to have crisp high-frequency detail and avoiding the blur common in few-step distillation
- Distillation loss: leverages a large teacher model (such as SDXL) to preserve compositional ability and fidelity to text semantics
- Performance: the paper reports that a single step clearly beats existing few-step methods (GAN, LCM), and 4 steps reach the level of its teacher SDXL12
- Follow-up: LADD (Latent ADD)13 no longer relies on a pixel-space DINOv2 discriminator, instead discriminating directly on features generated by a pretrained latent diffusion model itself; the entire training happens in latent space, supporting high-resolution, multi-aspect-ratio generation. Applied to SD3 (8B), it yields SD3-Turbo, which with 4-step unguided sampling comes close to the best text-to-image models of its time
5.3.3 InstaFlow: Reflow for text-to-image
InstaFlow14 applies the Reflow idea from Rectified Flow (see §6.5) to large-scale text-to-image generation: first generate “noise–image” pairs with a teacher ODE, retrain on these pairs to straighten the curved mapping into a nearly straight line, then distill the straightened flow into a one-step model. When the path is straight enough, a single Euler step is accurate:
$$x_1 \approx x_0 + v_\theta(x_0, 0), \quad x_0 \sim \mathcal{N}(0, \mathbf{I})$$Here the Rectified Flow convention carries over: $t = 0$ is noise and $t = 1$ is an image (consistent with §6.4), and $v_\theta$ is the learned velocity field. InstaFlow achieves a one-step FID of 23.3 on MS COCO 2017-5k (the previous best progressive distillation was 37.2), validating Reflow’s role in complex conditional generation; see §6.5.2 for details.
5.3.4 Distribution Matching Distillation (DMD): from “aligning trajectories” to “aligning distributions”
Methods that imitate the teacher’s trajectory point by point (PD, CD, Reflow) share a common ceiling: the student must reproduce exactly the teacher’s one output for each noise, which is hard to do in a single step. Distribution Matching Distillation (DMD)15 swaps the objective: it only requires the distribution of samples $p_{\text{fake}}$ produced by the one-step generator $G_\theta$ to match the teacher distribution $p_{\text{real}}$, i.e., minimizing the KL divergence (at each noise level). Its gradient can be written as the difference of two score functions:
$$\nabla_\theta D_{\mathrm{KL}}\left( p_{\text{fake}} \,\Vert\, p_{\text{real}} \right) \approx \mathbb{E}_{z, t, x_t}\left[ w_t \left( s_{\text{fake}}(x_t, t) - s_{\text{real}}(x_t, t) \right) \frac{\partial G_\theta(z)}{\partial \theta} \right]$$| Symbol | Meaning |
|---|---|
| $G_\theta(z)$ | One-step generator: takes noise $z$ and directly outputs an image |
| $x_t$ | Sample obtained by forward-noising $G_\theta(z)$ to time $t$ |
| $s_{\text{real}}$ | Score given by the teacher diffusion model (frozen) |
| $s_{\text{fake}}$ | Score from another diffusion model trained online on the generator’s current output, representing the “generated distribution” |
| $w_t$ | Time-dependent normalization weight |
Intuition: $s_{\text{real}}$ pushes samples toward high-density regions of the real distribution, while $s_{\text{fake}}$ pushes them away from regions where the generated distribution has already “clustered”; their difference gives the direction “in which to change.” This gradient is formally related to the SDS / VSD used for 3D generation in Chapter 7 (§7.2.2).
DMD also adds a regression loss based on the teacher’s multi-step output to stabilize training; the paper reports one-step FID 2.62 on ImageNet 64×64 and FID 11.49 on zero-shot COCO-30k15. DMD216 removes the costly regression dataset, uses two-timescale updates to stabilize training of $s_{\text{fake}}$, and adds a GAN loss that uses real data directly, while also supporting multi-step sampling.
5.4 Model efficiency optimizations
The methods so far reduce the number of steps; the methods in this section reduce the cost of each step. The two can be combined.
5.4.1 Token pruning, merging, and routing
Diffusion Transformers (DiT, §6.1) treat an image as a sequence of patches, and the $O(N^2)$ complexity of self-attention ($N$ being the number of tokens) is the main bottleneck. Representative ideas include:
- Token Merging (ToMe for SD)17: merge redundant tokens by similarity before attention, then “unmerge” them afterward, accelerating Stable Diffusion without training
- Token pruning: use importance scores such as attention maps to dynamically skip low-importance tokens
- Token routing: as in TREAD18, during training have a randomly selected subset of tokens bypass several intermediate layers and go straight to deeper ones, cutting training compute while improving convergence
The shared idea behind dynamic pruning / routing can be sketched as (not a specific formula from any one paper):
$$\rho_t = f\left( \mathrm{Entropy}(A_t),\, t \right)$$| Symbol | Meaning |
|---|---|
| $\rho_t$ | Fraction of tokens retained at timestep $t$ |
| $A_t$ | Attention distribution at the current step; $\mathrm{Entropy}(\cdot)$ is its entropy |
| $f$ | Rule or small network that decides how many tokens to keep |
5.4.2 Caching strategies: DeepCache and FORA
Diffusion sampling is iterative, and intermediate features at adjacent timesteps are highly similar. Caching strategies exploit this temporal redundancy to skip repeated computation.
DeepCache19 is designed for U-Net. The high-level semantic features from the deep (low-resolution) branch of a U-Net change slowly between adjacent steps, while the shallow branches are cheap to compute. So DeepCache runs a full pass every few steps and caches the high-level features of the deep upsampling branch; on intermediate steps it recomputes only the shallow branches and concatenates them with the cached features via skip connections. The paper reports about a 2.3× speedup on Stable Diffusion v1.5 (with CLIP Score dropping by only 0.05)19:
| |
FORA (Fast-FORward CAching)20 is designed specifically for DiT architectures, caching the intermediate outputs of Attention and MLP in each layer:
| Parameter | Meaning | Typical value |
|---|---|---|
| $N$ | Cache interval | 2–5 |
cache[Attn][k] | Attention output of layer $k$ | — |
cache[MLP][k] | MLP output of layer $k$ | — |
FORA uses static caching: it recomputes and caches features every $N$ steps and reuses them for the $N - 1$ steps in between. The paper reports several-fold speedups on DiT with only slight effects on FID and IS, and no retraining20.
Advanced caching methods:
- $\Delta$-DiT21: finds that the early blocks of a DiT mainly determine contours while the late blocks mainly determine detail, so it caches the late blocks early in sampling and the early blocks late in sampling
- TeaCache22: uses the difference in inputs after modulation by the timestep embedding to estimate the difference in outputs (then recalibrates with a polynomial), adaptively deciding which steps can reuse the cache; aimed mainly at video diffusion models
- TaylorSeer23: moves from “reuse” to “prediction” — it approximates higher derivatives with finite differences of historical cached features, then uses a Taylor expansion to predict features at future timesteps, giving smaller error at high speedup ratios
5.5 Code practice: a unified interface for multiple samplers
The wrapper below is built on diffusers schedulers and switches between solvers through a single interface:
| |
| Parameter | Description |
|---|---|
scheduler_name | Solver type: ddpm / ddim / dpm++ / unipc / deis |
num_steps | Number of sampling steps, typically 10–50 |
cfg_scale | Guidance scale $w$, using the first form in §3.3.2, $\hat{\epsilon} = \epsilon_{\text{uncond}} + w(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})$; $w = 1$ is equivalent to no CFG, enabled when $w > 1$, typically 5–10 |
model | Noise-prediction U-Net or DiT |
Note
This code is conceptual: in practice, the scheduler should be loaded from the pretrained model’s configuration (e.g.,
DPMSolverMultistepScheduler.from_pretrained(..., subfolder="scheduler")) so that the noise schedule and prediction type ($\epsilon$ / $v$) match training; the unconditional branch usually uses the embedding of an empty string rather than an all-zero tensor.
5.6 Chapter summary and selection guide
| Use case | Recommended sampler | Recommended steps | Key consideration |
|---|---|---|---|
| Highest quality, no real-time requirement | DDPM / DPM-Solver++ | 100–1000 | Quality first, accept slowness |
| General high-quality generation | DPM-Solver++(2M) / UniPC | 15–25 | Balance quality and speed |
| Real-time interaction (<1 s) | ADD / DMD2 / LCM / InstaFlow | 1–4 | Requires a distilled model |
| Image editing / inversion | DDIM | 50 | Determinism ensures reproducibility and invertibility |
| Text-to-image (large CFG) | DPM-Solver++(2M) / UniPC | 15–20 | Stability at large guidance scales |
| Compute-limited devices | DeepCache + DDIM / DPM-Solver++ | 20–50 (with caching) | Training-free acceleration |
| DiT deployment | FORA / TeaCache + DPM-Solver++ | 20 (with caching) | Optimized for Transformers |
Sampling acceleration has grown from the original training-free ODE solvers into a field where numerical methods, schedule optimization, knowledge distillation, adversarial training, and architecture optimization advance in parallel. For real deployment, layered acceleration is recommended: first pick a higher-order solver (DPM-Solver++ or UniPC) to bring the step count down to 15–25, pairing it with an optimized schedule (such as AYS) if needed; if more speed is required, add a caching strategy (DeepCache / FORA / TeaCache); and for extreme speed, switch to a distilled model (ADD, DMD2, LCM) for 1–4 step generation.
Chapter 6: Frontier Paradigms: DiT, Consistency Models, and Flow Matching
Diffusion models are undergoing paradigm shifts in three directions: the backbone is moving from convolutional U-Nets to Transformers; sampling is moving from multi-step iteration to one-step or few-step generation; and the training objective is moving from score matching to the more general flow matching. This chapter discusses these three threads in turn: Diffusion Transformers (DiT) as a new generation of scalable backbones, Consistency Models as a theoretical framework for few-step generation, and Flow Matching and its variants as a mathematical paradigm that unifies generative modeling.
6.1 DiT: Diffusion Transformers
6.1.1 Architecture design and innovations
Diffusion Transformers (DiT) were proposed by Peebles and Xie (ICCV 2023)24. The core idea is to bring the Vision Transformer (ViT) into (latent) diffusion models, replacing the traditional U-Net backbone. DiT first “patchifies” the latent representation into a sequence of tokens, then passes it through standard Transformer blocks:
$$z \in \mathbb{R}^{I \times I \times C} \xrightarrow{\text{Patchify}} \text{tokens} \in \mathbb{R}^{(I/p)^2 \times d}$$| Symbol | Meaning |
|---|---|
| $I$ | Side length of the latent feature map (e.g., $I = 32$ for a 256×256 image after an 8× downsampling VAE) |
| $p$ | Patch size (the “/2” in DiT-XL/2 means $p = 2$) |
| $C$ | Number of latent channels (4 for SD-VAE) |
| $d$ | Transformer hidden dimension |
| $(I/p)^2$ | Sequence length (number of tokens) |
DiT compared several ways of injecting conditioning (in-context token, cross-attention, adaLN, adaLN-Zero), and adaLN-Zero worked best. It has two parts:
① Adaptive LayerNorm (adaLN): instead of fixed learnable affine parameters, LayerNorm regresses its scale and shift from the conditioning embedding $c$ (the sum of the timestep and class embeddings):
$$\mathrm{adaLN}(h, c) = \left( 1 + \gamma(c) \right) \odot \frac{h - \mu(h)}{\sqrt{\sigma^2(h) + \varepsilon}} + \beta(c)$$② Zero-initialized gated residual (Zero): the output of each residual branch is multiplied by a per-dimension gate $\alpha(c)$ regressed from $c$:
$$h \leftarrow h + \alpha(c) \odot \mathrm{Block}\left( \mathrm{adaLN}(h, c) \right)$$The linear layers that regress $\gamma, \beta, \alpha$ are initialized to zero, so at the start of training $\alpha = 0$ and every DiT block is the identity map, which makes training deep networks more stable.
| Symbol | Meaning |
|---|---|
| $\gamma(c), \beta(c)$ | Scale and shift regressed from the conditioning embedding (written scale and shift in code, applied as $1 + \gamma$) |
| $\alpha(c)$ | Per-dimension gate before the residual connection (gate in code), zero-initialized |
| $\mu(h), \sigma^2(h)$ | Mean and variance of the input feature $h$ over the feature dimension |
| $\varepsilon$ | Small constant preventing division by zero |
| $\odot$ | Elementwise multiplication |
The structure of a DiT block is as follows (the attention branch and the FFN branch each have a set of shift / scale / gate, for 6 parameter sets in total):
| |
| Parameter | Description | Typical value |
|---|---|---|
hidden_size | Hidden dimension | 384 (S) – 1152 (XL) |
num_heads | Number of attention heads | 6 (S) – 16 (XL) |
mlp_ratio | FFN expansion ratio | 4.0 |
num_blocks | Number of stacked DiT blocks | 12 (S/B) – 28 (XL) |
6.1.2 Scalability: bigger is better
DiT’s most important contribution is validating the scalability of Transformers for diffusion models. The paper measures model complexity by forward compute (Gflops) and finds that, whether by deepening / widening the network or reducing the patch size (increasing the number of tokens), higher Gflops means lower FID, with a strong correlation. The table below shows results for each model scale on ImageNet 256×256 with patch size $p = 2$24:
| Model | Parameters | FID-50K (400K training steps, no CFG) | Gflops |
|---|---|---|---|
| DiT-S/2 | 33M | 68.4 | 6.1 |
| DiT-B/2 | 130M | 43.5 | 23.0 |
| DiT-L/2 | 458M | 23.3 | 80.7 |
| DiT-XL/2 | 675M | 19.5 | 118.6 |
| DiT-XL/2 (7M training steps, CFG = 1.5) | 675M | 2.27 | 118.6 |
FID declines steadily as compute increases, and even the largest model, DiT-XL/2, showed no sign of saturating at 7M training steps. This predictable scalability makes DiT an ideal choice for large-scale training:
- Parameter count can scale smoothly into the billions (e.g., Hunyuan-DiT, Flux, the MMDiT variant of SD3, and video models such as Sora)
- Engineering-friendly: it can reuse mature infrastructure developed for large language models, such as data parallelism and model parallelism
- Unified spatiotemporal processing: video is just an extra sequence of patches along the time dimension (see §7.1)
6.1.3 DiT versus U-Net
graph LR
subgraph U-Net["U-Net architecture"]
U1["Encoder
multi-scale convolution"] --> U2["Bottleneck
attention"] --> U3["Decoder
upsampling + Skip"]
end
subgraph DiT["DiT architecture"]
D1["Patchify
linear projection"] --> D2["Transformer Blocks
×N"] --> D3["Unpatchify
reconstruct image"]
end
U1 -.->|"Trait: local inductive bias
multi-scale features"| U1
D2 -.->|"Trait: global attention
predictable scaling"| D2
| Dimension | U-Net | DiT |
|---|---|---|
| Inductive bias | Strong (locality, translation equivariance) | Weak (global self-attention) |
| Long-range dependencies | Mainly through attention in low-resolution layers | Direct interaction between any pair of tokens |
| Conditioning | Timestep via AdaGN / embedding addition, text via cross-attention | adaLN-Zero adaptive modulation (text models often add cross-attention or joint attention) |
| Scalability | Gains from scaling are less regular | Compute and FID strongly correlated, scaling predictable |
| Computational complexity | Convolution linear in pixel count | Attention $O(N^2)$ in token count |
| Modality unification | Video needs 3D / pseudo-3D convolutions | Unified spatiotemporal token sequence |
| Representative models | SD 1.5 / 2.1, SDXL | SD3, Flux, Sora, HunyuanDiT |
DiT’s limitation is the quadratic complexity of self-attention. At common image resolutions, though, this is acceptable: a 256×256 image yields $32 \times 32$ latents through the VAE, and with $p = 2$ that is only $16^2 = 256$ tokens; at 512×512 it is only $32^2 = 1024$. Follow-up work such as the Hourglass Diffusion Transformer (HDiT) introduces hierarchical (hourglass-shaped) structures that let the Transformer work efficiently directly in high-resolution pixel space.
6.2 Consistency Models
Consistency Models (CM)25 were proposed by Song and colleagues at OpenAI in 2023 (ICML 2023), with the goal of one-step or few-step generation through self-consistency constraints.
6.2.1 Core idea: the consistency function
Recall §1.7.3: the probability flow ODE connects each noisy point $x_T$ deterministically to a data point. All points on the same trajectory share the same “endpoint.” CM directly learns this “one-step jump to the endpoint” mapping, the consistency function $f: (x_t, t) \mapsto x_\varepsilon$:
$$f(x_t, t) = f(x_{t'}, t'), \quad \forall\, t, t' \in [\varepsilon, T] \text{ and } x_t, x_{t'} \text{ lie on the same PF-ODE trajectory}$$| Symbol | Meaning |
|---|---|
| $f$ | Consistency function |
| $x_t$ | Noisy sample at time $t$ (CM uses EDM’s setting, where $t$ is the noise standard deviation $\sigma$) |
| $\varepsilon$ | A very small positive number (e.g., 0.002); for numerical stability, the trajectory endpoint is taken at $t = \varepsilon$ rather than $0$ |
| $x_\varepsilon$ | Trajectory endpoint, approximately clean data |
| $T$ | Maximum noise level (e.g., 80) |
To satisfy the boundary condition $f(x_\varepsilon, \varepsilon) = x_\varepsilon$ automatically, CM uses a skip-connection parameterization:
$$f_\theta(x, t) = c_{\text{skip}}(t)\, x + c_{\text{out}}(t)\, F_\theta(x, t), \qquad c_{\text{skip}}(t) = \frac{\sigma_{\text{data}}^2}{(t - \varepsilon)^2 + \sigma_{\text{data}}^2}, \quad c_{\text{out}}(t) = \frac{\sigma_{\text{data}} (t - \varepsilon)}{\sqrt{\sigma_{\text{data}}^2 + t^2}}$$| Symbol | Meaning |
|---|---|
| $c_{\text{skip}}(t)$ | Skip coefficient, satisfying $c_{\text{skip}}(\varepsilon) = 1$; approaches 0 as noise increases |
| $c_{\text{out}}(t)$ | Output scaling coefficient, satisfying $c_{\text{out}}(\varepsilon) = 0$ |
| $F_\theta$ | Backbone neural network (freely parameterized) |
| $\sigma_{\text{data}}$ | Data standard deviation (often 0.5 for images), consistent with EDM preprocessing (§2.4.2) |
6.2.2 Consistency Distillation: distilling from a diffusion model
Consistency Distillation (CD) distills a pretrained diffusion model into a CM. The procedure: first noise the data to $t_{n+1}$ to get $x_{t_{n+1}}$, then use an ODE solver driven by the teacher diffusion model to take one small step back, obtaining the neighboring point $\hat{x}_{t_n}^{\phi}$ on the same trajectory, and finally require the student’s outputs at these two points to agree:
$$\mathcal{L}_{\text{CD}}(\theta, \theta^-) = \mathbb{E}\left[ \lambda(t_n)\, d\left( f_\theta(x_{t_{n+1}}, t_{n+1}),\; f_{\theta^-}(\hat{x}_{t_n}^{\phi}, t_n) \right) \right]$$| Symbol | Meaning |
|---|---|
| $\theta$ | Parameters of the consistency model being trained |
| $\theta^-$ | Target-network parameters, an exponential moving average (EMA) of $\theta$, with no gradient backpropagation (stop-gradient) |
| $\phi$ | Frozen pretrained diffusion model (teacher), used only for the ODE solver |
| $x_{t_{n+1}}$ | Sample obtained by forward-noising the data, $x + t_{n+1} z$ |
| $\hat{x}_{t_n}^{\phi}$ | Numerical solution obtained by taking one step from $x_{t_{n+1}}$ to $t_n$ with the teacher-driven ODE solver |
| $\lambda(t_n)$ | Time-dependent weight function |
| $d(\cdot, \cdot)$ | Distance metric (LPIPS works best in the paper) |
Intuition: as long as the outputs at neighboring points always agree, this “propagates” along the trajectory, so the outputs are consistent everywhere on it and are finally pinned to $x_\varepsilon$ by the boundary condition.
Key design choices in CD:
- ODE solver: uses a second-order Heun solver (§2.4.4), with $N = 18$ discretization steps working best on CIFAR-10
- Distance metric: LPIPS beats $\ell_1$ and $\ell_2$ because it better matches perceptual similarity
- EMA target network: build the target from an EMA of $\theta$, giving a smoother supervision signal (note: this “target network” is not the teacher diffusion model $\phi$)
- Performance: one-step FID 3.55 and two-step FID 2.93 on CIFAR-10, and one-step FID 6.20 on ImageNet 64×6425
6.2.3 Consistency Training: training from scratch
Consistency Training (CT) does not rely on a pretrained diffusion model; it trains a CM directly from data. Its key trick: use the same noise $z$ to noise the same data point $x$ to two adjacent noise levels, and treat these two points as “approximately neighboring points on the same trajectory” (this amounts to replacing the teacher’s score with an unbiased single-sample estimate):
$$\mathcal{L}_{\text{CT}}(\theta, \theta^-) = \mathbb{E}\left[ \lambda(t_n)\, d\left( f_\theta(x + t_{n+1} z,\; t_{n+1}),\; f_{\theta^-}(x + t_n z,\; t_n) \right) \right], \quad z \sim \mathcal{N}(0, \mathbf{I})$$| Symbol | Meaning |
|---|---|
| $t_n$ | The $n$-th noise level in the discrete time grid $\{t_n\}_{n=1}^{N}$ |
| $x$ | Training data |
| $z$ | The same Gaussian noise shared by both terms |
CT is sensitive to hyperparameters; the paper uses a curriculum schedule that progressively increases the discretization step count $N$ and adjusts the EMA decay rate over training. The original CT achieves a one-step FID of 8.70 on CIFAR-10, still behind CD.
6.2.4 Self-consistency and multi-step sampling
CM’s multi-step sampling achieves progressive refinement by alternating “denoise → re-noise → denoise again”:
- Start from $\hat{x}_T \sim \mathcal{N}(0, T^2 \mathbf{I})$ and get $x \leftarrow f_\theta(\hat{x}_T, T)$;
- For a chosen decreasing sequence of noise levels $\tau_1 > \tau_2 > \dots > \tau_{N-1}$, do the following in order:
Each round first re-noises the current estimate to a lower noise level, then maps back to data in one step, trading extra computation for quality.
| Sampling mode | Steps | Feature |
|---|---|---|
| Single-step | 1 | Fastest, slightly lower quality |
| Multi-step | 2–5 | Better quality, still much faster than regular diffusion sampling |
| Zero-shot editing | Variable | With masks or constraints, can directly handle inpainting, colorization, super-resolution, and similar tasks |
6.2.5 Improvements to consistency models: iCT and sCM
- iCT (Improved Consistency Training)26: Song and Dhariwal analyze the theory of CT and find that the EMA in the target network is an overlooked flaw, so they remove the EMA from the target network; they also replace LPIPS (which introduces evaluation bias) with the Pseudo-Huber loss from robust statistics, use log-normal noise sampling, and adopt a curriculum that “doubles the discretization steps every fixed number of iterations.” Without any distillation, one-step FID reaches 2.51 on CIFAR-10 and 3.25 on ImageNet 64×64, dropping further to 2.24 and 2.77 with two steps, both beating CD.
- sCM (continuous-time consistency model)27: discrete-time CMs introduce extra hyperparameters and discretization error, while earlier continuous-time versions were unstable to train. sCM proposes a simplified framework (TrigFlow) that unifies the diffusion-model and CM parameterizations, identifies the root cause of the instability, and improves the parameterization, network architecture, and training objective, scaling continuous-time CM training to 1.5 billion parameters (ImageNet 512×512). With only two sampling steps, FID reaches 2.06, 1.48, and 1.88 on CIFAR-10, ImageNet 64×64, and ImageNet 512×512, respectively.
6.2.6 Latent Consistency Models (LCM) and LCM-LoRA
LCM28 moves consistency distillation into the latent space of Stable Diffusion and folds CFG directly in: it treats the guided reverse process as an “augmented” probability flow ODE and has the student predict the solution of this ODE directly, so sampling no longer needs two forward passes. The paper reports that distilling a 768×768, 2–4 step LCM from a pretrained latent diffusion model with CFG takes only about 32 A100 GPU hours.
LCM-LoRA29 goes further by distilling only the LoRA parameters, using much less GPU memory, and finds that the resulting LoRA can serve as a universal acceleration module: plug it into various fine-tuned models or style LoRAs built on the same base model and get few-step (roughly 2–8 step) images with no further training. You can think of it as a “neural-network version of a PF-ODE solver.”
6.3 Consistency Trajectory Models (CTM)
Consistency Trajectory Models (CTM)30, proposed by Kim and colleagues (ICLR 2024), generalize CM’s “can only jump to the endpoint” into a mapping from any time on the trajectory to any earlier time:
$$G_\theta(x_t, t, s) \approx x_s, \quad \forall\, s \le t$$| Symbol | Meaning |
|---|---|
| $t$ | Starting time (the noisier side) |
| $s$ | Target time (the cleaner side) |
| $x_s$ | Solution obtained by evolving from $x_t$ to $s$ along the PF-ODE |
CTM’s network parameterization borrows the first-order form of the ODE’s exact solution (a DDIM-like interpolation):
$$G_\theta(x_t, t, s) = \frac{s}{t}\, x_t + \left( 1 - \frac{s}{t} \right) g_\theta(x_t, t, s)$$| Symbol | Meaning |
|---|---|
| $g_\theta$ | Main neural network |
| $\frac{s}{t},\, 1 - \frac{s}{t}$ | Interpolation coefficients, ensuring $G_\theta(x_t, t, t) = x_t$ (the identity map) when $s = t$ |
Two special cases are instructive: at $s = 0$, $G_\theta(x_t, t, 0) = g_\theta(x_t, t, 0)$, which is exactly the CM consistency function; and as $s \to t$, $g_\theta(x_t, t, t)$ approximates the denoiser $\mathbb{E}[x_0 \mid x_t]$, from which the score follows directly via Tweedie’s formula (§1.6). So the same network can both take big jumps and output the score.
Advantages of CTM:
- Flexibility: can jump from any noise level to any cleaner level, supporting mixed deterministic and stochastic sampling with “long jumps + re-noising,” with quality improving monotonically as the compute budget grows (avoiding the quality degradation that can occur with CM multi-step sampling)
- Unified framework: covers both diffusion models (small steps) and consistency models (jump to the endpoint) as special cases
- Training: combines a trajectory-consistency loss, a denoising score matching (DSM) loss, and an adversarial loss
- Performance: one-step FID reaches 1.73 on CIFAR-10 and 1.92 on ImageNet 64×6430
- Controllable generation: because it can output the score, it can directly reuse guidance and conditional-generation methods from the diffusion community, and can also compute likelihoods
6.4 Flow Matching
Flow Matching (FM)31, proposed by Lipman and colleagues (ICLR 2023), provides a simulation-free training framework for continuous normalizing flows (CNFs): training does not require solving an ODE, only stochastic gradient descent on a regression objective.
Note
This section and §6.5 follow the convention of the original Flow Matching / Rectified Flow papers: $t = 0$ is noise and $t = 1$ is data, with the subscript $x_0$ denoting noise and $x_1$ data. This is exactly the opposite of the “$x_0$ is data” convention in the diffusion-model chapters 1–2 and 5, so take care when reading. §6.4.3 gives the conversion between the two conventions, while MeanFlow in §6.6 switches back to “$t = 0$ is data.”
6.4.1 Conditional Flow Matching (CFM)
FM learns a time-dependent velocity field that continuously pushes the noise distribution $p_0$ to the data distribution $p_1$ through an ODE:
$$\frac{\mathrm{d}x(t)}{\mathrm{d}t} = v_\theta(x(t), t), \quad x(0) \sim p_0, \quad x(1) \sim p_1$$| Symbol | Meaning |
|---|---|
| $v_\theta(x, t)$ | Parameterized velocity field (neural network) |
| $p_0$ | Source distribution (usually a standard Gaussian) |
| $p_1$ | Target data distribution |
| $t \in [0, 1]$ | Continuous time variable |
Ideally, we want $v_\theta$ to regress to the true velocity field $u_t(x)$ that generates the marginal probability path $p_t$: $\mathcal{L}_{\text{FM}} = \mathbb{E}_{t, x \sim p_t}\left[ \lVert v_\theta(x, t) - u_t(x) \rVert^2 \right]$. But $u_t(x)$ itself is intractable. The key of Conditional Flow Matching (CFM) is to instead regress to a tractable conditional velocity field $u_t(x \mid z)$:
$$\mathcal{L}_{\text{CFM}}(\theta) = \mathbb{E}_{t \sim \mathcal{U}[0,1],\, z \sim q(z),\, x \sim p_t(\cdot \mid z)} \left[ \left\lVert v_\theta(x, t) - u_t(x \mid z) \right\rVert^2 \right]$$| Symbol | Meaning |
|---|---|
| $z$ | Conditioning variable (e.g., a data point $x_1$, or a noise–data pair $(x_0, x_1)$) |
| $q(z)$ | Distribution of the conditioning variable |
| $u_t(x \mid z)$ | Conditional velocity field with a known analytic form |
| $p_t(x \mid z)$ | Conditional probability path |
Why is this substitution valid? The true marginal velocity field is exactly the posterior mean of the conditional velocity fields, $u_t(x) = \mathbb{E}\left[ u_t(x \mid z) \mid x_t = x \right]$, from which one can show $\nabla_\theta \mathcal{L}_{\text{FM}} = \nabla_\theta \mathcal{L}_{\text{CFM}}$. In other words, the two objectives have the same optimum: regressing “each individual path” is equivalent to regressing “the flow obtained by superposing all paths” — the same idea as denoising score matching replacing the marginal score with the conditional score (§1.5.3).
For a Gaussian conditional path $p_t(x \mid x_1) = \mathcal{N}\left(x; \mu_t(x_1), \sigma_t(x_1)^2 \mathbf{I}\right)$, the conditional velocity field has a general formula:
$$u_t(x \mid x_1) = \frac{\sigma_t'(x_1)}{\sigma_t(x_1)} \left( x - \mu_t(x_1) \right) + \mu_t'(x_1)$$where $'$ denotes differentiation with respect to $t$. The OT path recommended by Lipman and colleagues takes $\mu_t = t x_1$ and $\sigma_t = 1 - (1 - \sigma_{\min}) t$, giving the sample and regression target:
$$x_t = \left( 1 - (1 - \sigma_{\min}) t \right) x_0 + t x_1, \qquad u_t = x_1 - (1 - \sigma_{\min}) x_0$$That is, it travels from noise to data along a straight line at constant speed ($\sigma_{\min}$ is a very small number, e.g., $10^{-4}$). The paper finds that this path trains faster, samples faster, and generalizes better than diffusion paths31.
6.4.2 Minibatch Optimal Transport Flow Matching (OT-CFM)
The CFM above pairs noise and data independently (coupling $q = p_0 \otimes p_1$): each individual path is straight, but different paths cross each other, so the marginal flow obtained by superposing them is still curved. OT-CFM32 introduces minibatch Optimal Transport: it first finds a low-transport-cost pairing of noise and data within each minibatch, then performs CFM on those pairs:
$$\pi^* = \arg\min_{\pi \in \Pi(p_0, p_1)} \mathbb{E}_{(x_0, x_1) \sim \pi} \left[ c(x_0, x_1) \right]$$| Symbol | Meaning |
|---|---|
| $\pi$ | Joint noise–data distribution (coupling), with marginals $p_0$ and $p_1$ |
| $c(x_0, x_1)$ | Transport cost (usually $\lVert x_0 - x_1 \rVert^2$) |
| $\pi^*$ | Optimal transport plan (in practice only approximated within each minibatch) |
Given a pair $(x_0, x_1) \sim \pi^*$, the conditional path and velocity are (the independent-coupling version I-CFM uses the same straight path; the only difference is how the pairs are formed):
$$p_t(x \mid x_0, x_1) = \mathcal{N}\left(x;\, (1 - t) x_0 + t x_1,\, \sigma^2 \mathbf{I}\right), \qquad u_t(x \mid x_0, x_1) = x_1 - x_0$$| Symbol | Meaning |
|---|---|
| $x_0 \sim p_0$ | Noise sample |
| $x_1 \sim p_1$ | Data sample |
| $\sigma$ | Path width (usually very small or zero) |
| $u_t$ | Constant straight-line velocity |
Advantages of OT pairing:
- Straighter marginal flow: pairing reduces crossings and avoids the curvature of marginal trajectories seen with independent pairing
- Faster sampling: a straighter flow needs fewer ODE integration steps
- More stable training: the regression target has lower variance. CFM also does not require the source distribution to be Gaussian, so it can be used to convert between any two distributions (e.g., image translation, single-cell trajectory inference)
6.4.3 A unified view of Flow Matching and diffusion models
Diffusion models can be seen as a special case of FM: Lipman and colleagues show that the marginal paths for VP / VE diffusion all belong to the family of Gaussian probability paths allowed by FM, just not straight. FM’s advantage is flexibility: you can use any non-diffusion path, including straight (OT) paths and geodesics on Riemannian manifolds. Stochastic Interpolants33 provides another framework unifying flows and diffusion from the interpolation viewpoint.
The two parameterizations are interchangeable. Write the interpolation in diffusion convention as $x_t = \alpha_t x_0 + \sigma_t \epsilon$ ($x_0$ is data); its velocity is $v_t = \dot{\alpha}_t x_0 + \dot{\sigma}_t \epsilon$ (dots denote differentiation with respect to $t$). So as long as the network gives $\hat{\epsilon}$ (or $\hat{x}_0$), you can compute $\hat{v}$, and vice versa. Take the Rectified Flow interpolation $\alpha_t = 1 - t$, $\sigma_t = t$ used by models such as SD3:
$$v_t = \epsilon - x_0, \qquad \hat{x}_0 = x_t - t\, \hat{v}, \qquad \hat{\epsilon} = x_t + (1 - t)\, \hat{v}, \qquad \nabla_x \log p_t(x_t) \approx -\frac{\hat{\epsilon}}{t}$$(Here $t = 1$ is noise, related to the FM convention at the start of this section by $t \mapsto 1 - t$.) This shows that “predicting noise,” “predicting data,” and “predicting velocity” are essentially different output heads of the same model, and tools such as CFG and DDIM-style solvers can be migrated between them.
In practice, the sampling distribution of timesteps matters a lot. SD334 compared these systematically and found that logit-normal sampling of $t$ (which puts more training samples at intermediate noise levels) works best:
$$\pi_{\text{ln}}(t; m, s) = \frac{1}{s \sqrt{2\pi}} \cdot \frac{1}{t (1 - t)} \exp\left( -\frac{\left( \operatorname{logit}(t) - m \right)^2}{2 s^2} \right), \qquad \operatorname{logit}(t) = \log \frac{t}{1 - t}$$
graph TD
A[Generative modeling frameworks] --> B[Diffusion models
Score Matching]
A --> C[Flow Matching
Velocity Regression]
B --> D[VP-SDE/VE-SDE
curved paths]
C --> E[OT path/OT-CFM
straighter paths]
C --> F[General flows
arbitrary paths]
D --> G[Usually need more steps]
E --> H[Typically 10-50 steps]
F --> I[Domain-tailored paths]
6.5 Rectified Flow
Rectified Flow (RF)35, proposed by Liu, Gong, and Liu (ICLR 2023), was developed independently at about the same time as FM and is closely related in spirit: it also trains a velocity field with the straight-line interpolation $x_t = (1 - t) x_0 + t x_1$. Its distinctive feature is iterative Reflow: gradually straightening the curved noise–data coupling by “re-pairing.”
6.5.1 The Reflow algorithm
The first round of Rectified Flow training (1-RF) is the same as FM, using independent coupling. Subsequent Reflow straightens the flow through “simulate → re-pair → retrain”:
- Simulate: starting from noise $x_0 \sim p_0$, solve the ODE with the current flow $v^{(k)}$ to get $\hat{x}_1 = \mathrm{ODESolve}\left(v^{(k)}, x_0\right)$
- Re-pair: use $\left\{ \left(x_0, \hat{x}_1\right) \right\}$ as the new training pairs (the pairing is now given deterministically by the ODE and no longer crosses)
- Retrain: train a new flow $v^{(k+1)}$ on these pairs with the same straight-line interpolation objective
The paper proves that each rectification yields a new deterministic coupling and that, for any convex cost function, the transport cost does not increase35. After repeated Reflow, the paths get straighter and straighter, so coarse discretization (even a single Euler step) simulates them accurately:
$$x_1^{(k)} \approx x_0 + v^{(k)}\left(x_0, 0\right)$$| Symbol | Meaning |
|---|---|
| $v^{(k)}$ | Velocity field after the $k$-th round of Reflow |
| $x_0$ | Initial noise |
| $x_1^{(k)}$ | Endpoint generated by the flow at round $k$ |
In practice, one usually does 1–2 rounds of Reflow and then distills the flow into a one-step model, because each Reflow round needs a large number of generated pairs, which is costly, and errors accumulate from round to round.
6.5.2 InstaFlow: Reflow for text-to-image
InstaFlow14 applies Reflow to large-scale text-conditioned generation. Its key contributions:
- performs Reflow in the latent space of Stable Diffusion, proposing a “text-conditioned Reflow + distillation” one-step generation pipeline
- finds that Reflow is essential to improving the pairing between noise and images, and thus to the success of the subsequent distillation
- one-step FID 23.3 on MS COCO 2017-5k (22.4 when scaled to 1.7B parameters); one-step FID 13.1 on MS COCO 2014-30k, with about 0.09 s per sample
- training cost of about 199 A100 GPU days14
6.5.3 PeRFlow: piecewise rectified flow
PeRFlow (Piecewise Rectified Flow)36 splits the time axis $[0, 1]$ into $K$ windows $[t_{k-1}, t_k]$ and does Reflow separately inside each window: first solve the ODE with the pretrained teacher within the window to get the points $x_{t_{k-1}}$ and $x_{t_k}$ at its two ends, then have the student regress, within that window, the straight-line velocity connecting the two points:
$$v(x_t, t) = \frac{x_{t_k} - x_{t_{k-1}}}{t_k - t_{k-1}}, \quad t \in [t_{k-1}, t_k]$$| Symbol | Meaning |
|---|---|
| $[t_{k-1}, t_k]$ | The $k$-th time window |
| $x_{t_{k-1}}, x_{t_k}$ | Points given by the teacher ODE at the two ends of the window |
Intuition: straightening an entire curved trajectory in one go is hard, but approximating it with a few line segments is much easier; each segment needs only one step, so $K$ segments need only $K$ steps.
Through a specialized parameterization, PeRFlow inherits the knowledge of the pretrained diffusion model, so it converges quickly and transfers well, serving as a universal plug-and-play accelerator: the trained acceleration weights work with many workflows built on the same pretrained model (such as ControlNet, Wonder3D, etc.) without retraining36.
6.5.4 SlimFlow: compressing steps and model together
SlimFlow37 compresses both the number of inference steps and the model size. The Rectified Flow framework relies on two operations, Reflow and distillation, and when the student is much smaller than the teacher two new problems arise: a mismatch between the initializations of the large teacher and small student during Reflow, and poor results from naive distillation on the small student. SlimFlow proposes, respectively:
- Annealing Reflow: performs Reflow in an annealed manner, easing the initialization mismatch between the large teacher and small student
- Flow-Guided Distillation: guides distillation of the small model more effectively within the Rectified Flow framework
- Result: on CIFAR-10, a model with 15.7 million parameters and one-step FID 5.02, better than the previous best one-step model (FID 6.47, 19.4M parameters)
6.6 MeanFlow, AlphaFlow, and Decoupled MeanFlow
Note
This section follows the convention of the original MeanFlow paper: $z_t = (1 - t)\, x + t\, \epsilon$, i.e., $t = 0$ is data and $t = 1$ is noise (the opposite of §6.4–§6.5, and the same as diffusion models). The conditional velocity is then $v_t = \epsilon - x$, and sampling runs from $t = 1$ to $t = 0$.
6.6.1 MeanFlow: one-step generation with average velocity
The core insight of MeanFlow38 (Geng et al., 2025) is this: Flow Matching learns the instantaneous velocity $v(z_t, t)$, and completing the trajectory requires integrating over many small steps; if you instead learn the average velocity over an interval directly, you can jump across any interval in one step. Average velocity is defined as displacement divided by time:
$$u(z_t, r, t) = \frac{1}{t - r} \int_{r}^{t} v(z_\tau, \tau)\, \mathrm{d}\tau, \qquad z_r = z_t - (t - r)\, u(z_t, r, t)$$| Symbol | Meaning |
|---|---|
| $u(z_t, r, t)$ | Average velocity field from $r$ to $t$ ($r \le t$) |
| $v(z_\tau, \tau)$ | Instantaneous velocity field |
| $z_t$ | State at time $t$ |
| $z_r$ | State at time $r$ (closer to data) |
With the average velocity, one-step generation is simply $z_0 = z_1 - u_\theta(z_1, 0, 1)$.
But the integral itself is intractable — how do we get a training target? Taking the total derivative of $(t - r)\, u = \int_r^t v\, \mathrm{d}\tau$ with respect to $t$ on both sides gives the MeanFlow identity:
$$u(z_t, r, t) = v(z_t, t) - (t - r) \frac{\mathrm{d}}{\mathrm{d}t} u(z_t, r, t), \qquad \frac{\mathrm{d}}{\mathrm{d}t} u = v(z_t, t)\, \partial_z u + \partial_t u$$It expresses the “average velocity” in terms of “instantaneous velocity + the time derivative of the average velocity,” and that derivative can be computed efficiently with one Jacobian-vector product (JVP). The training target is therefore:
$$\mathcal{L}_{\text{MF}}(\theta) = \mathbb{E}_{t, r, x, \epsilon} \left[ \left\lVert u_\theta(z_t, r, t) - \mathrm{sg}\left( u_{\text{tgt}} \right) \right\rVert_2^2 \right], \qquad u_{\text{tgt}} = v_t - (t - r) \left( v_t\, \partial_z u_\theta + \partial_t u_\theta \right)$$where $v_t = \epsilon - x$ is the conditional velocity and $\mathrm{sg}(\cdot)$ denotes stop-gradient. When $r = t$, the target reduces to ordinary Flow Matching ($u = v$); MeanFlow mixes in a fraction of $r = t$ samples during training.
MeanFlow needs no pretraining, distillation, or curriculum learning — training from scratch suffices: it reaches one-step (1-NFE) FID 3.43 on ImageNet 256×256, significantly better than earlier one-step diffusion / flow models and substantially narrowing the gap between one-step and multi-step models38.
6.6.2 AlphaFlow: understanding and improving MeanFlow
AlphaFlow ($\alpha$-Flow)39 analyzes MeanFlow’s training dynamics in depth and finds that its objective naturally decomposes into two parts:
- Trajectory Flow Matching: makes the model match the true velocity
- Trajectory Consistency: makes the model satisfy self-consistency
Gradient analysis shows that these two terms are strongly negatively correlated, causing optimization conflict and slow convergence. The paper also finds that the “$r = t$ boundary case” supervision in MeanFlow actually acts as a surrogate loss for trajectory flow matching, easing the conflict, but it consumes more than 75% of MeanFlow’s training compute. To address this, $\alpha$-Flow proposes a unified family of objectives:
$$\mathcal{L}_\alpha(\theta) = \mathbb{E}_{t, r, z_t} \left[ \alpha^{-1} \left\lVert u_\theta(z_t, r, t) - \left( \alpha\, \tilde{v}_{s,t} + (1 - \alpha)\, u_{\theta^-}(z_s, r, s) \right) \right\rVert_2^2 \right], \qquad s = \alpha r + (1 - \alpha) t$$| Symbol | Meaning |
|---|---|
| $\alpha \in (0, 1]$ | Interpolation parameter determining where the intermediate time $s$ lies between $r$ and $t$ |
| $\tilde{v}_{s,t}$ | Conditional velocity over the short segment from $t$ to $s$ ($\epsilon - x$ in the implementation) |
| $z_s$ | Point reached from $z_t$ along that velocity to $s$: $z_t - (t - s)\, \tilde{v}_{s,t}$ |
| $u_{\theta^-}$ | Stop-gradient model output |
This family unifies several methods: $\alpha = 1$ gives trajectory flow matching, $\alpha = \frac{1}{2}$ corresponds to the Shortcut Model, and as $\alpha \to 0$ the gradient matches MeanFlow’s. AlphaFlow accordingly uses curriculum learning: start with $\alpha = 1$ early in training, then smoothly anneal to $\alpha \to 0$ (MeanFlow), decoupling the two conflicting objectives in time:
- on class-conditional ImageNet 256×256 with the vanilla DiT backbone trained from scratch, it consistently beats MeanFlow at every scale
- the largest model, $\alpha$-Flow-XL/2+, reaches one-step FID 2.58 and two-step FID 2.1539
- it depends less on the $r = t$ boundary-case supervision and allocates training compute more efficiently
6.6.3 Decoupled MeanFlow: turning a flow model into a flow map
Models that learn average velocity (also called flow maps) usually need architectural changes to accept both times $r$ and $t$, which makes it hard to reuse already-pretrained flow models. Decoupled MeanFlow (DMF)40 proposes a decoding strategy that requires no architectural changes:
The core idea is to treat the first few blocks of a DiT as an “encoder” (still conditioned on the current time $t$), treat the last few blocks as a “decoder,” and switch the decoder’s time conditioning from the current time $t$ to the target time $r$:
$$h_{\text{out}} = \mathrm{Decoder}\left( h_{\text{enc}}, r \right), \qquad h_{\text{enc}} = \mathrm{Encoder}\left( z_t, t \right)$$| Symbol | Meaning |
|---|---|
| $h_{\text{enc}}$ | Intermediate representation output by the encoder (the early blocks) |
| $r$ | Target time (not the current time $t$) |
| $\mathrm{Decoder}$ | The last few blocks of the original model |
This way, a pretrained flow model can be turned directly into a flow map. The paper also finds that training a flow model first and then converting is more efficient and works better than training a flow map from scratch. DMF uses a two-stage training strategy, “Flow Matching warmup + MeanFlow fine-tuning”:
| |
| Parameter | Meaning | Notes |
|---|---|---|
warmup_steps | Warmup steps | First build a good flow field with FM (or initialize directly from a pretrained flow model) |
use_cauchy | Cauchy loss | The MeanFlow loss has high variance and is sensitive to outliers; the Cauchy (Lorentzian) loss is nearly linear near zero and suppresses large outliers |
phi(t, r) | Adaptive weight | A loss-weighting function learned over $(t, r)$ (maximum likelihood by modeling residuals as a Cauchy distribution) |
r sampling | Target time | Take the max / min of two logit-normal samples to get $(t, r)$, and adjust the distribution so $t$ and $r$ are farther apart and $r$ is closer to 0, strengthening one-step generation |
DMF’s performance40:
- ImageNet 256×256: one-step FID 2.16, 4-step FID 1.51
- ImageNet 512×512: one-step FID 2.12, 4-step FID 1.68
- 4-step quality nearly matches multi-step flow models, with over 100× faster inference
6.7 Comparison and summary of frontier paradigms
graph LR
subgraph Training["Evolution of training paradigms"]
T1["Diffusion models
Score Matching"] --> T2["Flow Matching
Velocity Regression"]
T2 --> T3["Rectified Flow
Reflow straightening"]
T3 --> T4["MeanFlow
average velocity field"]
T4 --> T5["AlphaFlow/DMF
optimization & compatibility"]
end
subgraph Architecture["Evolution of architectures"]
A1["U-Net
convolutional backbone"] --> A2["DiT
Transformer backbone"]
A2 --> A3["HDiT/MMDiT
hierarchical/multimodal"]
end
subgraph Sampling["Sampling acceleration"]
S1["1000 steps
DDPM"] --> S2["50 steps
DDIM"]
S2 --> S3["10-20 steps
DPM-Solver++"]
S3 --> S4["1-4 steps
CM/ADD/DMD/DMF"]
end
| Paradigm | Core idea | Training | Sampling steps | Representative models / methods | Strengths | Limitations |
|---|---|---|---|---|---|---|
| DiT | Transformer replaces U-Net | Standard diffusion / flow training | Unchanged | SD3, Flux, Sora | Scalable, global modeling | Quadratic attention complexity |
| Consistency Models | Trajectory self-consistency | Distillation or from scratch | 1–4 | CM, iCT, LCM, sCM | One-step / few-step, supports zero-shot editing | Training-sensitive, ceiling on few-step quality |
| CTM | Jumps between arbitrary times | Distillation + DSM + adversarial | 1–arbitrary | CTM | Flexible, can output scores | Complex training |
| Flow Matching | Velocity field regression | Simulation-free training | 10–50 | SD3, Flux | Efficient training, flexible paths | Still needs multi-step sampling |
| OT-CFM | Minibatch optimal transport pairing | Simulation-free training | Fewer than I-CFM | FM + minibatch OT | Straighter marginal flow, lower variance | Minibatch OT approximation limited in high dimensions |
| Rectified Flow | Iterative Reflow straightening | Reflow + distillation | 1–4 | InstaFlow, PeRFlow | Plug-and-play acceleration | Needs many generated pairs |
| Distribution Matching Distillation | Align distributions, not trajectories | Distillation (dual score networks) | 1–4 | DMD, DMD2 | High one-step quality | Needs extra training of the fake score network |
| MeanFlow | Average velocity field | From scratch | 1–2 | MeanFlow | One-step generation without distillation | High training variance, needs JVP |
| AlphaFlow | Curriculum learning decouples conflicting objectives | From scratch | 1–2 | $\alpha$-Flow | Faster convergence, higher quality | Requires curriculum design |
| Decoupled MeanFlow | Flow model to flow map | Pretraining + fine-tuning | 1–4 | DMF | Compatible with pretrained flow models | Needs two-stage training |
6.8 Code practice: a minimal Flow Matching implementation
Below is a minimal implementation of conditional flow matching with the OT path (straight-line interpolation), using the time convention of §6.4 ($x_0$ is noise, $x_1$ is data):
| |
| Parameter | Description | Typical value |
|---|---|---|
sigma | Path randomness | 0.0 (deterministic straight line) or 0.001 |
num_steps | Number of ODE integration steps | 20–50 (can be distilled to 1–4) |
backbone | Velocity-field network | DiT or U-Net |
cond | Conditioning information | Text embeddings, class embeddings, etc. |
Note
In the example,
forwardtakes only(x_t, t), whilecompute_lossandsamplepass incond; in real use, add the conditioning argument according to the backbone’s interface. In actual training, $t$ is often sampled from a non-uniform distribution such as logit-normal (§6.4.3).
References
Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. NeurIPS, 2020. ↩︎
Song J, Meng C, Ermon S. Denoising diffusion implicit models. ICLR, 2021. ↩︎
Lu C, Zhou Y, Bao F, Chen J, Li C, Zhu J. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS, 2022. ↩︎ ↩︎
Lu C, Zhou Y, Bao F, Chen J, Li C, Zhu J. DPM-Solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv:2211.01095, 2022 (later published in Machine Intelligence Research). ↩︎ ↩︎
Saharia C, Chan W, Saxena S, et al. Photorealistic text-to-image diffusion models with deep language understanding (Imagen). NeurIPS, 2022. ↩︎
Zhao W, Bai L, Rao Y, Zhou J, Lu J. UniPC: A unified predictor-corrector framework for fast sampling of diffusion models. NeurIPS, 2023. ↩︎ ↩︎
Zhang Q, Chen Y. Fast sampling of diffusion models with exponential integrator. ICLR, 2023. ↩︎ ↩︎
Sabour A, Fidler S, Kreis K. Align your steps: Optimizing sampling schedules in diffusion models. ICML, 2024. ↩︎ ↩︎
Bao F, Li C, Zhu J, Zhang B. Analytic-DPM: An analytic estimate of the optimal reverse variance in diffusion probabilistic models. ICLR, 2022 (Outstanding Paper Award). ↩︎ ↩︎
Salimans T, Ho J. Progressive distillation for fast sampling of diffusion models. ICLR, 2022. ↩︎ ↩︎
Meng C, Rombach R, Gao R, et al. On distillation of guided diffusion models. CVPR, 2023. ↩︎
Sauer A, Lorenz D, Blattmann A, Rombach R. Adversarial diffusion distillation. ECCV, 2024. ↩︎ ↩︎
Sauer A, Boesel F, Dockhorn T, Blattmann A, Esser P, Rombach R. Fast high-resolution image synthesis with latent adversarial diffusion distillation. SIGGRAPH Asia, 2024. ↩︎
Liu X, Zhang X, Ma J, Peng J, Liu Q. InstaFlow: One step is enough for high-quality diffusion-based text-to-image generation. ICLR, 2024. ↩︎ ↩︎ ↩︎
Yin T, Gharbi M, Zhang R, Shechtman E, Durand F, Freeman W T, Park T. One-step diffusion with distribution matching distillation. CVPR, 2024. ↩︎ ↩︎
Yin T, Gharbi M, Park T, Zhang R, Shechtman E, Durand F, Freeman W T. Improved distribution matching distillation for fast image synthesis. NeurIPS, 2024. ↩︎
Bolya D, Hoffman J. Token merging for fast stable diffusion. CVPR Workshops, 2023. ↩︎
Krause F, Phan T, Gui M, Baumann S A, Hu V T, Ommer B. TREAD: Token routing for efficient architecture-agnostic diffusion training. arXiv:2501.04765, 2025. ↩︎
Ma X, Fang G, Wang X. DeepCache: Accelerating diffusion models for free. CVPR, 2024 (arXiv:2312.00858). ↩︎ ↩︎
Selvaraju P, Ding T, Chen T, Zharkov I, Liang L. FORA: Fast-forward caching in diffusion transformer acceleration. arXiv:2407.01425, 2024. ↩︎ ↩︎
Chen P, Shen M, Ye P, et al. $\Delta$-DiT: A training-free acceleration method tailored for diffusion transformers. arXiv:2406.01125, 2024. ↩︎
Liu F, Zhang S, Wang X, et al. Timestep embedding tells: It’s time to cache for video diffusion model (TeaCache). CVPR, 2025. ↩︎
Liu J, Zou C, Lyu Y, Chen J, Zhang L. From reusing to forecasting: Accelerating diffusion models with TaylorSeers. ICCV, 2025. ↩︎
Peebles W, Xie S. Scalable diffusion models with transformers. ICCV, 2023. ↩︎ ↩︎
Song Y, Dhariwal P, Chen M, Sutskever I. Consistency models. ICML, 2023. ↩︎ ↩︎
Song Y, Dhariwal P. Improved techniques for training consistency models. ICLR, 2024. ↩︎
Lu C, Song Y. Simplifying, stabilizing and scaling continuous-time consistency models. ICLR, 2025. ↩︎
Luo S, Tan Y, Huang L, Li J, Zhao H. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv:2310.04378, 2023. ↩︎
Luo S, Tan Y, Patil S, et al. LCM-LoRA: A universal Stable-Diffusion acceleration module. arXiv:2311.05556, 2023. ↩︎
Kim D, Lai C-H, Liao W-H, et al. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. ICLR, 2024. ↩︎ ↩︎
Lipman Y, Chen R T Q, Ben-Hamu H, Nickel M, Le M. Flow matching for generative modeling. ICLR, 2023. ↩︎ ↩︎
Tong A, Fatras K, Malkin N, et al. Improving and generalizing flow-based generative models with minibatch optimal transport. TMLR, 2024. ↩︎
Albergo M S, Vanden-Eijnden E. Building normalizing flows with stochastic interpolants. ICLR, 2023. ↩︎
Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis (Stable Diffusion 3). ICML, 2024. ↩︎
Liu X, Gong C, Liu Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR, 2023. ↩︎ ↩︎
Yan H, Liu X, Pan J, Liew J H, Liu Q, Feng J. PeRFlow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv:2405.07510, 2024. ↩︎ ↩︎
Zhu Y, Liu X, Liu Q. SlimFlow: Training smaller one-step diffusion models with rectified flow. ECCV, 2024. ↩︎
Geng Z, Deng M, Bai X, Kolter J Z, He K. Mean flows for one-step generative modeling. arXiv:2505.13447, 2025. ↩︎ ↩︎
Zhang H, Siarohin A, Menapace W, Vasilkovsky M, Tulyakov S, Qu Q, Skorokhodov I. AlphaFlow: Understanding and improving MeanFlow models. arXiv:2510.20771, 2025. ↩︎ ↩︎
Lee K, Yu S, Shin J. Decoupled MeanFlow: Turning flow models into flow maps for accelerated sampling. arXiv:2510.24474, 2025. ↩︎ ↩︎
