A Survey of Diffusion Models
Date: 2026-07-02 (revised 2026-11-19: checked paper years and sources, added missing branches) Search strategy: coarse-to-fine, with 5 searches covering the big picture, structure, and frontier work
This article is the “map” of the whole diffusion model note series: it opens with a one-page list of the main technical branches and representative work, then points you to the corresponding chapter notes for the underlying principles (see Section 6, “Reading Guide”). Years generally refer to the conference or journal year in which the paper was formally published, with the conference name in parentheses.
1. Big-Picture Findings
Diffusion models have become the dominant paradigm for generative AI in deep learning, with breakthrough results in images, video, 3D, and audio. The core idea is to first define a forward process that gradually adds noise to the data until it turns into pure noise, then train a neural network to learn the reverse denoising at each step. At generation time, you start from pure noise and “denoise” step by step to get a new sample.
Mathematically, diffusion models have three mutually equivalent viewpoints, and nearly every branch that follows plays with these three viewpoints:
| Viewpoint | Core object | Representative work |
|---|---|---|
| Variational inference (discrete time) | Evidence lower bound (ELBO), noise prediction | DDPM |
| Score matching | Score of the data distribution $\nabla_{\mathbf{x}} \log p(\mathbf{x})$ | NCSN |
| Stochastic / ordinary differential equation (continuous time) | Reverse SDE, probability flow ODE | Score SDE, EDM, Flow Matching |
Key survey papers:
- Yang et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications”, ACM Computing Surveys, 2023
- Cao et al., “A Survey on Generative Diffusion Models”, IEEE TKDE, 2024
- Chen et al., “Comprehensive exploration of diffusion models in image generation: a survey”, Artificial Intelligence Review, 2025
- Ma et al., “Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices”, IEEE TPAMI, 2025
- Ma et al., “Video diffusion generation: comprehensive review and open problems”, Artificial Intelligence Review, 2025
2. Structural Map: Main Technical Branches
2.1 Theoretical Foundations
- Score Matching: Hyvärinen (JMLR 2005), Vincent (Neural Computation 2011) — estimates the score while bypassing the normalization constant; denoising score matching (DSM) is the training foundation of modern diffusion models
- Sliced Score Matching: Song et al. (UAI 2019) — uses random projections to estimate the trace term in score matching
- NCSN (Score-based Generative Models): Song & Ermon (NeurIPS 2019) — Noise Conditional Score Network + annealed Langevin sampling; improved version NCSNv2 (NeurIPS 2020)
- Unified SDE/ODE framework: Song et al. (ICLR 2021) — continuous-time diffusion that unifies VP/VE-SDE, and introduces the reverse SDE, probability flow ODE, and predictor-corrector sampling
- VDM (Variational Diffusion Models): Kingma et al. (NeurIPS 2021) — characterizes the noise schedule by signal-to-noise ratio (SNR), and proves that the continuous-time ELBO depends only on the SNR endpoints
- EDM: Karras et al. (NeurIPS 2022) — clarifies the diffusion design space (preconditioning, loss weighting, noise distribution, sampler)
2.2 Discrete Probabilistic Diffusion
- Proposal of the diffusion probabilistic model: Sohl-Dickstein et al. (ICML 2015) — the earliest framework, inspired by non-equilibrium thermodynamics
- DDPM: Ho et al. (NeurIPS 2020) — Denoising Diffusion Probabilistic Model, the foundational work
- DDIM: Song et al. (ICLR 2021) — a non-Markovian deterministic implicit model that allows few-step sampling without retraining
- Improved DDPM: Nichol & Dhariwal (ICML 2021) — cosine noise schedule and learned variance
- ADM (Ablated Diffusion Model): Dhariwal & Nichol (NeurIPS 2021) — improved U-Net architecture + Classifier Guidance, the first to outperform GANs across the board on ImageNet
2.3 Sampling Acceleration Methods
- Training-Free (training-free solvers): DDIM, DPM-Solver, DPM-Solver++, UniPC, DEIS, AYS (Align Your Steps)
- Training-Required (requires additional training): Progressive Distillation (Salimans & Ho, ICLR 2022), Consistency Models (Song et al., ICML 2023)
- Adversarial / distillation methods: Adversarial Diffusion Distillation (Sauer et al., ECCV 2024), Latent Adversarial Diffusion Distillation (Sauer et al., 2024), Truncated Diffusion Models (Zheng et al., ICLR 2023)
2.4 Latent Diffusion & Conditional Generation
- LDM: Rombach et al. (CVPR 2022) — performs diffusion in the latent space of a pretrained autoencoder, greatly reducing compute
- Stable Diffusion: an open-source text-to-image model family based on LDM (SD 1.x / 2.x / SDXL / SD 3 / SD 3.5)
- Classifier Guidance: Dhariwal & Nichol (NeurIPS 2021) — guides sampling with the gradient of a noise classifier
- Classifier-Free Guidance (CFG): Ho & Salimans (2022) — joint conditional/unconditional training, guidance without an extra classifier
- Large-scale text-to-image: GLIDE (Nichol et al., ICML 2022), DALL·E 2 (Ramesh et al., 2022), Imagen (Saharia et al., NeurIPS 2022)
- ControlNet: Zhang et al. (ICCV 2023) — spatial conditioning control via a trainable copy + zero convolutions
- T2I-Adapter / IP-Adapter: Mou et al. (AAAI 2024) / Ye et al. (2023) — lightweight structural conditioning and image-prompt adaptation
- SDXL: Podell et al. (ICLR 2024) — larger U-Net, dual text encoders, and micro-conditioning
- SD 3 / SD 3.5 / FLUX.1: Esser et al. (ICML 2024) among others — flow matching models based on DiT (MMDiT) and Rectified Flow
2.5 Personalization and Image Editing
- Personalization fine-tuning: Textual Inversion (Gal et al., ICLR 2023), DreamBooth (Ruiz et al., CVPR 2023), LoRA (Hu et al., ICLR 2022; widely used in diffusion models)
- Image editing: SDEdit (Meng et al., ICLR 2022), Prompt-to-Prompt (Hertz et al., ICLR 2023), Null-text Inversion (Mokady et al., CVPR 2023), InstructPix2Pix (Brooks et al., CVPR 2023)
- Image inpainting: RePaint (Lugmayr et al., CVPR 2022), plus dedicated models such as SD-Inpainting
2.6 Architectural Innovation
- DiT (Diffusion Transformers): Peebles & Xie (ICCV 2023) — a scalable Transformer backbone that shows clear scaling laws
- U-Net evolution: from the U-Net in DDPM to the ADM U-Net, then to the U-Net with cross-attention in LDM
- Dynamic DMs: dynamic computation allocation, such as DyDiT, AdaDiff, Learning-to-Cache
2.7 Alternative Paradigms (2023–2025 Frontier)
- Flow Matching: Lipman et al. (ICLR 2023) — simulation-free training of continuous normalizing flows
- Rectified Flow: Liu et al. (ICLR 2023) — straightens ODE paths, with Reflow straightening trajectories further
- Stochastic Interpolants: Albergo & Vanden-Eijnden (ICLR 2023) and follow-up work — unifies flows and diffusion
- Poisson Flow (PFGM / PFGM++): Xu et al. (NeurIPS 2022 / ICML 2023) — a generative model based on the Poisson equation (electrostatic field)
- Consistency Models: Song et al. (ICML 2023) — self-consistency mapping, 1–2 step sampling
- Consistency Trajectory Models: Kim et al. (ICLR 2024) — consistency trajectories that can jump between any two points on a trajectory
- MeanFlow: Geng et al. (2025) — learns the average velocity field, 1-step generation
- Inductive Moment Matching: Zhou, Ermon & Song (ICML 2025) — single-stage training, 1–2 step sampling
- AlphaFlow: Zhang et al. (Snap, ICLR 2026) — decomposes the MeanFlow objective into trajectory flow matching and trajectory consistency, using curriculum learning to ease the conflict between the two
- Decoupled MeanFlow: Lee, Yu & Shin (KAIST, 2025) — converts a pretrained flow model into a flow map without changing the architecture, 1–4 step generation
2.8 Cross-Modal Diffusion
- Video: Video LDM (Blattmann et al., CVPR 2023), AnimateDiff, Stable Video Diffusion (SVD), Latte, Sora (OpenAI, 2024)
- 3D: Point-E (OpenAI, 2022), DreamFusion (Poole et al., ICLR 2023), Shap-E (2023), SV3D (ECCV 2024), LN3Diff (ECCV 2024), Free3D (CVPR 2024)
- Audio: AudioLDM, AudioTurbo, VoiceFlow, ReFlow-TTS
- Text / discrete data: D3PM (Austin et al., NeurIPS 2021), ARDM, multinomial diffusion, absorbing-state (mask) diffusion
2.9 Discretization and Sampling Optimization
- DPM-Solver: Lu et al. (NeurIPS 2022) — a high-order ODE solver that exploits the semi-linear structure, about 10–20 steps
- DPM-Solver++: Lu et al. (arXiv 2022; Machine Intelligence Research 2025) — a high-order solver for guided sampling (data-prediction parameterization + multi-step method)
- DPM-Solver-v3: Zheng et al. (NeurIPS 2023) — a solver that exploits empirical statistics of the model
- UniPC: Zhao et al. (NeurIPS 2023) — a unified predictor-corrector framework
- DEIS: Zhang & Chen (ICLR 2023) — exponential integrator
- AYS: Sabour et al. (ICML 2024) — optimizes the sampling timestep schedule
- ERK-Guid: (ICLR 2026) — “Error as Signal: Stiffness-Aware Diffusion Sampling via Embedded Runge-Kutta Guidance” — estimates stiffness from the difference between the Heun and Euler solutions of different orders, reducing the local truncation error without increasing the number of network evaluations
- Bespoke Solvers: custom solvers tailored to specific models
3. Frontier Trends (2024–2026)
- DiT + Flow Matching becomes mainstream: SD 3 / SD 3.5 and FLUX.1 all use a DiT-style backbone + flow matching (Rectified Flow) objective
- 1–2 step generation quality approaches that of multi-step: Consistency Models, MeanFlow, InstaFlow, Decoupled MeanFlow
- Rectified Flow widely validated: more than a theoretical framework — it has been deployed in practice in the Stable Diffusion 3 series and FLUX.1
- Video generation heats up: led by Sora, but consistency, long videos, and physical plausibility remain challenges
- 3D generation shifts from optimization to direct generation: SV3D, LN3Diff, and others need no per-instance optimization
- Better controllability: the ControlNet ecosystem expands, along with lightweight methods such as IP-Adapter and T2I-Adapter
- Model efficiency: dynamic computation, caching, quantization, pruning, distillation
- Scientific computing applications: drug design (FlowSBDD), molecular generation, multiscale uncertainty quantification
- Test-time scaling: improving sampling quality by using more compute
- Integration with RL / robotics: Diffusion Policy, trajectory planning
4. Key Controversies and Points of Conflict
- Rectified Flow vs. traditional diffusion: is it better than traditional diffusion in every setting? The evidence so far is mainly in image generation; from the viewpoint in Sections 1.8–1.9, the two are largely just different choices of noise schedule and prediction target (loss weighting)
- Generalization of Flow Matching: its performance on discrete data such as text is still unverified
- The quality–speed trade-off of Consistency Models: 1-step generation quality still lags behind multi-step diffusion
- Optimality of the DPM-Solver schedule: AYS shows that traditional heuristic schedules are systematically suboptimal at low NFE
- U-Net vs. Transformer: DiT wins at large scale, but U-Net still has advantages on small and medium-sized tasks
5. Dimensions That Need Deeper Investigation
- Mathematical equivalences and differences among the various noise schedules
- Different variants of Score Matching (Sliced, Generalized, Denoising)
- Theoretical comparison of Classifier Guidance vs. Classifier-Free Guidance
- Temporal modeling mechanisms in video diffusion (3D convolution vs. temporal attention)
- Representation spaces for 3D diffusion (NeRF, 3D Gaussian Splatting, Voxel, Point Cloud)
- Mathematical frameworks for discrete diffusion (multiclass, absorbing state, continuous relaxation)
- Error analysis and convergence of different samplers
- Theoretical foundations of model distillation (Progressive Distillation, Consistency Distillation)
- Mathematical unification of Flow Matching and score-based models
- Computational efficiency optimization (token pruning, dynamic routing, caching)
6. Reading Guide
| Topic | Corresponding notes and chapter |
|---|---|
| Mathematical foundations (forward/reverse processes, ELBO, score matching, Tweedie, SDE/ODE, noise schedule, prediction-target parameterization) | Chapter 1 |
| Classic models (DDPM, Improved DDPM, DDIM, NCSN/NCSNv2, EDM) | Chapter 2 |
| Conditional generation and latent diffusion (Classifier Guidance, CFG, LDM, Stable Diffusion series, text encoders) | Chapter 3 |
| Controllable generation and editing (ControlNet, T2I-Adapter, IP-Adapter, Inpainting, LoRA/DreamBooth/Textual Inversion, SDEdit/P2P/InstructPix2Pix) | Chapter 4 |
| Sampling algorithms, distillation, and efficiency optimization | Chapter 5 |
| Frontier paradigms (DiT, Consistency Models, Flow Matching, Rectified Flow, MeanFlow) | Chapter 6 |
| Cross-modal diffusion (video, 3D, audio, discrete data, scientific applications) | Chapter 7 |
| Practical guide (hyperparameters, common pitfalls, code implementation) | Chapter 8 |
