Chapter 7: Cross-Modal Diffusion — Video, 3D, Audio, and Discrete Data
Diffusion models first broke through in image generation, but their core idea — starting from a simple distribution and approaching a complex data distribution through iterative denoising — is not limited to two-dimensional pixels. In recent years, researchers have extended the diffusion framework to a range of modalities, including video, three-dimensional geometry, audio, and discrete symbol sequences, creating the active frontier of cross-modal diffusion. This chapter walks through these directions one by one, focusing on three questions: what is hard about each modality, which part of the diffusion framework the main methods change, and the commonalities and trade-offs among them.
Note
This chapter keeps the notation from earlier chapters: $t$ is the diffusion time step, $T$ is the total number of diffusion steps, and $\epsilon_\theta$ is the noise prediction network. To avoid confusion, the video frame count is written $N_f$, not $T$.
7.1 Video Diffusion Models
Video adds a time dimension on top of images, and the core challenge in generation is temporal coherence: objects should change continuously in shape, color, and position from one frame to the next, rather than flickering and deforming the way independent per-frame sampling does. Video diffusion models must model temporal dynamics effectively while preserving spatial detail.
7.1.1 A Framework for Temporal Modeling in Video Generation
A video can be represented as a tensor $v \in \mathbb{R}^{N_f \times C \times H \times W}$. Video diffusion models usually perform “temporal inflation” on top of a pretrained image model — that is, they insert temporal layers responsible for cross-frame interaction into the U-Net or Transformer. The main architectural choices include:
| Architecture | Temporal modeling approach | Representative models | Features |
|---|---|---|---|
| 3D U-Net (factorized convolution + temporal attention) | Extends the 2D convolution to a 3D convolution that acts only on space ($1 \times 3 \times 3$), then inserts temporal attention | VDM (Ho et al. 2022) | One of the earliest video diffusion models; images and video can be trained jointly |
| Temporal attention / temporal convolution plug-in | Inserts cross-frame attention or a 1D temporal convolution after the spatial layers of a pretrained image model | Video LDM, SVD | Reuses the image prior, flexibly captures long-range temporal dependencies |
| Pluggable motion module | Freezes the image model and trains only a separate temporal module | AnimateDiff | Low training cost, transferable to personalized models |
| Spacetime Transformer | Cuts video into spacetime patches as tokens and applies (factorized or joint) spacetime attention | Latte, W.A.L.T, Sora, CogVideoX | Strong scalability, supports variable-length inputs |
Notation:
| Symbol | Meaning |
|---|---|
| $v$ | Video tensor |
| $N_f$ | Frame count (length of the time dimension) |
| $C$ | Channel count (3 in pixel space, i.e., RGB) |
| $H, W$ | Spatial height and width |
| $\epsilon_\theta$ | Noise prediction network with parameters $\theta$ |
7.1.2 Video LDM: Video Generation with Aligned Latents
Video LDM (Blattmann et al., CVPR 2023)1 is the pioneering work that extended latent diffusion models (LDM, §3.4) from images to video. The core idea is to reuse the autoencoder and denoising network of a pretrained image LDM and learn only “temporal alignment” — inserting temporal layers into the U-Net (temporal attention and 3D convolution, learnably mixed with the outputs of the original spatial layers), while also adding temporal layers to the VAE decoder to keep frames coherent.
Video LDM’s forward noising process matches that of the image LDM, except that it acts on the sequence of latents obtained by encoding each frame, $z \in \mathbb{R}^{N_f \times C' \times H' \times W'}$:
$$ q(z_t \mid z_{t-1}) = \mathcal{N}\left(z_t; \sqrt{1 - \beta_t}\, z_{t-1}, \beta_t \mathbf{I}\right) $$where $z_0 = \mathcal{E}(v)$ is the output of the VAE encoder (applied frame by frame), and $\beta_t$ is the noise schedule parameter.
Notation:
| Symbol | Meaning |
|---|---|
| $z_t$ | Latent at diffusion time step $t$ (containing all frames) |
| $\mathcal{E}$ | VAE encoder |
| $\beta_t$ | Noise variance parameter at step $t$ |
| $\mathcal{N}(\mu, \Sigma)$ | Gaussian distribution with mean $\mu$ and covariance $\Sigma$ |
Another key design in Video LDM is temporal fine-tuning of the decoder. The image VAE’s decoder has never seen video, so decoding frames independently causes flicker. Video LDM therefore inserts temporal layers into the decoder and fine-tunes it adversarially with a patch-discriminating video discriminator built from 3D convolutions, keeping the decoded result coherent across frames. The encoder stays frozen, which keeps the latent space compatible with the original image LDM (so the image model can be reused directly).
7.1.3 Stable Video Diffusion (SVD): Scaling Up Video Diffusion
Stable Video Diffusion (Blattmann et al., 2023)2 carries out a systematic study of training scale on top of Video LDM, proposing and validating a three-stage training paradigm:
- Text-to-image pretraining: built on Stable Diffusion 2.1, laying the foundation for visual quality and semantic understanding;
- Large-scale video pretraining: trained on a large video dataset that has been systematically curated (including automatic labeling and filtering), to learn motion priors;
- High-quality video fine-tuning: fine-tuned on a smaller but higher-quality video subset to improve generation fidelity.
One important conclusion of the paper is that carefully curated pretraining data is essential to final video quality. SVD follows the Video LDM setup, inserting temporal convolution and temporal attention after every spatial convolution and attention layer, adding roughly 656M parameters and bringing the U-Net to about 1.52B parameters in total2. The model controls motion magnitude through micro-conditioning such as motion_bucket_id, and controls frame rate through a frame-rate micro-condition.
The publicly released SVD weights are mainly image-to-video models: SVD generates 14 frames and SVD-XT generates 25, both at 576×1024 resolution. They are trained with the EDM framework (§2.4) and use classifier-free guidance (CFG).
7.1.4 AnimateDiff: A Plug-and-Play Motion Module
AnimateDiff (Guo et al., ICLR 2024)3 takes a lighter-weight approach: instead of modifying the weights of a pretrained text-to-image diffusion model, it introduces a separate motion module and trains only that module on video data.
The motion module is essentially a set of Transformer layers that perform self-attention along the time axis (with temporal positional encoding and a zero-initialized output projection), inserted at every resolution level of the Stable Diffusion U-Net. During training, the image backbone is frozen and only the motion module is updated. The biggest advantage of this design is transferability: a trained motion module can be dropped directly into any Stable Diffusion model derived from the same base — including fine-tuned or personalized versions (such as DreamBooth or LoRA, §4.5) — to “animate” it, with no retraining per model. The paper also introduces MotionLoRA, which adapts the motion module to new motion patterns (such as different camera movements) with a small amount of data and compute.
AnimateDiff’s denoising network can be written as “frozen image parameters + trainable motion parameters”:
$$ \hat{\epsilon} = \epsilon_{\theta_{\text{img}}, \theta_{\text{motion}}}\left( z_t, t, c_{\text{text}} \right), \qquad \theta_{\text{img}} \text{ frozen, only } \theta_{\text{motion}} \text{ trainable} $$At sampling time, we denoise iteratively as usual to obtain $z_0$, then decode frame by frame to get the video $v_{\text{gen}} = \mathcal{D}(z_0)$. The training objective is still the standard noise prediction loss $\left\lVert \epsilon - \hat{\epsilon} \right\rVert_2^2$.
Notation:
| Symbol | Meaning |
|---|---|
| $v_{\text{gen}}$ | Generated video |
| $z_t$ | Latent at diffusion time step $t$ (all frames) |
| $c_{\text{text}}$ | Text condition embedding |
| $\theta_{\text{img}}$ | Frozen image U-Net parameters (replaceable with any personalized version) |
| $\theta_{\text{motion}}$ | Trainable parameters of the motion module |
| $\mathcal{D}$ | VAE decoder |
7.1.5 Latte: A Latent Diffusion Transformer
Latte (Ma et al., 2024)4 replaces the U-Net backbone of video diffusion with a Transformer, validating the effectiveness of the DiT (§6.1) architecture for video generation. Latte first extracts spacetime tokens from video latents, then models the video distribution with a stack of Transformer blocks. To cope with the huge number of video tokens, the paper designs four efficient variants from the angle of “how to factorize the spatial and temporal dimensions”:
- Variant 1 (interleaved fusion): spatial Transformer blocks (attention only within a single frame) and temporal Transformer blocks (attention only across frames at the same spatial position) stacked alternately;
- Variant 2 (late fusion): the same number of blocks, but all spatial blocks stacked first, followed by all temporal blocks;
- Variant 3 (factorized attention): inside each Transformer block, spatial self-attention first, then temporal self-attention;
- Variant 4 (factorized attention heads): split the heads of multi-head attention into two groups, one for space and one for time, then merge them at the end.
The paper also systematically compares video patch embedding schemes (per-frame tiling vs. cross-frame spacetime tube tiling), ways of injecting time-step / class information, temporal positional encodings, and training strategies; variant 1 performs best in experiments. Latte achieved state-of-the-art results at the time on four standard datasets — FaceForensics, SkyTimelapse, UCF101, and Taichi-HD — and extends to text-to-video generation4.
7.1.6 Sora: A Unified Spacetime DiT Paradigm
Sora (OpenAI, 2024)5 is a landmark work in video generation. Its technical report only disclosed high-level methods, which can be summarized as follows:
Video compression network: instead of encoding frame by frame, Sora trained an autoencoder that compresses in both time and space, mapping video to latents with lower resolution and a shorter time dimension, then training and generating in this latent space.
Spacetime patches and a diffusion Transformer: the compressed latents are cut into spacetime patches to serve as Transformer tokens. Since an image is just a video with one frame, this representation can handle images and video uniformly. Sora trains directly on data at native resolution, duration, and aspect ratio, rather than cropping video to a fixed size; at generation time, it controls output size by arranging grids of noise patches of different sizes. The report states that Sora is a diffusion Transformer trained to predict the original “clean” patches from noisy ones.
Re-captioning: following the approach of DALL·E 3, first train a highly descriptive video captioning model to generate detailed captions for all training videos; at inference time, use GPT to expand the user’s short prompt into a detailed description, significantly improving adherence to complex text instructions.
Because implementation details are not public, the following uses only the general diffusion reverse process to illustrate the denoising step for models like Sora:
$$ p_\theta(z_{t-1} \mid z_t, c) = \mathcal{N}\left(z_{t-1};\, \mu_\theta(z_t, t, c),\, \Sigma_\theta(t)\right) $$where $z_t$ is the compressed spacetime latent, $c$ is the text condition, and $\mu_\theta$ is given by the Transformer network.
Notation:
| Symbol | Meaning |
|---|---|
| $z_t$ | Compressed spacetime latent |
| $c$ | Text condition / prompt |
| $\mu_\theta$ | Denoising mean predicted by the network |
| $\Sigma_\theta(t)$ | Preset or predicted time-dependent covariance |
7.1.7 CogVideo, CogVideoX, and the Open-Source Ecosystem
CogVideo (Hong et al., 2022) is an early open-source large-scale text-to-video model, but it is not a diffusion model: built on the pretrained text-to-image model CogView2, it uses an autoregressive Transformer to generate video tokens and adopts a hierarchical generation strategy — first generating low-frame-rate keyframes, then generating intermediate frames through interpolation. Its successor CogVideoX (Yang et al., ICLR 2025)6 moves to the diffusion framework, introducing a 3D causal VAE (which compresses in both space and time and is causal along the time axis, so the first frame can be encoded independently as an image), a Transformer with “expert-adaptive LayerNorm,” and 3D full attention, improving generation quality and long-video capability. Recent open-source video diffusion models (such as HunyuanVideo and Wan) also commonly follow the technical route of “3D causal VAE + spacetime DiT + flow matching training.”
The core trade-off in temporal modeling is computation. Suppose each frame has $h \times w$ tokens in latent space, with $N_f$ frames in total:
- 3D full attention attends over all $N_f h w$ tokens together, giving the strongest expressiveness, but at a cost of $O\left( (N_f h w)^2 \right)$;
- Factorized spacetime attention (within frames first, then across frames, as in Latte variant 3) costs $O\left( N_f h w \cdot (h w + N_f) \right)$, which is much cheaper at high resolution and long video lengths, but information across frames and positions can only be exchanged indirectly.
7.1.8 Long Video and Autoregressive Video Diffusion
The models above generate a clip of fixed length in one pass. To generate longer videos, common approaches include:
- Segment generation + overlap: condition the next segment on the last few frames of the previous one (a common usage for models such as SVD);
- Diffusion Forcing7: during training, sample a noise level independently for each token (frame) in the sequence, so the model can both denoise the whole sequence like a diffusion model and “generate what comes next given what came before” like an autoregressive model, enabling variable-length, continuously extendable sequence generation. This idea of “different noise levels per frame” has become an important foundation for autoregressive video diffusion and world models.
7.2 Three-Dimensional Diffusion Models
Three-dimensional generation requires the model to output data with spatial structure (point clouds, meshes, voxels, implicit fields, 3D Gaussians, or multi-view images), and the core challenges are multi-view consistency and valid geometry / topology. At the same time, high-quality 3D training data is far scarcer than image data. Two broad paradigms have emerged: direct 3D diffusion (training a diffusion model on some 3D representation) and 3D generation with 2D priors (using a pretrained 2D diffusion model to guide the optimization or reconstruction of a 3D representation).
7.2.1 Direct Diffusion on 3D Representations
Point-E: Three-Stage Point Cloud Generation
Point-E (Nichol et al., OpenAI, 2022)8 breaks text-to-3D generation into a cascade of three steps:
- Text to image: use a GLIDE text-to-image diffusion model fine-tuned on 3D renders to generate a synthetic viewpoint image;
- Image to coarse point cloud: conditioned on that image, use a diffusion model to generate a coarse point cloud of 1024 points (with colors);
- Point cloud upsampling: use another diffusion model conditioned on the low-resolution point cloud to refine it to 4096 points.
If a mesh is needed, Point-E also trains a regression model to predict an SDF from the point cloud, then extracts the surface with Marching Cubes. Point-E generates quickly (a minute or two on a single GPU), but has limited fidelity because point clouds lack surface topology information.
Shap-E: Conditional Diffusion of Implicit Functions
Shap-E (Jun & Nichol, OpenAI, 2023)9 instead generates the parameters of an implicit function, which can be rendered either as a textured mesh or as a neural radiance field (NeRF). Training proceeds in two stages:
- Encoder training: train an encoder that maps a 3D asset deterministically to the parameters of an implicit function (an MLP);
- Conditional diffusion: train a conditional diffusion model on the encoder’s output parameters, generating new implicit-function parameters conditioned on text or an image.
The paper finds that although Shap-E must model a higher-dimensional output space and support multiple representations at once, it still converges faster than Point-E, with comparable or better sample quality, and directly yields a renderable textured mesh.
Voxel, Point Cloud, and Triplane Diffusion
Other representative representations include:
- Point cloud diffusion: for example PVD (Point-Voxel Diffusion), which denoises point coordinates directly with a point-voxel hybrid network;
- Implicit field / voxel diffusion: for example Diffusion-SDF, which diffuses over implicit or voxelized representations such as signed distance fields (SDFs);
- Triplane diffusion (e.g., NFD): encode a 3D shape into 2D feature maps on three orthogonal planes, then diffuse over those feature maps — reusing 2D diffusion architectures while substantially lowering computational complexity.
7.2.2 Score Distillation Sampling (SDS): Generating 3D with 2D Priors
DreamFusion (Poole et al., Google, ICLR 2023 outstanding paper)10 proposed an entirely new 3D generation paradigm: without training any 3D diffusion model, it treats a pretrained 2D text-to-image diffusion model as a “critic” and uses Score Distillation Sampling (SDS) to optimize a differentiable 3D representation (such as a NeRF) so that images rendered from any viewpoint look “like” real images matching the text description.
Core SDS Equation
Given the parameters $\phi$ of a differentiable 3D representation (such as a NeRF’s MLP weights) and the image rendered from viewpoint $\pi$, $x = g(\phi, \pi)$, SDS updates $\phi$ iteratively with the following gradient:
$$ \nabla_\phi \mathcal{L}_{\text{SDS}} = \mathbb{E}_{t, \pi, \epsilon} \left[ w(t) \left( \epsilon_\theta(x_t; t, c) - \epsilon \right) \frac{\partial x}{\partial \phi} \right] $$where $x_t = \alpha_t x + \sigma_t \epsilon$ is the forward-noised rendered image (continuous-time notation, as in §5.1.3), $\epsilon_\theta$ is the noise prediction of the frozen pretrained 2D diffusion model, $c$ is the text condition, and $w(t)$ is a time-step weight.
Where does this gradient come from? If we differentiate the diffusion model’s training loss $w(t) \lVert \epsilon_\theta(x_t; t, c) - \epsilon \rVert^2$ directly with respect to $\phi$, the chain rule produces a U-Net Jacobian term $\partial \epsilon_\theta / \partial x_t$, which is expensive and unstable to compute. DreamFusion found that simply dropping this term works even better, giving the concise form above. Since $\epsilon_\theta - \epsilon$ is proportional to the (negative) score difference, SDS can be understood as using the score provided by the 2D diffusion model to push each viewpoint’s rendering toward “high-probability image regions given the text.”
Notation:
| Symbol | Meaning |
|---|---|
| $\phi$ | Trainable parameters of the 3D representation (e.g., a NeRF) |
| $g(\phi, \pi)$ | Differentiable rendering function for images from viewpoint $\pi$ |
| $x$ | Rendered 2D image |
| $x_t$ | Image after noising $x$ |
| $\epsilon_\theta$ | Noise prediction network of the frozen pretrained 2D diffusion model |
| $c$ | Text condition / prompt |
| $w(t)$ | Time-step-dependent weight coefficient |
| $\alpha_t, \sigma_t$ | Signal and noise coefficients |
Variants and Improvements
SDS suffers from oversaturation, over-smoothing, poor diversity, and the “polyhedron / Janus problem” (an object growing a frontal face when viewed from many directions), and requires a very large CFG scale (DreamFusion uses 100). Later improvements include:
- Magic3D (Lin et al., NVIDIA, CVPR 2023): a coarse-to-fine two-stage approach — first optimize a coarse neural field with a low-resolution diffusion prior, then extract a DMTet mesh and fine-tune it with a high-resolution latent diffusion model;
- Fantasia3D (Chen et al., ICCV 2023): decouples geometry and appearance (material) and optimizes them separately;
- ProlificDreamer / VSD (Wang et al., NeurIPS 2023)11: introduces Variational Score Distillation, treating the 3D parameters as a distribution rather than a single point, and replacing the random noise $\epsilon$ in SDS with a LoRA diffusion model $\epsilon_\psi$ fine-tuned online on the current renderings:
VSD works at a normal CFG scale (such as 7.5), substantially reducing oversaturation and over-smoothing, with better detail and diversity. It shares the same “difference of two scores” idea as distribution matching distillation (DMD, §5.3.4) in Chapter 5;
- MVDream (Shi et al., ICLR 2024): trains a multi-view diffusion model that can generate multiple consistent viewpoints at once, and uses it in place of the single-view 2D prior for SDS, substantially alleviating the Janus problem.
7.2.3 Multi-View-Consistent Diffusion and 3D Reconstruction
SV3D: From Video Diffusion to Multi-View Synthesis
SV3D (Voleti et al., Stability AI, ECCV 2024)12 cleverly repurposes a video diffusion model into a multi-view diffusion model: an “orbital video” shot by circling an object is itself a set of multi-view images, with each frame corresponding to one viewpoint. Building on Stable Video Diffusion, SV3D adds camera pose conditioning (elevation and azimuth), enabling camera-controllable multi-view synthesis from a single image.
Its 3D generation pipeline is also coarse-to-fine: first optimize a NeRF supervised by the generated multi-view images, then extract a DMTet mesh for fine-tuning. For regions invisible outside the generated viewpoints, SV3D introduces a Masked SDS loss that applies score distillation only to those invisible regions; it also jointly optimizes a decoupled lighting model to reduce the problem of lighting being “baked into” the texture.
Consistent123 and Free3D
- Consistent123 (Weng et al., 2023)13: synthesizes multiple novel views at once, passing information between views through cross-view attention and shared self-attention, and uses progressive classifier-free guidance to improve multi-view consistency in one-image-to-3D;
- Free3D (Zheng & Vedaldi, CVPR 2024)14: like Zero-1-to-3, it starts from a pretrained 2D image generator and fine-tunes it for novel-view synthesis, but uses no explicit 3D representation and trains no additional 3D reconstruction network. Its key is ray conditioning normalization (RCN), which tells each pixel its own viewing direction; combined with lightweight multi-view attention and shared multi-view noise, this improves consistency across views.
Combining 3D Gaussian Splatting with Diffusion
3D Gaussian Splatting (3DGS) has become a new vehicle for 3D generation because it is differentiable and renders extremely fast:
- DreamGaussian (Tang et al., ICLR 2024): combines SDS with 3DGS, replacing NeRF’s occupancy pruning with progressive densification of 3D Gaussians, which converges much faster; it then converts the Gaussians to a textured mesh and fine-tunes in UV space. Generating a high-quality textured mesh from a single image takes about 2 minutes, roughly 10 times faster than existing methods at the time;
- Diffusing directly on 3D Gaussian parameters: treating each Gaussian’s position, covariance, color, and opacity as the object of diffusion is an active research direction;
- Feed-forward reconstruction: for example Splatter Image (Szymanowicz et al., CVPR 2024) regresses 3D Gaussian parameters from a single image in one network pass (it is itself a regression model, not a diffusion model). A common efficient pipeline in practice is “multi-view diffusion model generates multi-view images + feed-forward large reconstruction model (LRM-style) reconstructs 3D,” completing one-image / text-to-3D in seconds to minutes.
7.2.4 Comparison of 3D Generation Methods
| Method | Representation | Condition | Advantages | Disadvantages | Key parameters |
|---|---|---|---|---|---|
| Point-E | Point cloud | Text / image | Fast generation | No topology, poor detail | Point count, diffusion steps |
| Shap-E | Implicit function parameters | Text / image | Directly outputs renderable mesh / NeRF | Complex training pipeline | Implicit function parameter dimensionality |
| DreamFusion (SDS) | NeRF / mesh | Text | Needs no 3D data, exploits a strong 2D prior | Oversaturation, Janus problem, slow per-sample optimization | guidance scale, learning rate |
| ProlificDreamer (VSD) | NeRF / mesh | Text | Better detail and diversity | Slower optimization | LoRA learning rate, CFG |
| SV3D | Multi-view images → NeRF / mesh | Single image + pose | Strong multi-view consistency, camera-controllable | Depends on a pretrained video model | Camera pose range |
| Free3D | No explicit 3D | Single image | No reconstruction needed, direct novel-view synthesis | Does not directly yield a 3D model | Pose encoding, view count |
| 3DGS + diffusion | 3D Gaussians | Text / image | Extremely fast rendering | Gaussian count can balloon | Gaussian count, spherical harmonics (SH) order |
7.3 Audio Diffusion Models
An audio signal is a one-dimensional waveform that varies over time, $a(t) \in \mathbb{R}$ (here $t$ is physical time), with strong temporal structure and frequency characteristics. Because raw waveforms have a high sampling rate (16–48 kHz) and extremely long sequences, audio diffusion models usually do not generate directly on the waveform; instead they diffuse in the time-frequency domain (such as mel spectrograms) or in a learned latent space, then recover the waveform with a vocoder or decoder.
7.3.1 The Spectrogram Latent Diffusion Paradigm
AudioLDM (Liu et al., ICML 2023)15 is a representative work applying the LDM framework to text-to-audio generation. Its pipeline is:
- Convert the audio waveform to a mel spectrogram $X \in \mathbb{R}^{L \times F}$ via STFT;
- Compress it to a latent $z \in \mathbb{R}^{C \times L' \times F'}$ with a VAE trained on mel spectrograms;
- Train a diffusion model in the latent space. During training, condition on CLAP’s audio embedding $E^{x}$; during sampling, swap in CLAP’s text embedding $E^{y}$ — because CLAP aligns audio and text in the same embedding space, the model can perform text-to-audio generation without paired “text-audio” data;
- After generation, recover the mel spectrogram with the VAE decoder, then convert it to a waveform with the HiFi-GAN vocoder.
AudioLDM’s forward process matches that of the image LDM (using $n$ for the diffusion step, to avoid confusion with the spectrogram’s time axis):
$$ q(z_n \mid z_{n-1}) = \mathcal{N}\left(z_n; \sqrt{1 - \beta_n}\, z_{n-1}, \beta_n \mathbf{I}\right) $$The training objective is noise prediction (conditioned on the audio embedding $E^{x}$ during training):
$$ \mathcal{L}(\theta) = \mathbb{E}_{z_0, \epsilon, n} \left[ \left\lVert \epsilon - \epsilon_\theta(z_n, n, E^{x}) \right\rVert_2^2 \right] $$Notation:
| Symbol | Meaning |
|---|---|
| $X$ | Mel spectrogram ($L$ time frames, $F$ mel bands) |
| $z$ | Audio latent after VAE compression |
| $E^{x}$ / $E^{y}$ | CLAP audio embedding (training) / CLAP text embedding (sampling) |
| $\epsilon$ | Injected Gaussian noise |
| $n$ | Discrete diffusion time step |
| $\epsilon_\theta$ | Noise prediction network (U-Net) |
Key AudioLDM configuration: AudioLDM-S has about 181 million parameters and AudioLDM-L about 739 million; the number of diffusion steps is $N = 1000$, with a linear noise schedule from $\beta_1 = 0.0015$ to $\beta_N = 0.0195$; sampling uses 200 DDIM steps and a CFG scale of $w = 2.0$ (in the CFG convention of the original AudioLDM paper)15.
7.3.2 A Lineage of Audio Generation Models
| Model | Representation | Condition | Key feature | Best suited for |
|---|---|---|---|---|
| AudioGen (Meta) | Discrete audio tokens | Text | Autoregressive generation (not diffusion), included as a contrast | General sound effects, ambient sound |
| AudioLDM | Continuous latents | Text / audio | Uses CLAP for text-to-audio without paired data; supports zero-shot editing | Sound effects, music, speech |
| MusicLDM | Continuous latents | Text | Retrains CLAP and HiFi-GAN for music on the AudioLDM / SD architecture; introduces beat-synchronous mixup to boost novelty and reduce plagiarism | Music generation |
| VoiceFlow | Mel spectrograms | Text | Uses flow matching (Rectified Flow) instead of diffusion, achieving high quality with only a few steps | Speech synthesis (TTS) |
| Stable Audio | Continuous latents | Text + duration | Latent diffusion based on a fully convolutional VAE, with duration controlled by a time embedding; can generate up to about 95 seconds of 44.1 kHz stereo | Music and sound-effect production |
The advantage of spectrogram / latent-space diffusion is this: mel spectrograms organize a high-sample-rate waveform into a regular, image-like two-dimensional representation (for 16 kHz audio with a hop length of 160 and 64 mel bands, that’s $100 \times 64 = 6400$ values per second, about $1/2.5$ of the raw waveform), and VAE compression reduces the dimensionality much further, making it easy to reuse image diffusion architectures directly. Mel spectrograms do discard phase information, however, so a high-quality vocoder (such as HiFi-GAN or BigVGAN) is needed to reconstruct the waveform; the fixed window length of the STFT also makes the reconstruction of transient sounds (such as the attack of a percussion note) imprecise.
Diffusion models are also used as vocoders themselves: models such as DiffWave and WaveGrad perform conditional diffusion directly on the waveform, recovering it from a mel spectrogram. In speech synthesis, flow-matching-based non-autoregressive TTS (such as VoiceFlow and Voicebox) has also become one of the mainstream directions.
7.3.3 Audio Editing and Style Transfer
A trained audio diffusion model supports a range of zero-shot editing tasks (the same ideas as SDEdit and inpainting in images; see §4.4):
- Audio inpainting: mask a specific time-frequency region of the latents and let the model fill in the missing content;
- Style transfer: noise an existing audio clip to an intermediate time, then denoise it under a new text condition (such as swapping “calm music” for “energetic music”), changing the style while preserving the overall structure;
- Super-resolution: treat the low-frequency content as known and generate the missing high-frequency detail.
7.4 Discrete-Data Diffusion Models
Standard diffusion models assume data lies in a continuous space $\mathbb{R}^d$ and add Gaussian noise in the forward process. But text, molecular graphs, and categorical data are inherently discrete — “adding 0.1 of Gaussian noise to a word” is meaningless. Extending the diffusion framework to discrete domains requires redefining “noising”: progressively destroying the data with random transitions between states.
7.4.1 Multicategory Discrete Diffusion (D3PM)
D3PM (Discrete Denoising Diffusion Probabilistic Models; Austin et al., NeurIPS 2021)16 generalizes Gaussian diffusion to discrete state spaces. Suppose each variable has $K$ possible values, represented by a one-hot row vector $x_t$; the forward process implements probabilistic transitions between states through a transition matrix $Q_t$:
$$ q(x_t \mid x_{t-1}) = \mathrm{Cat}\left(x_t;\, p = x_{t-1} Q_t\right) $$where $[Q_t]_{ij} = q(x_t = j \mid x_{t-1} = i)$, and each row of $Q_t \in \mathbb{R}^{K \times K}$ sums to 1. As in the Gaussian case, multiple transitions compose directly, giving closed-form marginals and forward posteriors at any time step:
$$ q(x_t \mid x_0) = \mathrm{Cat}\left(x_t;\, p = x_0 \bar{Q}_t\right), \quad \bar{Q}_t = Q_1 Q_2 \cdots Q_t; \qquad q(x_{t-1} \mid x_t, x_0) = \mathrm{Cat}\left(x_{t-1};\, p = \frac{x_t Q_t^{\top} \odot x_0 \bar{Q}_{t-1}}{x_0 \bar{Q}_t x_t^{\top}}\right) $$So during training, $x_t$ can be sampled in one step, while the reverse network learns to predict $x_0$ (or the posterior directly). For multidimensional data $x \in \{1, \dots, K\}^{D}$, each dimension transitions independently:
$$ q(x_t \mid x_{t-1}) = \prod_{d=1}^{D} \mathrm{Cat}\left(x_t^{d};\, p = x_{t-1}^{d} Q_t\right) $$Notation:
| Symbol | Meaning |
|---|---|
| $x_t$ | Discrete state at time step $t$ (one-hot row vector) |
| $K$ | Number of categories per dimension |
| $D$ | Data dimensionality (e.g., sequence length) |
| $Q_t$, $\bar{Q}_t$ | Single-step / cumulative transition probability matrices |
| $\mathrm{Cat}(\cdot\,; p)$ | Categorical distribution parameterized by probability vector $p$ |
| $\odot$ | Elementwise multiplication |
D3PM explores several transition matrix designs:
- Uniform: with some probability, resample the current value uniformly at random to any category;
- Absorbing: with some probability, transition to a special
[MASK]absorbing state; once masked, the value no longer changes, similar to BERT’s masked language modeling; - Discretized Gaussian: suited to ordered categories (such as pixel values), favoring transitions to neighboring values to mimic the locality of Gaussian diffusion;
- Nearest token embedding: define transitions by similarity in the word embedding space, so a word is more likely to become a semantically similar word.
Experiments show that absorbing diffusion performs best on text: starting from a fully masked state, the model progressively “reveals” the original text, an interesting contrast to autoregressive language models.
7.4.2 Autoregressive Diffusion Models (ARDM)
ARDM (Autoregressive Diffusion Models; Hoogeboom et al., ICLR 2022)17 reveals the connection between absorbing diffusion and “order-agnostic autoregressive models” (order-agnostic ARM): in the continuous-time limit, each variable is absorbed into the mask state at some random time, so the forward process is equivalent to choosing a “masking order” uniformly from all permutations of the variables; the reverse process “unmasks” in the opposite order, which amounts to generating variables one by one in random order. ARDM’s advantages are:
- Parallel generation: unlike standard autoregressive models, which must generate token by token in a fixed order, ARDM can predict multiple not-yet-generated variables in parallel in a single step;
- Dynamic-programming grouping: dynamic programming can choose the optimal “how many variables to generate per step” for a given step budget, trading off network calls against likelihood, so generation completes in far fewer steps than the data dimensionality;
- Likelihood: reaches $\le 1.43$ BPC (bits per character, lower is better) on Text8.
7.4.3 Continuous-Time and Masked Diffusion Language Models
SEDD (Score Entropy Discrete Diffusion; Lou et al., ICML 2024) extends the notion of a score to discrete space: it learns the probability ratio $p_t(y)/p_t(x)$ between different states (called a concrete score) and introduces a score entropy loss, substantially narrowing the perplexity gap between diffusion language models and autoregressive models.
MDLM (Masked Diffusion Language Models; Sahoo et al., NeurIPS 2024)18 shows that simple masked diffusion, combined with modern training techniques, is already very powerful. In continuous time, its negative ELBO simplifies to a compact form:
$$ \mathcal{L}_{\text{NELBO}}^{\infty} = \mathbb{E}_{q} \int_{0}^{1} \frac{\alpha_t'}{1 - \alpha_t} \sum_{\ell} \log \left\langle x_\theta^{\ell}(z_t, t),\, x^{\ell} \right\rangle \mathrm{d}t $$| Symbol | Meaning |
|---|---|
| $\alpha_t$ | Probability that a token is not masked at time $t$, decreasing monotonically from $\alpha_0 \approx 1$ to $\alpha_1 \approx 0$ |
| $\alpha_t'$ | Derivative of $\alpha_t$ with respect to $t$ (negative, so the overall loss is positive) |
| $z_t$ | Partially masked sequence |
| $x_\theta^{\ell}(z_t, t)$ | Network’s predicted distribution for the original token at position $\ell$ |
| $\left\langle \cdot, x^{\ell} \right\rangle$ | Extracts the predicted probability corresponding to the true token $x^{\ell}$ (the sum is nonzero only at masked positions) |
Intuition: this objective is “a weighted average of BERT-style masked language modeling losses at different mask rates,” bridging generative diffusion models and encoder-only BERT. Along this line, LLaDA (Nie et al., 2025)19 trained an 8-billion-parameter masked diffusion language model from scratch, performing on par with same-size autoregressive baselines (such as LLaMA3 8B) under a pretraining-plus-supervised-fine-tuning paradigm, and excelling at “reversal curse” tasks (such as completing a poem backward), showing that diffusion models are viable for large-scale language modeling as well.
7.4.4 Continuous Relaxations and Diffusion in Embedding Space
Another strategy leaves Gaussian diffusion itself unchanged and instead diffuses discrete data in a continuous space:
- Bit Diffusion / Analog Bits (Chen et al., ICLR 2023): represent integer data as binary bits, perform continuous diffusion on the real-valued bits, then threshold back to discrete values after generation;
- Plaid (Gulrajani & Hashimoto, NeurIPS 2023): map discrete tokens into a continuous embedding space, perform Gaussian diffusion on the embeddings, and narrow the gap to autoregressive models through likelihood training and scaling-law studies;
- RDLM (Riemannian Diffusion Language Model; Jo & Hwang, NeurIPS 2025)20: exploits the geometry of categorical distributions to define continuous diffusion on a statistical manifold, linking discrete diffusion to continuous flows on manifolds, and uses radial symmetry to enable simulation-free training; it reaches $\le 1.32$ BPC on Text8, surpassing all previous diffusion models.
7.4.5 Comparison of Discrete Diffusion Methods
| Method | Noising / modeling approach | Text8 BPC (lower is better) | Key feature |
|---|---|---|---|
| Multinomial Diffusion | Uniform transitions | $\le 1.72$ | One of the earliest discrete diffusion models |
| D3PM Uniform | Uniform transitions | $\le 1.61$ | Systematically explores multiple transition matrices |
| D3PM Absorb | Absorbing (masking) | $\le 1.45$ | Best text performance among discrete transitions |
| ARDM | Continuous-time absorbing / order-agnostic autoregressive | $\le 1.43$ | Supports parallel generation |
| SEDD Absorb | Continuous time, concrete score | $\le 1.39$ | Score-based discrete diffusion |
| MDLM | Continuous-time masked diffusion | $\le 1.40$ | Compact objective, scales to large models (LLaDA) |
| RDLM | Continuous diffusion on a statistical manifold | $\le 1.32$ | Best diffusion model on Text8 as of the paper’s publication |
Numbers are taken from the original results for each method as compiled in the RDLM paper (all are likelihood upper bounds)20.
7.5 Scientific Applications: Molecules, Proteins, and Materials Generation
The use of diffusion models in the natural sciences is growing rapidly, especially when it comes to generating three-dimensional structures that satisfy physical and chemical constraints. Such data shares a common property: the structure is invariant under rotation and translation (and sometimes reflection), so the network usually needs to be E(3) / SE(3) equivariant — rotate the input molecule as a whole, and the predicted noise or velocity should rotate with it in the same way.
7.5.1 Molecule Generation and Structure-Based Drug Design
FlowSBDD (Zhang et al., 2024)21 brings Rectified Flow (§6.5) to structure-based drug design (SBDD): given a protein pocket, generate a 3D ligand molecule that binds to it. The paper notes that conventional diffusion models, constrained by their fixed learning objective, struggle to flexibly incorporate additional optimization targets or conditions; FlowSBDD builds on the Rectified Flow framework (with the time convention of §6.4, where $x_0$ is noise and $x_1$ is the molecule):
$$ \frac{\mathrm{d}x_t}{\mathrm{d}t} = v_\theta\left(x_t, t, c_{\text{pocket}}\right), \quad x_0 \sim p_0, \; x_1 \sim p_1 $$where $x_t$ is the molecule’s atom coordinates and atom types, $v_\theta$ is the velocity field network, and $c_{\text{pocket}}$ is the protein pocket condition. Because the training objective is simply to regress the velocity along a linear interpolation path, FlowSBDD can:
- flexibly add extra losses (such as a bond-length loss) to improve chemical validity;
- take additional conditions as input, or replace the initial Gaussian with another distribution;
- reach an average Vina Dock score of −8.50 and 75.0% diversity on CrossDocked202021.
Notation:
| Symbol | Meaning |
|---|---|
| $x_t$ | Molecular state at time $t$ (coordinates + atom types) |
| $v_\theta$ | Velocity field parameterized by the network |
| $c_{\text{pocket}}$ | Conditional features of the protein pocket |
| $p_0, p_1$ | Initial distribution (noise) and target distribution (real molecules) |
Other representative work includes:
- EDM (E(3) Equivariant Diffusion Model; Hoogeboom et al., ICML 2022): uses an E(3)-equivariant graph network to denoise atom coordinates (continuous) and atom types simultaneously, generating 3D molecules. Note that it shares its name with, but is completely different from, the EDM of Karras et al. in Chapter 2 (Elucidating the Design space of diffusion Models);
- GeoDiff (Xu et al., ICLR 2022): given a molecular graph, generates its 3D conformer while guaranteeing rotation-translation invariance;
- TargetDiff (Guan et al., ICLR 2023): SE(3)-equivariant diffusion under pocket conditioning, jointly generating a ligand’s atom positions and types;
- DiffSBDD (Schneuing et al., 2022 / 2024)22: E(3)-equivariant 3D conditional diffusion that generates ligand molecules directly in the pocket.
7.5.2 Protein Structure Generation
Protein backbone generation requires outputting an amino-acid chain structure that is physically foldable and designable. The main methods include:
- ProteinSGM: a score-based generative model that diffuses over inter-residue distance and angle matrices, then rebuilds 3D coordinates with Rosetta;
- FoldingDiff: represents the protein backbone as a sequence of internal angles (torsion and bond angles) and generates over the angles with DDPM, which is naturally rotation-translation invariant and avoids the complexity of modeling 3D coordinates directly;
- Chroma (Ingraham et al., Nature 2023): diffuses directly on backbone atom coordinates using correlated noise that follows polymer statistics, and supports a range of composable conditional constraints;
- RFdiffusion (Watson et al., Nature 2023): a diffusion model fine-tuned from the structure prediction network RoseTTAFold, supporting unconditional backbone generation as well as conditional tasks such as motif scaffolding and binder design, validated by extensive experiments;
- Protpardelle: an all-atom (including side chains) protein diffusion model that generates sequence and structure together;
- AlphaFold 3 (Abramson et al., Nature 2024): in the structure prediction task, uses a diffusion module to generate all atom coordinates directly, replacing AlphaFold 2’s structure module, and can predict complexes of proteins, nucleic acids, small molecules, and more.
7.5.3 Materials Design and Crystal Structure Generation
Applications of diffusion models in materials science include:
- Crystal structure generation: models such as CDVAE and DiffCSP diffuse under the periodic constraints of the crystal lattice, generating structures with a specified stoichiometric ratio;
- MatterGen (Zeni et al.)23: proposes a diffusion process that progressively refines atom types, coordinates, and the periodic lattice together, generating stable, diverse inorganic materials spanning the entire periodic table; adapter modules fine-tune it on labeled data to steer generation toward target properties (such as chemical composition, symmetry, magnetism, and electronic and mechanical properties). The paper reports that the fraction of generated structures that are “novel and stable” is more than twice that of previous generative models, and closer to local energy minima;
- Property-guided generation: use classifier guidance or classifier-free guidance (§3.2–§3.3) to generate new materials with target properties such as a desired band gap or ionic conductivity.
graph TD
A[Cross-Modal Diffusion Models] --> B[Video Diffusion]
A --> C[3D Diffusion]
A --> D[Audio Diffusion]
A --> E[Discrete Diffusion]
A --> F[Scientific Applications]
B --> B1[Latent Video: VideoLDM/SVD]
B --> B2[Motion Adapter: AnimateDiff]
B --> B3[Video DiT: Latte/Sora/CogVideoX]
B --> B4[Long Video: Diffusion Forcing]
C --> C1[Direct 3D: Point-E/Shap-E]
C --> C2[2D-to-3D Lifting: DreamFusion/SDS/VSD]
C --> C3[Multi-View: SV3D/Consistent123]
C --> C4[3DGS: DreamGaussian]
D --> D1[AudioLDM]
D --> D2[MusicLDM/Stable Audio]
D --> D3[VoiceFlow]
E --> E1[D3PM]
E --> E2[ARDM/SEDD/MDLM]
E --> E3[Continuous Relaxation: Plaid/RDLM]
F --> F1[FlowSBDD]
F --> F2[TargetDiff/E3-EDM]
F --> F3[RFdiffusion/AlphaFold3]
F --> F4[Crystal Generation: MatterGen]
Chapter 8: Practical Guide — Parameter Selection, Code Implementation, and Pitfalls
Getting from theory to a diffusion model that actually runs and produces good results means clearing several hurdles: engineering implementation, hyperparameter tuning, and training stability. This chapter provides a practice-oriented framework for choosing methods, reusable code templates, evaluation approaches, and a guide to troubleshooting common problems.
Warning
Unless a source is explicitly cited, the hyperparameters and resource figures in this chapter’s tables are empirical ranges or order-of-magnitude estimates, meant to help build intuition; for a specific project, defer to the official configuration of the model you use and to your own experiments.
8.1 A Decision Tree for Choosing a Method
How do you choose the right diffusion model architecture and training strategy for different generation tasks? The decision tree below lays out a systematic logic for making that choice.
graph TD
Q1{Data type?} -->|Image| Q2
Q1 -->|Video| V1[VideoLDM/SVD/Latte/CogVideoX]
Q1 -->|3D| Q3
Q1 -->|Audio| A1[AudioLDM/Stable Audio]
Q1 -->|Text/Discrete| D1[MDLM/SEDD/LLaDA]
Q1 -->|Molecule/Protein| S1[E3-EDM/FlowSBDD/RFdiffusion]
Q2{Resolution/quality requirement?} -->|High resolution| Q4
Q2 -->|Fast prototype| Q5
Q3{Do you have 3D training data?} -->|Yes| C1[Direct 3D diffusion: Shap-E/Point-E]
Q3 -->|No| C2[2D-to-3D lifting: DreamFusion/SDS/VSD]
Q3 -->|Single image to 3D| C3[SV3D/Multi-view diffusion + reconstruction]
Q4{Compute budget?} -->|Ample| I1[Train LDM/DiT from scratch]
Q4 -->|Limited| I2[Fine-tune Stable Diffusion/Flux]
Q5{Condition type?} -->|Text| I3[SD + LoRA/DreamBooth]
Q5 -->|Class| I4[DiT/ADM]
V1 --> V2{Do you already have an image model?} -->|Yes| V3[AnimateDiff Motion Module]
V2 -->|No| V4[End-to-end video LDM training]
Key decision points:
| Scenario | Recommended approach | Rationale |
|---|---|---|
| Fast text-to-image | Fine-tune Stable Diffusion + LoRA | Lowest training cost, reuses the community ecosystem |
| High-resolution images | LDM / latent-space DiT, or cascaded diffusion | Latent space is compute-efficient; cascades preserve detail in pixel space |
| Video animation | AnimateDiff + personalized SD | No need to train a full video model; plug and play |
| Controllable long video | SVD-XT / segment generation + overlap, or a Diffusion Forcing-style method | Balances quality against VRAM limits |
| Text-to-3D (no 3D data) | SDS / VSD + a multi-view prior (such as MVDream) | Exploits the 2D prior for zero-shot 3D generation |
| Single image to 3D | SV3D, or multi-view diffusion + feed-forward reconstruction | Strong multi-view consistency, camera-controllable |
| Sound effects / ambient sound | AudioLDM | Efficient in latent space, supports zero-shot text editing |
| Music generation | MusicLDM / Stable Audio | Trained specifically for music, supports long audio and duration control |
| Molecule design | FlowSBDD / TargetDiff | Equivariance guarantees, flexible pocket conditioning |
| Protein backbone | RFdiffusion / Chroma | Strong structural priors, supports conditional generation |
| Discrete text | MDLM / SEDD (research), LLaDA (large scale) | Parallel generation, leading perplexity among diffusion models |
8.2 Comparing and Recommending Key Hyperparameters
Hyperparameters have a large effect on training stability and generation quality in diffusion models. The tables below summarize empirical values across modalities.
8.2.1 General Training Hyperparameters
| Hyperparameter | Image LDM | Video LDM | Audio LDM | 3D diffusion | Discrete diffusion | Notes |
|---|---|---|---|---|---|---|
| Learning rate | $1 \times 10^{-4}$ ~ $5 \times 10^{-4}$ | $5 \times 10^{-5}$ ~ $1 \times 10^{-4}$ | $3 \times 10^{-5}$ ~ $1 \times 10^{-4}$ | $1 \times 10^{-4}$ ~ $2 \times 10^{-4}$ | $1 \times 10^{-4}$ ~ $3 \times 10^{-4}$ | Video / fine-tuning usually lower; the original DiT paper uses a constant $1 \times 10^{-4}$ |
| Batch size | 256~2048 | 64~256 | 96~512 | 64~256 | 128~512 | Strictly limited by VRAM; gradient accumulation can help |
| Training steps | 300k~1M | 100k~500k | 100k~500k | 200k~1M | 300k~1M | Depends on dataset size |
| Diffusion steps $T$ | 1000 | 1000 | 1000 | 1000 | 1000 / continuous time | At sampling time, DDIM / DPM-Solver can reduce this to 20~200 |
| Noise schedule | Linear / Cosine / flow matching | Linear / EDM / flow matching | Linear | Cosine | Mask-rate schedule | At high resolution, watch for schedule shift (see §8.4.4) |
| EMA decay | 0.9999 | 0.9999 | 0.9999 | 0.9999 | 0.9999 | Strongly recommended; sample with the EMA weights |
| Gradient clipping | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | Prevents exploding gradients |
| Mixed precision | FP16 / BF16 | BF16 | FP16 / BF16 | FP32 / FP16 | BF16 / FP32 | BF16 has a wider numeric range and is less prone to overflow |
| CFG conditioning dropout rate | 0.1 | 0.1 | 0.1 | 0.1 | 0.1~0.15 | Randomly drop conditions during training (§3.3.1) |
EMA (exponential moving average) weights are updated as follows, and sampling uses $\theta_{\text{EMA}}$ rather than the in-training $\theta$:
$$ \theta_{\text{EMA}} \leftarrow \mu\, \theta_{\text{EMA}} + (1 - \mu)\, \theta $$where $\mu$ is the decay rate (e.g., 0.9999). Intuitively, this averages the parameters over roughly the last $1/(1 - \mu)$ steps (about 10,000 steps here), smoothing out training noise.
8.2.2 Sampling Hyperparameters
| Hyperparameter | Image generation | Video generation | Audio generation | 3D (SDS) | Recommended value |
|---|---|---|---|---|---|
| Guidance scale ($w$) | 3~7.5 | 1.5~3.0 (image-to-video) | 2.0~3.0 | ~100 (SDS); ~7.5 (VSD) | 7.5 by default for the SD series |
| Sampling steps | 20~50 | 25~50 | 20~200 | 10k~100k optimization iterations | Steps and quality are nonlinearly related |
| Sampler | DDIM / DPM++ / Euler | DDIM / DPM-Solver / EDM-Euler | DDIM | — | DPM++ 2M is a common, safe choice |
| Resolution | 512~1024 | 576~1024 | Determined by spectrogram size | Render 64~512 | Must match the training resolution |
| Random seed | Fixed / varied | Fixed | Fixed | Random viewpoint per step | A fixed seed supports reproducibility |
Note
The $w$ in this table follows the diffusers convention $\hat{\epsilon} = \epsilon_{\text{uncond}} + w\left(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}}\right)$, where $w = 1$ means no guidance. Some papers write $\hat{\epsilon} = \epsilon_{\text{cond}} + w'\left(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}}\right)$ instead; the two are related by $w = 1 + w'$. Check which convention a source uses before comparing its numbers.
8.2.3 Modality-Specific Parameters
| Modality | Special parameter | Typical value | Effect |
|---|---|---|---|
| Video | motion_bucket_id | 0~255 (SVD default 127) | Controls motion magnitude; higher means more vigorous motion |
| Video | fps micro-condition | 5~30 | Affects the perceived frame rate of the generated video |
| Video | Frame count $N_f$ | 14 / 25 (SVD / SVD-XT) | VRAM grows at least linearly with $N_f$; under 3D full attention, computation grows roughly as $N_f^2$ |
| Audio | Number of mel bands | 64~128 | Affects frequency resolution |
| Audio | hop length / window | 160 / 1024 (16 kHz) | Determines the time-frequency resolution trade-off |
| 3D SDS | guidance scale | ~100 | Much higher than for image generation: SDS is mode-seeking, and small CFG gives blurry, over-smoothed results |
| 3D SDS | Learning rate | on the order of $1 \times 10^{-3}$ | Optimizing NeRF / 3DGS parameters usually needs a higher learning rate |
| Discrete | Transition matrix type | absorb / uniform | Absorbing (masking) works best on text |
| Discrete | Sampling steps | tens ~ 1000 | Masked diffusion can trade off steps against quality flexibly |
8.3 Evaluation Metrics
Generative models have no “ground truth answer,” so we need a complementary set of metrics to measure fidelity (how much the samples resemble real data), diversity (whether the data distribution is covered), and conditional alignment (whether the prompt is followed).
FID (Fréchet Inception Distance): feed real and generated samples separately through an Inception-v3 network, take their features, fit a Gaussian to each — $\mathcal{N}(\mu_r, \Sigma_r)$ and $\mathcal{N}(\mu_g, \Sigma_g)$ — and compute the Fréchet distance between the two Gaussians:
$$ \mathrm{FID} = \left\lVert \mu_r - \mu_g \right\rVert_2^2 + \operatorname{Tr}\left( \Sigma_r + \Sigma_g - 2 \left( \Sigma_r \Sigma_g \right)^{1/2} \right) $$Lower FID is better, and it reflects both fidelity and diversity. It is sensitive to sample count and implementation details, so comparisons must use the same number of samples (commonly 50,000, i.e., FID-50K), the same reference set, and the same evaluation code.
Other commonly used metrics:
| Metric | What it measures | Modality | Notes |
|---|---|---|---|
| IS (Inception Score) | Fidelity and class diversity | Images | Higher is better; does not compare against real data |
| Precision / Recall | Fidelity and coverage, separately | Images | Splits FID’s two failure modes apart |
| CLIP score | Image-text alignment (cosine similarity of CLIP embeddings) | Text-to-image | Higher is better; often plotted against FID as a trade-off curve |
| FVD (Fréchet Video Distance) | Video version of FID, using a video feature network (I3D) | Video | Measures both per-frame quality and temporal coherence |
| FAD / FD (Fréchet Audio Distance) | Audio version of FID, using an audio feature network | Audio | Used by AudioLDM and others |
| BPC / perplexity (PPL) | Likelihood (or an upper bound on it) | Text / discrete | Diffusion models usually report the upper bound given by the ELBO |
| Human preference evaluation | Subjective quality and prompt adherence | All | Increasingly important in the era of large models |
8.4 Common Pitfalls and a Troubleshooting Guide
8.4.1 Unstable Training
Symptoms: loss diverges, NaN gradients appear, and generated samples collapse to a solid color or noise.
Root causes and fixes:
| Root cause | Diagnosis | Solution |
|---|---|---|
| Learning rate too high | Loss oscillates over the first 1000 steps, then diverges | Lower the learning rate to around $1 \times 10^{-4}$ and use warmup |
| Poor noise schedule | Loss stops decreasing late in training, or quality is poor at high / low noise levels | Switch to a cosine schedule, the EDM framework, or flow matching; check the time-step sampling distribution |
| Exploding gradients | Gradient norm far above normal | Enable gradient clipping (max_norm=1.0) and check initialization |
| Numeric overflow | Inf / NaN under FP16 | Switch to BF16, or compute the loss and normalization layers in FP32 |
| VAE numeric issues | Latents have values that are too large or overflow | Scale latents by the official factor (e.g., 0.18215 for SD) and check the VAE version |
| Poor temporal-layer training | Video training diverges | Freeze the spatial layers and train the temporal layers alone first, then fine-tune jointly |
Key equation: a common learning rate schedule with warmup (the “inverse square root” schedule from the original Transformer paper); let the learning rate at training step $k$ be:
$$ \mathrm{lr}(k) = \mathrm{lr}_{\text{base}} \cdot \min\left( \frac{k}{k_{\text{warmup}}},\; \left( \frac{k}{k_{\text{warmup}}} \right)^{-1/2} \right) $$The first $k_{\text{warmup}}$ steps ramp up linearly to $\mathrm{lr}_{\text{base}}$, then decay as $1/\sqrt{k}$. The training script in §8.6.4 uses another equally common schedule, “linear warmup + cosine decay.”
Notation:
| Symbol | Meaning |
|---|---|
| $\mathrm{lr}(k)$ | Learning rate at step $k$ |
| $\mathrm{lr}_{\text{base}}$ | Base (peak) learning rate |
| $k$ | Current training step (using $k$ rather than $t$, to avoid confusion with the diffusion time step) |
| $k_{\text{warmup}}$ | Number of warmup steps (usually 1000~10000) |
8.4.2 Loss of Diversity and Failure of Conditioning
Symptoms: generated samples lack diversity and keep producing a few similar results, or conditional generation ignores the text / class prompt. Strict “mode collapse” is rare in diffusion models; what’s more common is diversity loss caused by guidance and data.
Root causes and fixes:
| Root cause | Solution |
|---|---|
| Conditioning dropout rate set poorly | Too low and the unconditional branch is undertrained, so CFG works poorly; too high and prompt adherence weakens. Usually 10%~15% |
| Skewed training data distribution | Resample the data, balance categories / text lengths, and improve annotation quality |
| Guidance scale too high | Lower $w$ at sampling time (generally < 10 for images, < 3 for video), or use tricks such as CFG rescaling / guidance intervals |
| Insufficient model capacity | Increase the channel count or network depth |
| Overfitting a small dataset | Use dropout (0.1~0.2), data augmentation, and early stopping; when fine-tuning, watch for memorizing training samples |
| Not using EMA | Always sample with EMA weights to avoid parameter oscillation |
8.4.3 Out-of-Memory (OOM) Errors
Symptoms: CUDA out of memory; training or sampling cannot finish.
Optimization strategies (in recommended order of trying):
- Mixed-precision training: FP16 / BF16 roughly halves activation memory (FP16 needs gradient scaling);
- Gradient checkpointing: trade time for space, substantially reducing activation memory (commonly 30%~50%, depending on the model);
- Lower the batch size: pair it with gradient accumulation to preserve the effective batch size;
- Efficient attention: use memory-efficient attention implementations such as FlashAttention / xFormers;
- Lower the resolution: downsample images / video / spectrograms, or train at low resolution first and fine-tune at high resolution;
- Distributed strategies: for large models, shard parameters and optimizer states with FSDP, DeepSpeed ZeRO, etc.;
- Video-specific: reduce the frame count $N_f$ or spatial resolution, or generate long video in segments.
8.4.4 Poor Sampling Quality
Symptoms: blurry generations, structural errors, unreadable text, and flickering video.
Checklist:
- Are you using the EMA model rather than the raw in-training weights?
- Is the CFG scale too high (for images, $w > 15$ often causes oversaturation)?
- Are there enough sampling steps (DDIM usually needs more than 20)?
- Is conditioning encoded correctly (do the text tokenizer and CLIP / T5 model versions match training)?
- Does the VAE decoder match (video needs a temporal VAE, or flicker is likely)?
- Do the noise schedule and prediction type ($\epsilon$ / $v$ / velocity) match training (if training used a linear schedule, you can’t compute coefficients as cosine at sampling time)?
- Terminal SNR and starting time step: common schedules are not pure noise at $t = T$ (terminal SNR is not zero), and some sampler implementations don’t start from the last time step, causing a train-inference mismatch — for instance, Stable Diffusion struggles to produce very bright or very dark images. Lin et al.24 recommend rescaling the schedule to zero terminal SNR, switching to $v$-prediction, having the sampler always start from the last time step, and rescaling CFG to prevent overexposure;
- Resolution and noise schedule shift: the higher the resolution, the more correlated neighboring pixels are, and the less information the same noise level “destroys,” so high-resolution models usually need to shift the time-step / noise schedule toward higher noise (such as SD3’s resolution-dependent time-step shift).
8.5 Comparing Computational Resource Requirements
| Task | Model size | Training VRAM (single GPU) | Training time | Inference VRAM | Notes |
|---|---|---|---|---|---|
| Image LDM (512×512) | ~1B parameters | 40GB+ | Days (fine-tuning) ~ weeks (from scratch) | 8GB | Training from scratch needs multiple GPUs and large-scale data |
| Image DiT (256×256) | ~675M (XL/2) | 32GB+ | Days ~ 2 weeks | 6GB | Scales better than U-Net |
| Video SVD (25 frames) | ~1.5B | 80GB | Weeks | ~16–24GB | Requires multi-GPU training |
| AnimateDiff | SD1.5 U-Net (~0.86B) + motion module (hundreds of millions of parameters) | 24GB+ | Days | 12GB | Trains only the motion module |
| AudioLDM-L | ~0.74B | 32GB | Days | 8GB | Also includes the VAE and HiFi-GAN vocoder |
| DreamFusion-style (SDS) | 2D diffusion model + NeRF | 16GB+ | Tens of minutes ~ hours per sample | 16GB | Per-sample optimization, not model training |
| SV3D multi-view | ~1.5B | — | Uses a pretrained model | 16GB+ | Multi-view generation + 3D optimization |
| 3DGS + SDS | 2D diffusion model + 3DGS | 16GB | Minutes (e.g., ~2 minutes for DreamGaussian) | 16GB | Much faster than NeRF |
| Molecule FlowSBDD | Tens of millions of parameters | 16GB | Days | 4GB | Equivariant networks have fewer parameters |
| Protein RFdiffusion | — | — | Uses a pretrained model | 16GB | Needs a structure database |
| Discrete-text MDLM-style | ~100M–1B | 24GB+ | Days | 8GB | Diffusion Transformer |
Training times are rough estimates for a single 8×A100 machine ("—" means no reliable figure was found); actual times depend on dataset size, resolution, and parallelism strategy.
8.6 Core PyTorch Code Implementations
8.6.1 The DDPM Training Loop
The code below shows a complete DDPM training forward pass, including time-step sampling, forward noising, conditioning dropout (to set up CFG), and the noise prediction loss:
| |
8.6.2 The DDPM Sampling Loop (with CFG)
| |
Warning
The code uses $\hat{\epsilon} = \epsilon_{\text{uncond}} + w\left(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}}\right)$, which expands to $w\, \epsilon_{\text{cond}} + (1 - w)\, \epsilon_{\text{uncond}}$. The comment’s claim that this is “equivalent to $(1 + w)\, \epsilon_{\text{cond}} - w\, \epsilon_{\text{uncond}}$” is false — the latter corresponds to the other convention $\epsilon_{\text{cond}} + w\left(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}}\right)$ (see the note in §8.2.2). Under this code’s convention,
guide_w=0is purely unconditional generation,guide_w=1is ordinary conditional generation, and onlyguide_w>1gives enhanced guidance.
8.6.3 Flow Matching Training (Rectified Flow)
Flow Matching learns the velocity field of a linear interpolation between noise and data directly, training stably and yielding straighter sampling paths (see §6.4–§6.5 for the principles). The code below uses the time convention of §6.4 ($x_0$ is noise, $x_1$ is data):
| |
8.6.4 A Complete Training Script Framework
| |
Tip
This framework omits EMA. In real training, maintain an extra set of EMA weights (updated after every
optimizer.step()using the formula in §8.2.1), and use the EMA weights both when saving checkpoints and when sampling; PyTorch’storch.optim.swa_utils.AveragedModelor diffusers’EMAModelcan be used directly.
8.7 A Quality Checklist
Before deploying a diffusion model, you can use the following checklist to verify that the implementation is correct:
- Correct equations: is the training objective $\lVert \epsilon - \epsilon_\theta \rVert^2$ (DDPM noise prediction) or $\lVert v - v_\theta \rVert^2$ (Flow Matching)? Is the time-direction convention (which end is the data) consistent throughout?
- Notation consistency: is every variable in every equation defined in the text?
- CFG implementation: are conditions dropped with about 10% probability during training? Does sampling compute conditional and unconditional predictions together, and does the guidance formula match the convention used for $w$?
- EMA usage: is an EMA model maintained during training? Are EMA weights used for sampling?
- Noise schedule: do training and sampling use the same noise schedule? When switching between DDIM / DDPM, do the computations of $\alpha$ and $\bar{\alpha}$ match?
- Time encoding: is the time step $t$ fed to the network the same way as in training (e.g., $t/T$ or continuous time)?
- Gradient checks: is gradient clipping enabled? Is FP16 mixed precision paired with GradScaler?
- Dimension checks: do the broadcast dimensions of the condition embedding match the data dimensions?
- Evaluation: do metrics such as FID use a fixed sample count, reference set, and evaluation code?
- Reproducibility test: with the random seed fixed, does the same input produce the same output?
- VRAM monitoring: does the training script log peak memory usage?
References
Document generated: 2026-07-02 (revised 2026-11-19)
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., & Kreis, K. (2023). Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. CVPR. ↩︎
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., … & Rombach, R. (2023). Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. arXiv:2311.15127. ↩︎ ↩︎
Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., & Dai, B. (2024). AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning. ICLR. ↩︎
Ma, X., Wang, Y., Chen, X., Jia, G., Liu, Z., Li, Y.-F., Chen, C., & Qiao, Y. (2024). Latte: Latent Diffusion Transformer for Video Generation. arXiv:2401.03048 (later published in TMLR 2025). ↩︎ ↩︎
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., & Ramesh, A. (2024). Video Generation Models as World Simulators. OpenAI Technical Report. ↩︎
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., … & Tang, J. (2025). CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. ICLR. ↩︎
Chen, B., Martí Monsó, D., Du, Y., Simchowitz, M., Tedrake, R., & Sitzmann, V. (2024). Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. NeurIPS. ↩︎
Nichol, A., Jun, H., Dhariwal, P., Mishkin, P., & Chen, M. (2022). Point-E: A System for Generating 3D Point Clouds from Complex Prompts. arXiv:2212.08751. ↩︎
Jun, H., & Nichol, A. (2023). Shap-E: Generating Conditional 3D Implicit Functions. arXiv:2305.02463. ↩︎
Poole, B., Jain, A., Barron, J. T., & Mildenhall, B. (2023). DreamFusion: Text-to-3D using 2D Diffusion. ICLR (arXiv:2209.14988, 2022). ↩︎
Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., & Zhu, J. (2023). ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. NeurIPS. ↩︎
Voleti, V., Yao, C.-H., Boss, M., Letts, A., Pankratz, D., Tochilkin, D., … & Jampani, V. (2024). SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion. ECCV. ↩︎
Weng, H., Yang, T., Wang, J., Li, Y., Zhang, T., Chen, C. L. P., & Zhang, L. (2023). Consistent123: Improve Consistency for One Image to 3D Object Synthesis. arXiv:2310.08092. ↩︎
Zheng, C., & Vedaldi, A. (2024). Free3D: Consistent Novel View Synthesis without 3D Representation. CVPR. ↩︎
Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., & Plumbley, M. D. (2023). AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. ICML. ↩︎ ↩︎
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., & van den Berg, R. (2021). Structured Denoising Diffusion Models in Discrete State-Spaces. NeurIPS. ↩︎
Hoogeboom, E., Gritsenko, A. A., Bastings, J., Poole, B., van den Berg, R., & Salimans, T. (2022). Autoregressive Diffusion Models. ICLR. ↩︎
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., & Kuleshov, V. (2024). Simple and Effective Masked Diffusion Language Models. NeurIPS. ↩︎
Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., & Li, C. (2025). Large Language Diffusion Models. arXiv:2502.09992. ↩︎
Jo, J., & Hwang, S. J. (2025). Continuous Diffusion Model for Language Modeling. NeurIPS (arXiv:2502.11564). ↩︎ ↩︎
Zhang, D., Gong, C., & Liu, Q. (2024). Rectified Flow for Structure Based Drug Design. arXiv:2412.01174 (ELLIS ML4Molecules Workshop 2024). ↩︎ ↩︎
Schneuing, A., Harris, C., Du, Y., Didi, K., Jamasb, A., Igashov, I., … & Correia, B. (2024). Structure-based Drug Design with Equivariant Diffusion Models. Nature Computational Science (arXiv:2210.13695, 2022). ↩︎
Zeni, C., Pinsler, R., Zügner, D., Fowler, A., Horton, M., Fu, X., … & Xie, T. (2025). A generative model for inorganic materials design (MatterGen). Nature (arXiv:2312.03687). ↩︎
Lin, S., Liu, B., Li, J., & Yang, X. (2024). Common Diffusion Noise Schedules and Sample Steps are Flawed. WACV (arXiv:2305.08891). ↩︎
