Chapter 3: Conditional Generation and Latent Diffusion Models
3.1 From Unconditional to Conditional Generation
Diffusion models first appeared in the form of unconditional generation: learning a data distribution $p(x)$ and sampling from noise to produce new samples similar to the training data. In practice, however, users need finer control—specifying a class, a text description, a style reference, and so on. Conditional generation aims to learn a conditional distribution $p(x \mid c)$, where $c$ is the conditioning signal, which can be a class label, a text embedding, an image mask, or a spatial structure map.
In the diffusion model framework, the core idea of conditional generation is to inject the conditioning signal $c$ into the denoising network $\epsilon_\theta(x_t, t, c)$. Depending on how the injection is done, there are two main categories:
- Conditioning at training time: the condition $c$ is fed into the denoising network as an input during training, yielding a conditional model $\epsilon_\theta(x_t, t, c)$. Classifier-Free Guidance (CFG) is a guidance technique built on this kind of model.
- Conditioning at inference time: the diffusion model itself is left unchanged (it can be an unconditional model), and an external classifier or another differentiable objective guides the generation direction during sampling, as in Classifier Guidance.
Both can be understood in a unified way from the perspective of the score: conditional generation needs the conditional score $\nabla_{x_t} \log p(x_t \mid c)$, and the $\epsilon$-prediction differs from the score only by a scaling factor, $\epsilon_\theta(x_t, t) \approx -\sqrt{1 - \bar{\alpha}_t} \, \nabla_{x_t} \log p(x_t)$ (see Section 1.9). The various guidance methods below all essentially construct or amplify this conditional score in different ways.
| Symbol | Meaning |
|---|---|
| $x$ | Data sample (image) |
| $c$ | Conditioning signal (text, class, image, etc.) |
| $x_t$ | Noised sample at timestep $t$ |
| $\epsilon_\theta$ | Denoising network with parameters $\theta$ |
| $p(x \mid c)$ | Conditional data distribution |
3.2 Classifier Guidance
3.2.1 Principle and Formulas
Classifier Guidance was proposed by Dhariwal & Nichol (2021). Its core idea is to use a classifier trained on noisy images to guide the sampling process of a pretrained unconditional diffusion model1.
By Bayes’ rule, the conditional score function decomposes as:
$$\nabla_{x_t} \log p(x_t \mid c) = \nabla_{x_t} \log p(x_t) + \nabla_{x_t} \log p(c \mid x_t)$$(Take the log of $p(x_t \mid c) = p(c \mid x_t) \, p(x_t) / p(c)$ and then the gradient. Since $\log p(c)$ is independent of $x_t$, its gradient is 0.)
| Symbol | Meaning |
|---|---|
| $\nabla_{x_t} \log p(x_t \mid c)$ | Conditional score function (target) |
| $\nabla_{x_t} \log p(x_t)$ | Score function learned by the unconditional diffusion model |
| $\nabla_{x_t} \log p(c \mid x_t)$ | Gradient of the classifier log-likelihood (guidance signal) |
| $c$ | Condition (e.g., a class label) |
| $x_t$ | Noisy sample at timestep $t$ |
In the DDPM noise-prediction framework, substituting the relation $\epsilon = -\sqrt{1 - \bar{\alpha}_t} \times \text{score}$ into the equation above and multiplying the classifier gradient by a guidance scale $s$ gives the corrected noise prediction of Classifier Guidance:
$$\hat{\epsilon}_\theta(x_t, t) = \epsilon_\theta(x_t, t) - s \cdot \sqrt{1 - \bar{\alpha}_t} \, \nabla_{x_t} \log p_\phi(c \mid x_t)$$| Symbol | Meaning |
|---|---|
| $\hat{\epsilon}_\theta$ | Corrected noise prediction |
| $\epsilon_\theta(x_t, t)$ | Noise prediction of the unconditional diffusion model |
| $s$ | Guidance scale, $s > 0$; $s = 1$ corresponds to the strict Bayesian conditional score, while in practice $s > 1$ is often used |
| $\bar{\alpha}_t$ | Cumulative signal retention coefficient, $\bar{\alpha}_t = \prod_{i=1}^t (1 - \beta_i)$; $1 - \bar{\alpha}_t$ is the cumulative noise variance |
| $p_\phi(c \mid x_t)$ | Classifier with parameters $\phi$ that predicts the probability of condition $c$ on the noisy image $x_t$ |
| $\nabla_{x_t} \log p_\phi(c \mid x_t)$ | Classifier gradient, pointing toward image regions that make condition $c$ more likely |
Taking $s > 1$ amounts to sampling from a “sharper” distribution $\tilde{p}(x_t \mid c) \propto p(x_t) \, p_\phi(c \mid x_t)^s$: samples the classifier is more confident about get amplified more. Increasing $s$ therefore improves condition adherence and the fidelity of individual samples (in the FID/IS trade-off, IS goes up), but too high a value reduces sample diversity and can cause mode collapse or visual artifacts.
3.2.2 Implementation, Pros and Cons, and Use Cases
Implementation notes:
- Requires a classifier trained at every noise level $x_t$ (a time-dependent classifier), which adds training cost.
- The classifier typically uses a U-Net or ResNet architecture, with $x_t$ and an embedding of timestep $t$ as inputs.
- At sampling time, every denoising step requires an extra classifier-gradient computation, increasing inference cost.
Advantages:
- Adds conditional control to an existing unconditional model without retraining the diffusion model.
- Flexible: swapping in different classifiers implements different conditions (class, attributes, text–CLIP similarity, etc.).
Disadvantages:
- Requires training an extra noise classifier, which is costly.
- Training a classifier on noisy data is difficult, its accuracy is limited, and its gradients can be unstable.
- The gradient scale $s$ needs careful tuning; too large a value tends to produce oversaturated images or mode collapse.
- Adding the classifier gradient to the diffusion model’s score can introduce adversarial artifacts.
Use cases:
- Quickly adding class conditioning to an existing unconditional diffusion model (such as early DDPM or ADM).
- When fine control over specific attributes of generated samples is needed (ImageNet class, age, color, etc.).
3.2.3 PyTorch Code Example
| |
3.3 Classifier-Free Guidance (CFG)
3.3.1 Principle and Training Strategy
Classifier-Free Guidance (CFG) was proposed by Ho & Salimans (2022). It needs no extra classifier: by randomly dropping the conditioning signal during training, a single model learns both conditional and unconditional generation2.
Training strategy:
- With probability $p_{\text{drop}}$ (often around $10\%$), replace the condition $c$ with the null condition (null token) $\emptyset$.
- The model input becomes $\epsilon_\theta(x_t, t, c)$ (conditioned) or $\epsilon_\theta(x_t, t, \emptyset)$ (unconditioned).
- The training loss is the standard MSE:
| Symbol | Meaning |
|---|---|
| $p_{\text{drop}}$ | Condition-drop probability, typically $0.1$ |
| $\emptyset$ | Null condition (null token), indicating unconditional generation |
| $c_{\text{masked}}$ | The condition replaced by $\emptyset$ with probability $p_{\text{drop}}$ |
| $\epsilon$ | The actual Gaussian noise added |
| $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$ | Forward diffusion noising formula |
3.3.2 Sampling Formula
At inference time, CFG extrapolates between the conditional and unconditional predictions:
$$\hat{\epsilon}_\theta(x_t, t, c) = \epsilon_\theta(x_t, t, \emptyset) + w \cdot \left( \epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \emptyset) \right)$$Equivalent form:
$$\hat{\epsilon}_\theta(x_t, t, c) = w \cdot \epsilon_\theta(x_t, t, c) + (1 - w) \cdot \epsilon_\theta(x_t, t, \emptyset)$$Note the two notation conventions: the original Ho & Salimans paper writes $(1 + w') \, \epsilon_\theta(x_t, t, c) - w' \, \epsilon_\theta(x_t, t, \emptyset)$, where $w' = w - 1$, so $w' = 0$ in that paper is ordinary conditional sampling. This note and mainstream libraries such as diffusers use the $w$ above (i.e.,
guidance_scale), where $w = 1$ corresponds to ordinary conditional sampling. Check which convention a paper uses before reading it.
| Symbol | Meaning |
|---|---|
| $w$ | CFG guidance scale, typically $w \in [1, 10]$ |
| $\epsilon_\theta(x_t, t, c)$ | Noise prediction of the conditional branch |
| $\epsilon_\theta(x_t, t, \emptyset)$ | Noise prediction of the unconditional branch (empty text) |
| $\hat{\epsilon}_\theta$ | Final corrected noise prediction used for denoising |
At $w = 1$, this reduces to standard conditional sampling; at $w > 1$, it extrapolates along the “conditional direction”, improving condition adherence and image fidelity at the possible cost of diversity. $w = 0$ corresponds to unconditional generation.
Why “classifier-free” guidance? Rewriting CFG in the language of scores ($\epsilon$ differs from the score only by a negative scaling factor):
$$\nabla_{x_t} \log p(x_t) + w \left( \nabla_{x_t} \log p(x_t \mid c) - \nabla_{x_t} \log p(x_t) \right) = \nabla_{x_t} \log p(x_t \mid c) + (w - 1) \, \nabla_{x_t} \log p(c \mid x_t)$$where we used Bayes’ formula $\nabla_{x_t} \log p(c \mid x_t) = \nabla_{x_t} \log p(x_t \mid c) - \nabla_{x_t} \log p(x_t)$. The difference between the conditional and unconditional branches is itself the gradient of an implicit classifier, so CFG is equivalent to Classifier Guidance with guidance scale $w - 1$, but without training a separate noise classifier. Correspondingly, CFG approximately samples from $\tilde{p}(x_t \mid c) \propto p(x_t) \, p(c \mid x_t)^{w}$.
Negative prompt: practical systems often replace the unconditional branch’s $\emptyset$ with the text embedding of “unwanted content”, $c_{\text{neg}}$, giving $\hat{\epsilon} = \epsilon_\theta(x_t, t, c_{\text{neg}}) + w \left( \epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, c_{\text{neg}}) \right)$. The extrapolation direction then points not only toward $c$ but also away from $c_{\text{neg}}$.
3.3.3 Characteristics, Parameters, and Cautions
| Feature | Description |
|---|---|
| Training cost | No extra classifier; a single model is trained jointly, costing less than Classifier Guidance |
| Inference cost | Two forward passes per denoising step (conditional + unconditional), doubling the computation |
| Key parameters | $p_{\text{drop}}$ (at training, recommended $0.05 \sim 0.15$); $w$ (at inference, recommended $3.0 \sim 7.5$) |
| Cautions | Too high $w$ causes over-sharpened images, excessive contrast, and mode collapse; too low $p_{\text{drop}}$ weakens the unconditional branch |
| Use cases | The standard configuration for modern text-to-image models (SD, DALL·E 2, Imagen, SDXL, SD3, etc.); some distilled models (such as FLUX.1 [schnell]) distill guidance into the model and no longer need CFG at inference |
3.3.4 PyTorch Code Example
| |
3.4 Latent Diffusion Model (LDM) Architecture in Detail
3.4.1 Core Motivation: From Pixel Space to Latent Space
Original diffusion models (such as DDPM and ADM) operate directly in pixel space, which is enormously expensive in computation and memory for high-resolution images (e.g., $512 \times 512$). The Latent Diffusion Model (LDM) was proposed by Rombach et al. (2022); its key insight is that pixel space contains a great deal of redundant information, so the diffusion process can run in a compressed latent space3.
3.4.2 The Complete LDM Architecture
LDM consists of three core components:
- Perceptual compression model (an autoencoder, often called a VAE): the encoder $\mathcal{E}$ compresses the image $x$ into a latent $z = \mathcal{E}(x)$; the decoder $\mathcal{D}$ reconstructs the latent into an image $\tilde{x} = \mathcal{D}(z)$. Stable Diffusion uses a spatial downsampling factor of $f = 8$ (8× smaller on each side), e.g., $512 \times 512 \times 3 \to 64 \times 64 \times 4$, reducing the number of tensor elements by about 48×.
- Denoising U-Net: performs diffusion denoising in the latent space $z$, predicting noise $\epsilon_\theta(z_t, t, c)$.
- Conditioning encoder: for example, a CLIP text encoder maps text $y$ to an embedding $c = \tau_\theta(y)$, which is injected into the U-Net via cross-attention. For conditions aligned with image space (such as a mask for inpainting or a low-resolution image for super-resolution), LDM can also feed the condition directly into $z_t$ by concatenating it along the channel dimension.
Two-stage training: first train the autoencoder on its own, then freeze it and train the diffusion model in its latent space. The autoencoder is trained with an L1 reconstruction loss + a perceptual loss (LPIPS) + a patch-based adversarial loss, so the decoder produces sharp rather than blurry results. At the same time, a very light regularizer is applied to the latent space—either a KL penalty with a small weight (KL-reg, which Stable Diffusion uses) or vector quantization (VQ-reg)—to prevent the latent variance from becoming too large and the distribution too arbitrary.
Why $f = 8$? The LDM paper compared $f \in \{1, 2, 4, 8, 16, 32\}$: too small an $f$ leaves the diffusion model processing plenty of redundant pixels and trains slowly; too large an $f$ loses too much detail in compression, lowering the ceiling on generation quality. $f = 4 \sim 16$ strikes a good balance between efficiency and quality, with LDM-4 and LDM-8 performing best overall.
Latent scaling factor: the encoded latents do not have unit variance, while the diffusion noise schedule assumes data with roughly unit variance. So before training, the latents are multiplied by a constant $s_z \approx 1/\hat{\sigma}_z$ (the reciprocal of the latent standard deviation estimated from data) to bring their variance to about 1, and divided back before decoding. In Stable Diffusion 1.x/2.x this constant is the familiar 0.18215 in the code; the SDXL VAE uses 0.13025. Forgetting this scaling is one of the most common mistakes when implementing inference or training yourself.
graph LR
subgraph Pixel Space
X[Image x
3×512×512]
D[VAE Decoder 𝒟]
X2[Reconstructed image x̃
3×512×512]
end
subgraph Latent Space
E[VAE Encoder ℰ]
Z0[Latent z₀
4×64×64]
FD[Forward diffusion
noising to zₜ]
UN[U-Net εθ
denoising prediction]
ZT[Noise z_T
4×64×64]
end
subgraph Conditioning
TE[Text encoder τθ
CLIP / T5]
C[Conditioning embedding c]
end
X --> E
E --> Z0
Z0 --> FD
FD --> ZT
ZT --> UN
UN --> Z0
Z0 --> D
D --> X2
TE --> C
C -->|cross-attention| UN
3.4.3 Forward and Reverse Diffusion Processes
Forward process (in latent space):
$$z_t = \sqrt{\bar{\alpha}_t} \, z_0 + \sqrt{1 - \bar{\alpha}_t} \, \epsilon, \quad \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$$| Symbol | Meaning |
|---|---|
| $z_0 = s_z \, \mathcal{E}(x)$ | Clean latent after VAE encoding (and multiplication by the scaling factor $s_z$) |
| $z_t$ | Noised latent at timestep $t$ |
| $\bar{\alpha}_t$ | Cumulative signal retention coefficient (same as pixel-space diffusion); the corresponding signal-to-noise ratio (SNR) is $\bar{\alpha}_t / (1 - \bar{\alpha}_t)$ |
| $\epsilon$ | Standard Gaussian noise |
Reverse process (training objective):
$$\mathcal{L}_{\text{LDM}} = \mathbb{E}_{\mathcal{E}(x), \epsilon, t, c} \left[ \left\| \epsilon - \epsilon_\theta(z_t, t, c) \right\|^2 \right]$$| Symbol | Meaning |
|---|---|
| $\epsilon_\theta$ | Latent-space denoising U-Net |
| $c$ | Conditioning embedding (text, class, etc.) |
| $z_t$ | Forward-noised latent as defined above |
Generation process (inference):
- Sample $z_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$.
- Iterative denoising: from $t = T$ down to $1$, use the DDPM/DDIM (or the higher-order solvers in Chapter 5) update rule to obtain $z_{t-1}$, usually with CFG.
- Decode: $x = \mathcal{D}(z_0 / s_z)$.
As you can see, the diffusion part of LDM is mathematically identical to pixel-space diffusion, with $x$ simply replaced by $z$; all the gain comes from “doing diffusion in a smaller, more semantic space”.
3.4.4 PyTorch Implementation Skeleton
| |
3.5 Evolution of the Stable Diffusion Series
Stable Diffusion (SD) is the most influential family of open-source text-to-image generation models built on the LDM architecture. Its evolution reflects continuous improvements in the VAE, text encoder, denoising backbone, and training objective.
| Version | Release date | Key features | Denoising backbone | Text encoder | Latent space (at native resolution) |
|---|---|---|---|---|---|
| SD v1.x | 2022-08 | First open-source release, laid the foundation for the community ecosystem | U-Net (860M) | CLIP ViT-L/14 | 4 channels, $64 \times 64$ (512 px) |
| SD v2.x | 2022-11 | Stronger OpenCLIP, v-prediction model at 768 px | U-Net (865M) | OpenCLIP ViT-H/14 | 4 channels, $96 \times 96$ (768 px) / $64 \times 64$ (512 px) |
| SDXL | 2023-07 (v0.9 in 2023-06) | Larger U-Net, micro-conditioning, optional Refiner | U-Net (2.6B) | CLIP ViT-L + OpenCLIP ViT-bigG | 4 channels, $128 \times 128$ (1024 px) |
| SD 3 | 2024-06 (Medium open-sourced; paper 2024-03) | MMDiT architecture, Rectified Flow | MMDiT (2B for the open-source Medium, up to 8B in the paper) | CLIP-L + OpenCLIP-bigG + T5-XXL | 16 channels, $128 \times 128$ (1024 px) |
| SD 3.5 | 2024-10 | QK normalization and other improvements; available as Large / Large Turbo / Medium | MMDiT (Large 8.1B) / MMDiT-X (Medium 2.5B) | CLIP-L + OpenCLIP-bigG + T5-XXL | 16 channels, $128 \times 128$ (1024 px) |
| FLUX.1 | 2024-08 | Rectified Flow Transformer, 12B parameters | Dual-stream + single-stream Transformer (12B) | CLIP-L + T5-XXL | 16 channels, $128 \times 128$ (1024 px) |
3.5.1 SD v1.x and v2.x
- SD v1.x: based on CompVis’s implementation of the LDM paper, using a CLIP ViT-L/14 text encoder and a 4-channel VAE latent space. It has the most vibrant community ecosystem and spawned a large number of fine-tuned models and LoRAs.
- SD v2.x: replaced the text encoder with OpenCLIP ViT-H/14 (text tower about 354M parameters, roughly 3× the CLIP ViT-L text tower) and trained from scratch on an NSFW-filtered subset of LAION-5B. The 768 px version (768-v) uses the $v$-prediction objective (see Section 1.9) and introduced models for depth guidance (depth2img), dedicated inpainting, and 4× super-resolution.
3.5.2 SDXL
SDXL (Podell et al., 2023)4 is a major scaling up of the architecture:
- Larger U-Net: parameter count grew from about 0.86B to 2.6B; Transformer blocks were concentrated at the lower-resolution levels (0/2/10 per level), and the lowest $8\times$ downsampling level was removed.
- Dual text encoders: uses CLIP ViT-L and OpenCLIP ViT-bigG together (their text towers total about 817M parameters), concatenating the sequence features from each one’s penultimate layer along the channel dimension into 2048 dimensions for cross-attention. The pooled embedding from OpenCLIP ViT-bigG is also used as a global condition, added to the timestep embedding.
- Micro-conditioning: the original size of the training image $(h_{\text{orig}}, w_{\text{orig}})$ and the random crop coordinates $(c_{\text{top}}, c_{\text{left}})$ are Fourier-feature encoded and added to the timestep embedding. This means small images need not be discarded (about 39% of SDXL’s pretraining data is smaller than $256^2$), and at inference you can set $(c_{\text{top}}, c_{\text{left}}) = (0, 0)$ to avoid the “cropped-out subject” problem common in early SD.
- Multi-aspect-ratio training: fine-tuning with aspect-ratio bucketing, with native support for non-square outputs.
- Base + optional Refiner: the Refiner is a model specialized for the low-noise end of the same latent space, adding SDEdit-like detail refinement to the Base model’s output. It is not pixel-level cascaded super-resolution and can be omitted.
- Retrained VAE: the latent space is still 4 channels, but the VAE was retrained with a larger batch and EMA, giving better reconstruction quality.
3.5.3 SD 3 / SD 3.5: From U-Net to MMDiT
SD 3 is a paradigm shift in architecture:
- MMDiT (Multimodal Diffusion Transformer): abandons the U-Net entirely and builds a Transformer backbone on top of DiT5. Image patches ($2 \times 2$) and text tokens are concatenated into a single sequence; the two modalities use separate weight sets but are combined during attention computation, enabling bidirectional information flow6. The timestep and pooled text vector are injected through adaLN modulation.
- Rectified Flow7: connects data and noise with the linear interpolation path $z_t = (1 - t) \, x_0 + t \, \epsilon$; the network predicts a velocity $v = \epsilon - x_0$, replacing DDPM’s $\epsilon$-prediction objective. During training, $t$ is sampled from a logit-normal distribution so the model focuses more on intermediate noise levels. The closer the path is to a straight line, the smaller the discretization error for few-step sampling (see Chapter 6 for details on flow matching).
- Three text encoders: CLIP-L + OpenCLIP-bigG + T5-XXL (the T5 encoder has about 4.7B parameters). In the paper, T5’s sequence length is 77 tokens (extended to 256 late in SD 3.5 Medium training; diffusers defaults to 256 at inference). T5 mainly improves complex prompts and text rendering. During training, each of the three encoders is independently zeroed out with about 46% probability, so at inference you can drop T5 to save VRAM, at the cost of worse complex-prompt handling and text rendering.
- 16-channel VAE: compared with 4 channels, it preserves richer detail, significantly improving text rendering and fine-structure generation.
SD 3.5 (October 2024) did not change the basic paradigm. Its main improvements are adding QK normalization to the Transformer blocks to stabilize training; an 8.1B-parameter MMDiT for the Large version; Large Turbo, its few-step distilled version; and a Medium version (2.5B) using an improved MMDiT-X architecture with multi-resolution training.
3.5.4 Flux
FLUX.1 was released by Black Forest Labs in August 2024 (the company was founded by several core authors of LDM / Stable Diffusion) and was one of the leading open-source (open-weight) text-to-image models at the time of its release8:
- 12B-parameter Rectified Flow Transformer: also based on flow matching, and larger than SD3 Medium.
- Hybrid single-stream/dual-stream blocks: 19 dual-stream blocks (image and text use separate weight sets and interact in attention, like MMDiT) + 38 single-stream blocks (the concatenated sequence shares weights), balancing modality interaction and computational efficiency.
- Multi-axis RoPE: applies rotary position encoding to each coordinate axis of a token’s position separately, making it easier to handle different resolutions and aspect ratios.
- Multiple versions: FLUX.1 [pro] is available only through an API; [dev] is an open-weight guidance-distilled model that takes the guidance strength as an input and needs only one forward pass at inference; [schnell] uses Latent Adversarial Diffusion Distillation (LADD)9 to generate high-quality images in 1–4 steps without CFG.
3.6 Text Encoders and Cross-Attention
3.6.1 Evolution of Text Encoders
The text encoder converts the text prompt $y$ into a vector representation $c$, and its quality directly determines the semantic accuracy of the generated image.
| Encoder | Parameters (text portion only) | Context length | Features | Representative models |
|---|---|---|---|---|
| CLIP ViT-L/14 | ~123M | 77 | General vision–language alignment, open source | SD 1.x, SDXL, SD3, FLUX.1 |
| OpenCLIP ViT-H/14 | ~354M | 77 | Trained on LAION-2B; stronger retrieval and zero-shot ability than ViT-L | SD 2.x |
| OpenCLIP ViT-bigG/14 | ~695M | 77 | Larger capacity, style/composition understanding | SDXL, SD3 |
| T5-XXL (v1.1, encoder only) | ~4.7B | Depends on the model (77 during SD3 training; 512 for FLUX.1 [dev]) | Deep semantics, complex relationships, long text | SD3, FLUX.1 |
SDXL’s dual-encoder strategy: concatenate the penultimate-layer sequence features of CLIP-L and OpenCLIP-bigG along the channel dimension (for cross-attention), and use the pooled embedding from OpenCLIP-bigG as a global condition, balancing fine-grained semantics with overall style.
T5-XXL in SD3/FLUX.1: T5 was originally an encoder–decoder language model; here only its encoder is used to extract deep semantic representations. Compared with the CLIP text tower (trained only to align with images and capped at 77 tokens), T5 is more sensitive to word order, quantitative relationships, and spelling. It noticeably alleviates “attribute binding errors” and “subject neglect” in early models and improves the rendering of text within images.
3.6.2 Cross-Attention Mechanism
Cross-attention is the core mechanism for conditioning in LDM. In the Transformer blocks at multiple resolution levels of the U-Net, image features serve as the Query (Q) and text embeddings as the Key (K) and Value (V):
$$\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{Q K^{\top}}{\sqrt{d}} \right) V$$$$Q = z W_Q, \quad K = c W_K, \quad V = c W_V$$| Symbol | Meaning |
|---|---|
| $z$ | U-Net intermediate-layer image features (flattened), of shape $(hw, d_z)$ |
| $c$ | Text encoder output, of shape $(L, d_c)$; $d_c$ can differ from $d_z$ |
| $W_Q, W_K, W_V$ | Learnable projection matrices that map $z$ and $c$ to the same dimension $d$ |
| $d$ | Dimension of the Query/Key (the dimension of a single head in multi-head attention); $\sqrt{d}$ prevents the dot product from growing too large and saturating the softmax |
| $hw$ | Spatial resolution of the image feature map (flattened) |
| $L$ | Number of text tokens (e.g., 77 or 512) |
The cross-attention map $A = \text{softmax}\left( Q K^{\top} / \sqrt{d} \right) \in \mathbb{R}^{hw \times L}$ quantifies how strongly each image position responds to each text token (each row sums to 1). Visualizing the attention map shows how the model “reads” the text: for example, the token for “cat” produces a strong response in the regions of the image where a cat appears. Later image-editing methods such as Prompt-to-Prompt exploit this mechanism directly, achieving semantic editing by replacing or modifying attention maps (see Section 4.6).
Chapter 4: Controllable Generation: ControlNet, Adapters, and Image Editing
4.1 ControlNet: Precise Structural Control
4.1.1 Architecture Design
ControlNet was proposed by Zhang et al. (2023) to add spatial structural conditioning (such as edges, depth, pose, or semantic segmentation maps) to a pretrained text-to-image diffusion model while preserving the original model’s generative ability and knowledge10.
Core design ideas:
- Locking the original model: freeze all parameters of the pretrained diffusion model (e.g., SD 1.5), preserving the strong prior knowledge it gained from training on billions of images.
- Trainable copy: create a trainable copy of the original model’s Encoder and Middle Block to learn the conditional control.
- Zero convolution: connect the copy to the original model with $1 \times 1$ convolution layers whose weights and biases are both initialized to zero. Initially the copy outputs zero, so training starts without destabilizing the original model; as training proceeds, the parameters grow and the control signal is injected smoothly.
Formally, for a network block $\mathcal{F}(\cdot; \Theta)$ in the original model, the output after adding ControlNet is:
$$y_c = \mathcal{F}(x; \Theta) + \mathcal{Z}\left( \mathcal{F}\left( x + \mathcal{Z}(c; \Theta_{z1}); \Theta_c \right); \Theta_{z2} \right)$$| Symbol | Meaning |
|---|---|
| $\Theta$ | Original model parameters (frozen) |
| $\Theta_c$ | Parameters of the trainable copy (initialized from $\Theta$) |
| $\mathcal{Z}(\cdot; \Theta_{z1})$, $\mathcal{Z}(\cdot; \Theta_{z2})$ | The two zero convolutions |
| $c$ | Conditioning features |
On the first training step both zero convolutions output 0, so $y_c = \mathcal{F}(x; \Theta)$ and the model behaves exactly like the original. Zero initialization does not prevent the network from learning: the weight gradient of a zero convolution is proportional to its input, which is nonzero, so the weights leave zero after the first update.
graph TB
subgraph ControlNet Architecture
C[Conditioning image
e.g., edge/depth/pose]
CE[Conditioning encoder
4 conv layers]
ZC1[Zero convolution
1×1 Conv, initialized to 0]
ENC[Trainable encoder copy
Copy of SD Encoder]
MID[Trainable middle block copy
Copy of SD Middle Block]
ZC2[Zero convolution
1×1 Conv, initialized to 0]
ADD[Element-wise addition
into SD Decoder]
end
subgraph Frozen Stable Diffusion
SD_ENC[SD Encoder
frozen]
SD_MID[SD Middle Block
frozen]
SD_DEC[SD Decoder
frozen]
OUT[Final output]
end
C --> CE
CE --> ZC1
ZC1 --> ENC
ENC --> MID
MID --> ZC2
ZC2 --> ADD
SD_ENC --> SD_MID --> SD_DEC
SD_DEC --> ADD
ADD --> OUT
4.1.2 Training Method and Conditioning Mechanism
ControlNet’s training objective is noise prediction in latent space:
$$\mathcal{L}_{\text{ControlNet}} = \mathbb{E}_{z_0, t, c_{\text{text}}, c_{\text{cond}}, \epsilon} \left[ \left\| \epsilon - \epsilon_\theta(z_t, t, c_{\text{text}}, c_{\text{cond}}) \right\|^2 \right]$$| Symbol | Meaning |
|---|---|
| $c_{\text{cond}}$ | Features of the spatial condition (e.g., a Canny edge map or OpenPose skeleton) after encoding |
| $c_{\text{text}}$ | Text conditioning embedding |
| $\epsilon_\theta$ | Joint output of the original SD U-Net and the ControlNet copy |
The conditioning image (e.g., $512 \times 512$) first passes through a small 4-layer convolutional network ($4 \times 4$ kernels, stride 2) to become a $64 \times 64$ feature map matching the SD latent resolution, then a zero convolution adds it to the trainable encoder’s input. The outputs of the trainable encoder’s layers and middle block pass through zero convolutions and are added to the corresponding levels of the original SD U-Net decoder’s skip connections (and the middle block), forming multi-scale residual control.
During training, the text prompt is dropped for 50% of samples, forcing ControlNet to infer semantic content directly from the spatial condition and improving robustness. ControlNet can be trained on small datasets (<50k) and also scales to millions of samples.
4.1.3 Supported Control Types and Usage Recommendations
| Control type | Input | Use cases |
|---|---|---|
| Canny | Edge detection map | Precise contour control, line-art coloring |
| Depth | Depth estimation map | 3D sense of space, scene composition |
| Normal | Normal map | Surface orientation, lighting consistency |
| OpenPose | Human pose skeleton | Human pose, action generation |
| MLSD | Straight-line segment detection | Architecture, interior design |
| HED | Soft edges | Natural contours, artistic style |
| Scribble | User scribbles | Quick sketch-to-image |
| Semantic Segmentation | Semantic segmentation map | Scene layout, object placement |
| Reference (note) | Reference image | Style transfer, content consistency |
Note: Reference-only is a training-free attention-injection preprocessor in the WebUI ControlNet extension, not a separately trained ControlNet model; see Section 4.3.2.
Key parameters:
controlnet_conditioning_scale(control strength): $0.0$ to $2.0$; higher values enforce stricter control but may weaken text creativity.guess_mode: no prompt required; lets ControlNet infer on its own.
Cautions:
- ControlNet must be paired with a matching SD version (an SD 1.5 ControlNet cannot be used with SDXL).
- Multiple ControlNets can be stacked (e.g., Canny + Depth + OpenPose), but their control strengths must be balanced.
- The control image resolution should match the target output resolution (typically $512 \times 512$ for SD 1.5).
4.1.4 PyTorch Code Example
| |
4.2 T2I-Adapter: Lightweight Structural Adaptation
4.2.1 Design Motivation and Architecture
T2I-Adapter was proposed by Mou et al. (Tencent ARC Lab et al., AAAI 2024) with the goal of providing a lighter and more efficient adaptation method than ControlNet11. Its key insights are:
- No need to copy the entire encoder: ControlNet’s trainable copy has a large parameter count (about 360M for SD 1.5 and 1.25B for SDXL), making training and inference expensive.
- Extract features only once: the Adapter’s only input is the conditioning image; it does not depend on the noised $z_t$ or timestep $t$, so it runs just once during the whole sampling process and its output is reused at every denoising step. ControlNet’s copy, by contrast, takes $z_t$ as input and must be recomputed at every step.
T2I-Adapter (SD 1.5 version) contains only about 77M parameters (roughly 1/5 of the same-version ControlNet; the SDXL version has about 79M) and is built from a simple downsampling convolutional network:
- Input: a conditioning image (e.g., an edge map, depth map, or sketch).
- Processing: four convolutional downsampling blocks extract multi-scale features (at resolutions $64\times64$, $32\times32$, $16\times16$, and $8\times8$).
- Injection: these features are added directly to the outputs of the frozen SD U-Net encoder’s layers as
down_block_additional_residuals.
graph LR
C[Conditioning image
e.g., sketch/edge]
A1[Conv + Down
64×64]
A2[Conv + Down
32×32]
A3[Conv + Down
16×16]
A4[Conv + Down
8×8]
SD[Frozen SD U-Net]
D1[Encoder layer 1]
D2[Encoder layer 2]
D3[Encoder layer 3]
D4[Encoder layer 4]
C --> A1 --> A2 --> A3 --> A4
A1 -->|+ residual| D1
A2 -->|+ residual| D2
A3 -->|+ residual| D3
A4 -->|+ residual| D4
D1 --> D2 --> D3 --> D4 --> SD_DEC[Decoder]
4.2.2 Features and Use Cases
| Feature | T2I-Adapter | ControlNet |
|---|---|---|
| Parameters | ~77M (SD 1.5) / 79M (SDXL) | ~361M (SD 1.5) / 1251M (SDXL) |
| Training speed | Faster (fewer parameters) | Slower |
| Inference cost | Very low (runs once per sampling) | Higher (the copy encoder runs at every denoising step) |
| Control precision | Slightly lower than ControlNet | Higher (pixel-level control) |
| Composability | Multiple Adapters can simply be added together | Multiple ControlNets require parallel inference |
| Use cases | Resource-constrained environments, rapid prototyping, style control | Precise structural control, production-grade applications |
T2I-Adapter supports many conditions: Sketch, Canny, Lineart, OpenPose, Depth (Zoe/MiDaS), Color, and Style. Multiple Adapters can be combined linearly at inference (adding their weights), enabling composite control such as “edge + color + pose”.
4.2.3 PyTorch Code Example
| |
4.3 IP-Adapter and Reference-only Image Conditioning
4.3.1 IP-Adapter’s Decoupled Cross-Attention
IP-Adapter (Image Prompt Adapter) was proposed by Ye et al. (2023) to address the image prompt problem: how to guide the style, content, or human identity of a generated image with a single reference image, without retraining the whole model12.
The core innovation is Decoupled Cross-Attention:
- Leave the original text cross-attention layers unchanged (frozen).
- The reference image first passes through a frozen CLIP image encoder to extract a global embedding, which a small trainable projection network maps to several (4 by default) “image tokens”.
- In every U-Net cross-attention block, add an image branch: it shares the Query with the text branch and learns new Key and Value projection matrices only for the image tokens.
- The output is a weighted sum of the two cross-attentions:
| Symbol | Meaning |
|---|---|
| $Q = z W_Q$ | Image feature query (from the U-Net) |
| $K_T = c_T W_K$, $V_T = c_T W_V$ | Text-conditioning Key/Value (frozen) |
| $K_I = c_I W_K'$, $V_I = c_I W_V'$ | Image-conditioning Key/Value (only $W_K', W_V'$ are trainable) |
| $c_I$ | Reference image embedding extracted by the CLIP image encoder |
| $\lambda$ | Image-conditioning strength weight |
| $z$ | U-Net intermediate features |
IP-Adapter adds only about 22M parameters. During training it freezes the U-Net, VAE, text encoder, and image encoder, training only the image projection network and the new $W_K', W_V'$, which makes it highly efficient. The reason to “decouple” is that if image tokens were appended directly after text tokens and shared the original $W_K, W_V$, image features would be hard to align with projections trained for text, and results would be much worse. At inference, setting $\lambda = 0$ reduces it to the original text-to-image model.
4.3.2 Reference-only and FaceID Variants
- IP-Adapter-FaceID: specialized for preserving human identity. It uses a face encoder (such as InsightFace) to extract an identity embedding in place of the CLIP image encoder, achieving more precise human consistency.
- Reference-only: a training-free control method that injects the reference image’s attention features (K/V) into the target generation branch during denoising, achieving zero-shot style/content transfer. It is commonly used as the “Reference” control type in the Stable Diffusion WebUI.
4.3.3 Use Cases and Combination Strategies
| Scenario | Recommended method | Description |
|---|---|---|
| Style transfer | IP-Adapter + ControlNet | ControlNet controls structure, IP-Adapter controls style |
| Human consistency | IP-Adapter-FaceID | Preserves facial identity while changing pose/background |
| Virtual try-on | IP-Adapter + GarmentNet | References a clothing image to generate a worn look |
| Brand visual consistency | IP-Adapter + text prompt | Keeps the same color palette/style across multiple generated images |
IP-Adapter is naturally compatible with ControlNet and T2I-Adapter and can be used alongside them. For example, ControlNet(Canny) + IP-Adapter(style reference) achieves precise control over “specified structure + specified style”.
4.3.4 PyTorch Code Example
| |
4.4 Inpainting and Outpainting
4.4.1 Inpainting: Local Repainting
Inpainting (image restoration / local repainting) aims to generate new content consistent with its surroundings in the region specified by a mask $M$. In the diffusion model framework, there are two ways to implement inpainting:
Method 1: dedicated inpainting models (e.g., SD-Inpainting, SDXL-Inpainting)
- Modify the U-Net’s input channels: concatenate the noised latent $z_t$, the latent encoding of the original image after masking $z_{\text{masked}}$ (the masked region is first erased, then passed through the VAE), and the mask downsampled to latent resolution $M_z$ along the channel dimension as the input.
- The number of input channels changes from 4 to 9 ($z_t: 4$ + $z_{\text{masked}}: 4$ + $M_z: 1$); the weights for the new channels are initialized to zero, and training continues from the original model.
- During training, the model learns to generate content inside the masked region while preserving the original structure outside it.
Method 2: repainting sampling (Repainting / Diffusion Inpainting)
- Uses a standard (non-dedicated) diffusion model with no extra training. After each denoising step ($t \to t-1$), the unmasked region is forcibly replaced by a noised version of the original image at the same noise level:
| Symbol | Meaning |
|---|---|
| $M_z$ | Binary mask in latent space (1 for regions to generate, 0 for regions to keep) |
| $\odot$ | Element-wise multiplication |
| $z_{t-1}^{\text{pred}}$ | Latent obtained by one denoising step from $z_t$ |
| $z_{\text{orig}}$ | VAE encoding of the original image |
The problem with this approach is that the masked region only “sees” noised versions of the known region, so the two sides may not be semantically consistent. RePaint (Lugmayr et al., 2022)13 adds resampling on top: after each step, it re-noises the result and repeats the “denoise–noise” cycle several times, letting information from the known region propagate fully into the masked region and markedly improving boundary consistency.
Key parameters:
denoising_strength(repaint amount): $0.0$ fully preserves the original image, $1.0$ regenerates it completely; $0.5 \sim 0.8$ is recommended. It works on the same principle as SDEdit (see Section 4.6.1): first noise the original image to $t_0 \approx \text{strength} \cdot T$, then start denoising only from $t_0$.- Mask-edge feathering: avoids harsh boundaries between generated and original content.
4.4.2 Outpainting: Image Extension
Outpainting (image extension) enlarges the canvas and generates new content outside the original image’s borders. How it works:
- Expand the canvas to the target size, placing the original image at the center (or at an offset).
- Treat the blank area as a “full mask” and the original image area as “no mask”.
- Use an inpainting model, or a standard diffusion model with a mask constraint, to generate new pixels in the blank area that are coherent with the original content. Since the blank area has no original content to preserve, this step usually needs full strength (strength $= 1.0$) generation, and a dedicated inpainting model is preferable.
- After generation, you can do a local repaint at lower strength (e.g., $0.3 \sim 0.5$) near the seams to blend the boundaries smoothly.
4.4.3 PyTorch Code Example (Inpainting)
| |
4.5 LoRA and DreamBooth: Fine-Tuning Diffusion Models
4.5.1 LoRA (Low-Rank Adaptation)
LoRA was originally proposed by Hu et al. (2022)14 for LLMs and was later widely applied to fine-tuning diffusion models. Its core assumption is that the weight change $\Delta W$ from fine-tuning is low-rank. So while freezing the original weights $W_0 \in \mathbb{R}^{d \times k}$, it trains only the low-rank factorization matrices $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, with rank $r \ll \min(d, k)$:
$$W = W_0 + \frac{\alpha}{r} B A$$| Symbol | Meaning |
|---|---|
| $W_0$ | Pretrained weight matrix (frozen) |
| $B, A$ | Trainable low-rank matrices, $r$ typically $4 \sim 64$; $A$ is initialized with random Gaussians and $B$ with zeros, so $\Delta W = 0$ at the start of training |
| $\alpha$ | Scaling hyperparameter (called lora_alpha in peft); the effective scale is $\alpha / r$. Common choices are $\alpha = r$ (scale 1) or $\alpha = 2r$ |
| $W$ | Effective forward-pass weights |
A $d \times k$ matrix originally has $dk$ parameters, while LoRA trains only $r(d + k)$. In diffusion models, LoRA is typically applied to $W_Q, W_K, W_V, W_O$ of the U-Net (or DiT) attention layers (self-attention and cross-attention), sometimes also to feed-forward layers and the text encoder. Its advantages include:
- Parameter-efficient: typically trains less than $1\%$ of the original model’s parameters (SD 1.5 LoRA files are usually a few MB to a few hundred MB, depending on rank and the layers covered).
- Pluggable: trained LoRA weights can be loaded or unloaded dynamically at inference without modifying the base model; you can also merge $\frac{\alpha}{r} B A$ directly into $W_0$, adding no inference cost at all.
- Composable: multiple LoRAs can be stacked (e.g., “person LoRA + style LoRA”).
Training tips:
- Learning rate: $10^{-4} \sim 5 \times 10^{-4}$ (usually higher than full-model fine-tuning).
- Rank $r$: $4 \sim 16$ suits specific objects/styles; $32 \sim 64$ suits complex concepts.
- Dataset: $5 \sim 20$ high-quality images are enough to train a personalized LoRA.
4.5.2 DreamBooth: Instance-Level Personalization
DreamBooth, proposed by Google Research, aims to inject a specific concept (such as a pet or a person) into a diffusion model using very few images (typically 3–5), so that the model can generate images of that concept in different scenes using a unique identifier token15.
Training strategy:
- Use a rare token (such as
sks) as the concept identifier, combined with the class name (e.g.,a photo of sks dog). - Optimize two losses at once:
- Instance reconstruction loss: on the 3–5 subject images, run standard denoising training conditioned on
a photo of sks dog, binding the identifiersksto that subject. - Prior preservation loss: first use the original model to generate a batch of same-class images with
a photo of dog, then use them as supervision during training. This prevents “language drift” (drawing every dog as this one) and the forgetting of within-class diversity.
- Instance reconstruction loss: on the 3–5 subject images, run standard denoising training conditioned on
In formula form (using this note’s $\epsilon$-prediction notation):
$$\mathcal{L}_{\text{DB}} = \mathbb{E}\left[ \left\| \epsilon - \epsilon_\theta(z_t, t, c_{\text{inst}}) \right\|^2 \right] + \lambda \, \mathbb{E}\left[ \left\| \epsilon' - \epsilon_\theta(z'_{t'}, t', c_{\text{cls}}) \right\|^2 \right]$$| Symbol | Meaning |
|---|---|
| $z_t$ | Noised version of the subject image’s latent |
| $z'_{t'}$ | Noised version of a class image’s latent generated by the original model |
| $c_{\text{inst}}$, $c_{\text{cls}}$ | Text embeddings of “a [V] dog” and “a dog” |
| $\lambda$ | Weight of the prior preservation loss (set to 1 in the paper) |
- The original DreamBooth performs full-parameter fine-tuning of the whole model (producing complete weights, about 2–4 GB for SD 1.5). Modern practice usually adopts DreamBooth + LoRA (optimizing the DreamBooth objective with a LoRA parameterization), greatly reducing VRAM and storage requirements.
| Method | Trained parameters | VRAM requirement | File size | Use cases |
|---|---|---|---|---|
| Full DreamBooth | Entire U-Net + text encoder | 24GB+ | ~4GB | Highest quality, full style learning |
| DreamBooth + LoRA | Low-rank matrices in attention layers | 12-16GB | 3-50MB | Balances quality and efficiency; recommended |
| Textual Inversion | Only the new word’s text embedding | 8GB | A few KB | Quick concept binding; lower quality |
| LoRA Only | Low-rank matrices | 8-12GB | 3-50MB | Style transfer, person fine-tuning |
4.5.3 Textual Inversion: Learning Just One “New Word”
Textual Inversion (Gal et al., 2023)16 goes to the opposite extreme: all model weights are frozen, and a single new word-embedding vector is learned for the new concept. Concretely, add a placeholder $S_*$ to the vocabulary (e.g., <my-cat>) and optimize its embedding $v_*$ using 3–5 images:
| Symbol | Meaning |
|---|---|
| $y$ | Prompt template containing the placeholder, e.g., “a photo of $S_*$” |
| $\tau_\theta(y; v)$ | Text encoding after replacing $S_*$’s embedding with $v$ |
| $v$ | The only optimized parameter (a 768-dimensional vector in SD 1.x) |
Because only one embedding is learned, the output is just a few KB and can be freely combined with other prompt words. But its expressive power is limited by “how much the original model can describe with one word”, and it usually reconstructs complex subjects less faithfully than DreamBooth or LoRA.
4.5.4 PyTorch Code Example (LoRA Training)
| |
4.6 Diffusion-Based Image Editing: SDEdit, DDIM Inversion, Prompt-to-Prompt, and InstructPix2Pix
The methods so far mainly address “how to control generation”; this section focuses on “how to modify an existing image”.
4.6.1 SDEdit: Noise Then Denoise (How img2img Works)
SDEdit (Meng et al., 2022)17 has a very simple idea: noise the input image (a photo or a rough scribble) to some intermediate time $t_0$, then denoise normally starting from $t_0$:
$$z_{t_0} = \sqrt{\bar{\alpha}_{t_0}} \, z_{\text{input}} + \sqrt{1 - \bar{\alpha}_{t_0}} \, \epsilon, \quad \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$$Noising erases detail but preserves the overall layout and colors; denoising then “fills” the detail back in, guided by the diffusion prior, to produce a realistic image. $t_0$ controls the trade-off between realism and faithfulness: larger $t_0$ gives results that look more natural and match the text better, but differ more from the input; smaller $t_0$ stays closer to the input. The img2img strength parameter in WebUI and diffusers is $t_0 / T$.
4.6.2 DDIM Inversion and Null-text Inversion
If you want to “change just one thing and leave everything else as is”, you first need to find an initial noise that exactly reconstructs the original image. DDIM ($\eta = 0$) is deterministic ODE sampling (see Section 2.2); you can run it “backward”, going step by step from $z_0$ to $z_T$:
$$z_{t+1} = \sqrt{\bar{\alpha}_{t+1}} \, \hat{z}_0(z_t) + \sqrt{1 - \bar{\alpha}_{t+1}} \, \epsilon_\theta(z_t, t), \qquad \hat{z}_0(z_t) = \frac{z_t - \sqrt{1 - \bar{\alpha}_t} \, \epsilon_\theta(z_t, t)}{\sqrt{\bar{\alpha}_t}}$$Here $\epsilon_\theta(z_t, t)$ approximates the noise that should be computed at $z_{t+1}$; with a small enough step size the error is minor. But when using a large CFG scale in a text-conditional model, errors accumulate and the reconstruction visibly deviates from the original. Null-text Inversion (Mokady et al., 2023)18 works as follows: it takes the unguided DDIM inversion trajectory as an “anchor” and, at each timestep, optimizes the null text embedding $\emptyset_t$ used by the unconditional branch so that the CFG-guided sampling trajectory closely follows the anchor trajectory. This achieves near-perfect reconstruction while preserving editability.
4.6.3 Prompt-to-Prompt: Editing Cross-Attention Maps
Hertz et al. (2023)19 observed that an image’s spatial layout is determined mainly by the cross-attention maps (see Section 3.6.2). So if you keep the initial noise fixed and inject the original prompt’s attention maps when generating with a new prompt, you can keep the composition unchanged and change only the semantics. There are three common operations:
| Operation | Example | How it works |
|---|---|---|
| Word substitution | “a cat” → “a dog” | For the first few steps, replace the new word’s attention map with the original word’s attention map |
| Adding a phrase | “a car” → “a red car” | Keep the original attention maps for existing tokens; new tokens use their own attention maps |
| Attention reweighting | Make “furry” more pronounced | Scale a token’s attention map up or down |
Combined with Null-text Inversion, Prompt-to-Prompt can also edit real photos.
4.6.4 InstructPix2Pix: Editing by Instruction
InstructPix2Pix (Brooks et al., 2023)20 directly trains a conditional diffusion model that maps “image + editing instruction → edited image”. Its training data is generated automatically: first, a fine-tuned GPT-3 produces “original caption / editing instruction / new caption” triplets; then Stable Diffusion + Prompt-to-Prompt generates the corresponding image pairs. The model concatenates the input image’s latent along the channel dimension onto $z_t$, and at sampling time applies separate guidance scales to the image condition $c_I$ and the text instruction $c_T$:
$$\tilde{\epsilon}_\theta(z_t, c_I, c_T) = \epsilon_\theta(z_t, \emptyset, \emptyset) + s_I \left( \epsilon_\theta(z_t, c_I, \emptyset) - \epsilon_\theta(z_t, \emptyset, \emptyset) \right) + s_T \left( \epsilon_\theta(z_t, c_I, c_T) - \epsilon_\theta(z_t, c_I, \emptyset) \right)$$| Symbol | Meaning |
|---|---|
| $s_I$ | Image guidance scale: larger values keep the result closer to the input image |
| $s_T$ | Text guidance scale: larger values make the edit stronger |
| $\emptyset$ | The corresponding condition is set to null (during training, each of the two conditions is randomly dropped separately) |
This is a natural generalization of CFG to the multi-condition case. At inference it needs only a single forward diffusion sampling pass—no inversion or per-image optimization.
| Method | Training required? | Inversion required? | Edits it excels at |
|---|---|---|---|
| SDEdit / img2img | No | No (direct noising) | Stylization, sketch-to-image, full repainting |
| Prompt-to-Prompt (+ Null-text Inversion) | No | Needed when editing real photos | Composition-preserving semantic substitution, attribute modification |
| InstructPix2Pix | Yes (one-time training) | No | Natural-language instruction-based editing |
| Inpainting (Section 4.4) | Optional | No | Local region repainting |
4.7 Method Comparison and Selection Guide
| Method | Control granularity | Training cost | Inference cost | Parameter changes | Typical applications |
|---|---|---|---|---|---|
| Classifier Guidance | Attribute/class | High (requires classifier training) | Medium | None | Conditioning an unconditional model |
| CFG | Text/class | Low (drop conditioning during training) | High (2× forward passes) | None | Standard in all modern T2I models |
| ControlNet | Spatial structure | Medium (copy training) | High (copy runs every step) | Adds a copy | Precise pose/edge/depth control |
| T2I-Adapter | Spatial structure | Low (lightweight network) | Low | None (adds residuals) | Fast structural control, resource-constrained settings |
| IP-Adapter | Image style/content | Low (22M parameters) | Low | Adds cross-attention layers | Style transfer, human consistency |
| LoRA | Concept/style/person | Low (1% of parameters) | None (load weights) | Low-rank update | Personalized generation, community models |
| DreamBooth | Instance concept | Medium (full or LoRA) | None | Full or low-rank | Specific objects/pets/people |
| Textual Inversion | Concept/style | Low (learns one embedding) | None | None (adds a word embedding) | Lightweight concept binding |
| Inpainting/Outpainting | Pixel-level | None (repaint sampling) or medium (dedicated model) | Medium | None | Image restoration, canvas extension |
| SDEdit / P2P / InstructPix2Pix | Whole image / semantic | None (SDEdit, P2P) or one-time training (IP2P) | Medium (P2P needs inversion for real images) | None | Stylization, semantic substitution, instruction editing |
Selection recommendations:
- Precise structural control (e.g., making a character strike a specific pose): ControlNet + OpenPose/Depth.
- Fast style transfer: IP-Adapter + text prompt.
- Personalized people/objects: DreamBooth + LoRA (3–10 images are enough).
- Modifying an existing image: use SDEdit (img2img) for overall stylization, Prompt-to-Prompt for composition-preserving semantic substitution, InstructPix2Pix for natural-language instructions, and Inpainting for local changes.
- Resource-constrained environments: use T2I-Adapter instead of ControlNet, or just fine-tune with LoRA.
- Complex multi-condition generation: combine
ControlNet(structure) + IP-Adapter(style) + LoRA(person).
References
Dhariwal, P., & Nichol, A. (2021). Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems (NeurIPS), 34, 8780-8794. ↩︎
Ho, J., & Salimans, T. (2022). Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. ↩︎
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10684-10695. ↩︎
Podell, D., English, Z., Lacey, K., et al. (2024). SDXL: Improving latent diffusion models for high-resolution image synthesis. ICLR (arXiv:2307.01952, 2023). ↩︎
Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 4195-4205. ↩︎
Esser, P., Kulal, S., Blattmann, A., et al. (2024). Scaling rectified flow transformers for high-resolution image synthesis. Proceedings of the 41st International Conference on Machine Learning (ICML). ↩︎
Liu, X., Gong, C., & Liu, Q. (2023). Flow straight and fast: Learning to generate and transfer data with rectified flow. International Conference on Learning Representations (ICLR) (arXiv:2209.03003, 2022). ↩︎
Black Forest Labs. (2024). FLUX.1 model release (official blog and GitHub repository black-forest-labs/flux; no standalone technical report has been published for this series). ↩︎
Sauer, A., Boesel, F., Dockhorn, T., Blattmann, A., Esser, P., & Rombach, R. (2024). Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403.12015. ↩︎
Zhang, L., Rao, A., & Agrawala, M. (2023). Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 3836-3847. ↩︎
Mou, C., Wang, X., Xie, L., et al. (2024). T2I-Adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. AAAI (arXiv:2302.08453, 2023). ↩︎
Ye, H., Zhang, J., Liu, S., et al. (2023). IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721. ↩︎
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., & Van Gool, L. (2022). RePaint: Inpainting using denoising diffusion probabilistic models. CVPR. ↩︎
Hu, E. J., Shen, Y., Wallis, P., et al. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR). ↩︎
Ruiz, N., Li, Y., Jampani, V., et al. (2023). DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 22500-22510. ↩︎
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2023). An image is worth one word: Personalizing text-to-image generation using textual inversion. ICLR. ↩︎
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., & Ermon, S. (2022). SDEdit: Guided image synthesis and editing with stochastic differential equations. ICLR. ↩︎
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., & Cohen-Or, D. (2023). Null-text inversion for editing real images using guided diffusion models. CVPR. ↩︎
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., & Cohen-Or, D. (2023). Prompt-to-prompt image editing with cross attention control. ICLR. ↩︎
Brooks, T., Holynski, A., & Efros, A. A. (2023). InstructPix2Pix: Learning to follow image editing instructions. CVPR. ↩︎
