Diffusion Models Explained – DDPM, DDIM, Score-Based Models and Stable Diffusion

Generative modelling — learning to produce new data samples from the same distribution as the training data — has been transformed by diffusion models. From 2022 onwards, Stable Diffusion, DALL-E, Midjourney, and Imagen displaced GANs as the state-of-the-art generative architecture for images, and diffusion-based models have since extended to audio (AudioLDM), video (Sora, Stable Video Diffusion), molecules (DiffDock), and protein structure (RFDiffusion). For data scientists, diffusion models matter as both a technical topic and a practical tool: they are the standard approach for synthetic data generation, image augmentation for computer vision tasks, and domain adaptation. This guide covers the complete diffusion model framework — the forward and reverse process mathematics, DDPM and the noise prediction objective, DDIM for accelerated sampling, score-based generative models, Latent Diffusion and Stable Diffusion, and text conditioning via Classifier-Free Guidance.

The Forward and Reverse Diffusion Process

The generative modelling insight behind diffusion models is deceptively simple: if you can learn to reverse a known noising process, you can generate new data by starting from noise and applying the learned reverse process. The forward process q(x_{1:T} | x_0) gradually destroys the structure of a data sample x_0 (a real image) over T time steps by adding small amounts of Gaussian noise at each step: q(x_t | x_{t-1}) = N(x_t; √(1−β_t) x_{t-1}, β_t I), where β_t is a small positive variance schedule that increases from β_1 ≈ 0.0001 to β_T ≈ 0.02. By the reparameterisation trick (exploiting the closure of Gaussians under composition), x_t can be sampled directly from x_0 in one step: x_t = √ᾱ_t x_0 + √(1−ᾱ_t) ε, where ε ~ N(0,I) and ᾱ_t = Π_{s=1}^t (1−β_s). This means at large T (T=1000 in DDPM), x_T is approximately isotropic Gaussian noise regardless of x_0 — the original image information is completely destroyed. The Gaussian distribution properties that make this tractable are covered in our statistics guide, and the calculus of the ELBO objective that connects the forward and reverse processes is covered in our calculus guide.

The reverse process p_θ(x_{0:T}) learns to invert the forward process: starting from x_T ~ N(0,I) (pure noise), it iteratively denoises by predicting p_θ(x_{t-1} | x_t) = N(x_{t-1}; μ_θ(x_t, t), Σ_θ(x_t, t)) — the distribution of the less-noisy x_{t-1} given the current noisy x_t. The key insight of Ho et al. (2020) is that p_θ can be parameterised through noise prediction: rather than directly predicting the mean of x_{t-1}, train a network ε_θ(x_t, t) to predict the noise ε that was added to x_0 to get x_t. The training objective is simply the MSE between the predicted and actual noise: L = E_{t, x_0, ε}[||ε − ε_θ(√ᾱ_t x_0 + √(1−ᾱ_t) ε, t)||²]. This objective is equivalent to a reweighted variational lower bound on log p(x_0), connecting diffusion models to the variational Bayesian framework of VAEs. The U-Net architecture — with encoder, bottleneck, and skip-connected decoder — is the standard noise prediction network, with the time step t embedded via sinusoidal positional encodings (the same technique used in Transformers) and injected at each residual block.

DDIM, Score-Based Models and Flow Matching

DDPM’s 1,000-step sampling process makes image generation slow — generating a single image takes ~20 seconds even on a modern GPU. DDIM (Song et al., 2020) resolved this by observing that the noisy image x_t can be interpreted as defining an implicit non-Markovian forward process whose marginals q(x_t | x_0) are identical to DDPM’s but whose reverse process is a deterministic ODE rather than a stochastic SDE. The DDIM update rule: x_{t-1} = √ᾱ_{t-1} × (x_t − √(1−ᾱ_t) ε_θ(x_t,t)) / √ᾱ_t + √(1−ᾱ_{t-1}) × ε_θ(x_t,t). Because the process is deterministic, x_0 is a deterministic function of the initial noise z = x_T, and the latent space has meaningful geometric structure — interpolation, editing, and inversion are all well-defined. DDIM enables 10–50 step generation by taking larger steps along the ODE trajectory, at only a small cost in sample quality.

Score-based generative models (Song and Ermon, 2019; Song et al., 2021) provide a continuous-time unification of all diffusion models: the forward process is a stochastic differential equation (SDE) dx = f(x,t)dt + g(t)dW, and the reverse process is the reverse-time SDE dx = [f(x,t) − g(t)² ∇_x log p_t(x)]dt + g(t)dW̄, where ∇_x log p_t(x) is the score function of the noisy data distribution. DDPM and DDIM are special cases of this framework with different SDE choices. The score function is estimated by training a network s_θ(x,t) ≈ ∇_x log p_t(x), equivalent to the noise prediction network ε_θ up to a scaling factor. Flow matching (Lipman et al., 2022) — the basis of Stable Diffusion 3 and subsequent models — replaces the noisy SDE with a deterministic probability flow ODE that maps the data distribution to a Gaussian in a straight-line path, enabling faster ODE solvers and better sample quality at low step counts. The connection to the linear algebra of optimal transport is foundational: flow matching paths are the displacement interpolation paths of optimal transport between the data and noise distributions.

Stable Diffusion — Latent Diffusion and Text Conditioning

Operating the diffusion process in pixel space at 512×512 resolution requires storing and computing gradients through 262,144-dimensional vectors. Latent Diffusion Models (LDMs) solve this by first training a VAE (Variational Autoencoder — covered in our dimensionality reduction guide) that compresses 512×512 images to 64×64×4 latent codes, then running the entire diffusion process in this compressed latent space. The U-Net operates on 16,384-dimensional vectors (64×64×4) rather than 262,144-dimensional vectors — a 16× reduction that enables training on consumer GPU clusters. At generation time, the VAE decoder maps the denoised latent code back to pixel space. The separation of concerns is elegant: the VAE handles perceptual compression (encoding high-frequency pixel details into a compact latent); the diffusion model handles semantic generation (learning the distribution of meaningful latent codes).

Text conditioning is added via cross-attention in the U-Net: the text prompt is encoded by a text encoder (CLIP’s text encoder in Stable Diffusion 1.x/2.x, OpenCLIP or T5 in SDXL/SD3) into a sequence of text embeddings, which are injected into the U-Net at each resolution level via cross-attention — the same multi-head attention mechanism used in Transformers, with the image features as queries and the text embeddings as keys and values. Classifier-Free Guidance (CFG) amplifies the text conditioning at inference time by combining conditional and unconditional predictions: ε_guided = ε_uncond + w × (ε_cond − ε_uncond). A guidance scale w=7.5 produces high-fidelity prompt-following images; w=1.0 is equivalent to unconditioned generation. CFG is conceptually similar to the RLHF reward model in that it steers generation toward high-reward (high-fidelity-to-prompt) samples, and the mathematical connection between CFG and reward-guided sampling is formalised in classifier guidance theory.

For data scientists, the practical applications of diffusion models are: (1) Synthetic data generation — generate additional training images for computer vision tasks using a fine-tuned diffusion model; (2) Image-to-image translation — style transfer, domain adaptation, medical image synthesis; (3) Data augmentation — augment small labelled datasets with generated images that preserve labels (inpainting, outpainting, controlled generation via ControlNet); (4) Privacy-preserving synthetic data — generate synthetic images or tabular data that preserves distributional properties without exposing real patient or customer records, connecting to the federated learning and privacy paradigms.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

Synthetic data generation — including diffusion-based approaches — is in our Synthetic Data Generation guide. GANs as the predecessor generative architecture are in our Generative AI guide. VAE and autoencoder architectures underlying Stable Diffusion’s latent space are in our Dimensionality Reduction guide. The Transformer and cross-attention mechanism used in DiT and text conditioning is in our Transformers guide. Self-supervised pre-training principles shared with diffusion model objectives are in our Self-Supervised Learning guide. Computer vision applications that benefit from diffusion-based augmentation are in our Computer Vision guide.