We have already encountered several types of generative models. A GAN resembles an adversarial game in which the generator produces realistic samples in a single step; an autoencoder learns compression and reconstruction; and a VAE explicitly models a probability distribution from which we can sample latent variables. A DDPM takes a different approach:
Rather than drawing an image in one step, it starts with noise and gradually “washes out” the image through denoising.
This may sound strange at first. Isn’t denoising simply a way to clean up a corrupted image? What does that have to do with generation? Why does repeatedly denoising pure noise eventually produce a realistic image?
The name DDPM (Ho et al. 2020), short for Denoising Diffusion Probabilistic Model, already identifies its three most important components:
Rather than deriving the complete DDPM immediately, this section first builds a map of the overall method. Sections 14.2, 14.3, and 14.4 will separately examine the forward noising process, reverse denoising process, training objective, and sampling procedure. For now, the central question is:
How can a generation problem be reformulated as a denoising problem?
import math
import random
import dnnlpy
import matplotlib.pyplot as plt
import torch
import torchvision.datasets as datasets
import torchvision.transforms.v2 as v2
dnnlpy.set_matplotlib_format('highdpi')
print('PyTorch version:', torch.__version__)PyTorch version: 2.13.0+cpuLet us begin with a very simple process.
Suppose we have a real image \(x_0\). If we continuously add Gaussian noise to it, the image gradually loses its original structure. At first, its edges may only become slightly blurry. As more noise is added, local textures begin to disappear, followed by the overall outline. When the noise is sufficiently strong, the final \(x_T\) can become very close to a standard Gaussian random variable.
In other words, we can construct the following path:
\[ x_0 \rightarrow x_1 \rightarrow x_2 \rightarrow \cdots \rightarrow x_T \]
The left side represents real data, while the right side follows a simple, familiar Gaussian distribution.
This direction is not difficult. We do not even need to train a neural network: by prescribing the amount of noise added at each step, we can gradually corrupt any image. The more interesting question is:
If a real image can gradually become noise along this path, can we travel the same path in reverse?
If so, we do not need to learn the difficult mapping
\[ \text{noise} \rightarrow \text{image} \]
directly. Instead, we can learn a sequence of smaller changes:
\[ \text{noise} \rightarrow \text{slightly less noisy} \rightarrow \text{slightly less noisy} \rightarrow \cdots \rightarrow \text{image} \]
Let us first conduct an intuitive experiment. To ensure that the last step is genuinely close to pure Gaussian noise, instead of simply writing x0 + sigma * noise, we mix the image signal and noise using different weights:
\[ x_t = \sqrt{s_t}\,x_0 + \sqrt{1-s_t}\,\epsilon, \qquad \epsilon\sim\mathcal N(0,I) \]
For now, we can interpret \(s_t\) simply as the amount of the original image signal that remains. When \(s_t=1\), the result is entirely the original image; when \(s_t=0\), it is entirely Gaussian noise. We will later see that \(\bar\alpha_t\) plays a similar role in a DDPM.
root = dnnlpy.get_data_root()
transform = v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)])
ds = datasets.MNIST(root, train=False, download=True, transform=transform)
idx = random.randrange(len(ds))
x0 = ds[idx][0].squeeze(0)
x0 = x0 * 2.0 - 1.0 # [0, 1] -> [-1, 1]
signal_levels = [1.0, 0.8, 0.5, 0.25, 0.1, 0.0]
fig = plt.figure(1, figsize=(8, 2))
axes = fig.subplots(1, len(signal_levels))
for ax, level in zip(axes, signal_levels, strict=True):
noise = torch.randn_like(x0)
xt = math.sqrt(level) * x0 + math.sqrt(1.0 - level) * noise
ax.imshow(xt, cmap='gray', vmin=-1.0, vmax=1.0)
ax.axis('off')
ax.set_title(f'Signal Level: {level:.2f}', fontsize=10)
fig.tight_layout(pad=0.5)
plt.show()
What matters most here is not the appearance of any particular intermediate image, but the overall trend. As the signal level decreases, the image retains less information while the noise grows stronger. When \(s_t=0\), the result is a sample of Gaussian noise.
Thus, a DDPM does not assume that pure noise still contains an image that a human could recognize. At a sufficiently large time step, an individual \(x_T\) contains almost no visible trace of its corresponding \(x_0\). The model can nevertheless generate an image from noise because, after training, it has learned which local adjustments gradually move a sample back toward the real data distribution.
This distinction is important. A diffusion model is not performing image restoration in the sense of “recovering a particular corrupted image.” It is learning a generative process. The initial noise used during generation never corresponded to a particular training image, and the final sample is gradually constructed by following the learned reverse process.
Suppose the model is given two tasks.
Task A: Generate an image of a cat in one step.
The input is random noise, and the output must be a complete, natural, structurally coherent, and detailed image of a cat. In a single step, the model must decide many interdependent factors: What is the cat’s pose? Which way is its face pointing? What is the geometric relationship among its body and limbs? What is in the background? How should the fur be rendered? How should the lighting and shadows remain consistent? How should local details conform to the overall structure?
From the perspective of probability distributions, this asks the model to transform a simple Gaussian distribution directly into a highly complex data distribution. Learning this mapping is not impossible—GANs do something similar—but the required transformation is substantial.
Task B: Given an image that retains some structure but has been corrupted by noise, make only a small correction.
This task is usually more local. The model does not need to decide every detail of the final image in a single computation. It only needs to determine which direction the current sample should move slightly, at its current noise level, to become more like real data.
If we imagine generation as traveling from the foot of a mountain to its summit, one-step generation asks the model to jump directly from the starting point to the destination. A DDPM instead divides the route into many short segments and decides only how to take the next small step. No individual step involves such a dramatic change.
There is also a deeper reason. If the current \(x_t\) contains only slightly more noise than \(x_{t-1}\), the two adjacent distributions are also relatively similar. Rather than asking a neural network to learn a complete mapping between two vastly different distributions in one step, a DDPM learns many reverse transitions between adjacent noise levels.
As a result, a difficult global generation problem is decomposed into many comparatively simple local denoising problems:
\[ p_\theta(x_{t-1}\mid x_t) \]
This expression means:
Given the current sample \(x_t\), what distribution should the slightly cleaner \(x_{t-1}\) follow?
Strictly speaking, the reverse process does not simply subtract the noise predicted by the network from the image. The network’s prediction is inserted into a reverse-update formula determined by the noise schedule, and the original DDPM sampling process usually also contains a stochastic term. More precisely, the network supplies the information required for the reverse transition, while the sampling formula uses that information to obtain \(x_{t-1}\) from \(x_t\). We will develop the exact form in Section 14.3.
Therefore, what a diffusion model truly exploits is the idea of iterative generation. The complexity of the task does not disappear; it is distributed across many time steps. The cost is equally direct: if the reverse process contains \(T\) steps, generating one image may require calling the network many times. DDPMs are stable to train, but their original sampling procedure is slow. This limitation motivated later developments such as DDIM, distillation, and many other fast sampling methods.
The overall structure of a DDPM consists of two processes that move in opposite directions.
The first is the forward noising process.
Starting with real data \(x_0\sim p_{\text{data}}(x)\), we gradually add Gaussian noise according to a predefined rule:
\[ x_0 \rightarrow x_1 \rightarrow x_2 \rightarrow \cdots \rightarrow x_T \]
This process is typically denoted by \(q\). It is not a neural network and has no trainable parameters; it is a probability process that we define ourselves. Each step depends only on the preceding step, so we can write
\[ q(x_{1:T}\mid x_0) = \prod_{t=1}^{T}q(x_t\mid x_{t-1}) \]
This is a Markov chain. To produce the current \(x_t\), we only need to know \(x_{t-1}\); we do not need to revisit the earlier states \(x_{t-2},x_{t-3},\ldots\).
As \(t\) increases, Gaussian noise obscures more and more of the data’s structure. If the noise schedule is designed appropriately, the final marginal distribution \(q(x_T)\) approaches a simple standard Gaussian distribution:
\[ q(x_T)\approx\mathcal N(0,I) \]
Why deliberately choose a Gaussian distribution as the endpoint? Because it is extremely easy to sample from. During generation, we cannot begin by drawing \(x_0\) from the real data distribution; doing so would defeat the purpose of generation. We can, however, create a random tensor at any time:
xT = torch.randn(1, 28, 28)The forward process therefore accomplishes something essential: it deliberately constructs a path from a complex but unknown data distribution to a simple Gaussian distribution that is easy to sample from.
The second is the reverse denoising process.
We now want to travel this path in reverse:
\[ x_T \rightarrow x_{T-1} \rightarrow x_{T-2} \rightarrow \cdots \rightarrow x_0 \]
If every true reverse conditional distribution \(q(x_{t-1}\mid x_t)\) were known, we could theoretically start with \(x_T\sim\mathcal N(0,I)\) and sample backward until we reached the data distribution. The problem is that this true reverse process depends on the unknown data distribution, so it is not directly available to us.
A DDPM therefore uses a parameterized model to approximate it:
\[ p_\theta(x_{t-1}\mid x_t) \]
Connecting all the reverse steps gives the generative process defined by the model:
\[ p_\theta(x_{0:T}) = p(x_T) \prod_{t=1}^{T}p_\theta(x_{t-1}\mid x_t) \]
where the starting distribution is typically chosen as
\[ p(x_T)=\mathcal N(0,I) \]
From this perspective, the generative logic of a DDPM is straightforward:
This is also why a diffusion model should not be understood merely as a powerful image denoiser. At any individual time step, it is indeed performing denoising; once all the time steps are connected, however, they define a complete probabilistic generative model.
Another easily overlooked point is that, although the forward process is drawn as a long chain, training does not require us to compute \(x_1\), then \(x_2\), and so on until reaching a randomly chosen \(x_t\). The DDPM forward process has a convenient closed form that lets us sample any \(x_t\) directly from \(x_0\) in one step:
\[ x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon, \qquad \epsilon\sim\mathcal N(0,I) \]
During training, we therefore usually sample a time step \(t\) at random and directly construct \(x_t\) at the corresponding noise level. Although the forward process conceptually contains \(T\) steps, producing a single training example does not require executing all \(T\) noising operations. This allows DDPM training to use minibatch parallelism as efficiently as ordinary supervised learning.
The next section derives this equation and explains the relationships among \(\alpha_t\), \(\bar\alpha_t\), and the commonly used \(\beta_t\).
Let us now examine what the neural network learns during training.
Using the closed form above, we first draw a real sample \(x_0\) from the dataset, choose a random time step \(t\), and sample Gaussian noise \(\epsilon\):
\[ \epsilon\sim\mathcal N(0,I) \]
We then directly construct
\[ x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon \]
Now \(x_t\) is a noisy sample that we deliberately created. Because we sampled the noise ourselves, the true \(\epsilon\) is also known. A DDPM therefore requires no manual labels indicating which noise appears in each image; its training targets are generated automatically.
The classic DDPM parameterization asks the network to predict this noise:
\[ \epsilon_\theta(x_t,t)\approx\epsilon \]
We can therefore use an ordinary mean squared error:
\[ \mathcal L_{\text{simple}} = \mathbb E_{x_0,t,\epsilon} \left[ \left\| \epsilon- \epsilon_\theta(x_t,t) \right\|_2^2 \right] \]
At first glance, this objective hardly looks like one for training a generative model: the input is a noisy image, the target is random noise, and the loss is MSE. What makes the network a generative model is that these denoising tasks cover every time step, from an almost clean image to almost pure noise.
The time step \(t\) is not an optional input. Consider two extremes:
Giving the network only \(x_t\) without telling it the current noise level would be like asking the same network to solve many different denoising tasks without revealing how difficult each one is. In practice, the model therefore encodes the time step \(t\) as a vector and combines it with the image features. When we introduce the DDPM network architecture, we will see exactly how this timestep embedding enters the U-Net.
From the perspective of the training loop, one DDPM training example can be summarized as
\[ x_0 \xrightarrow{\text{sample }t,\epsilon} x_t \xrightarrow{\epsilon_\theta(x_t,t)} \hat\epsilon \xrightarrow{\text{MSE with }\epsilon} \mathcal L \]
The pseudocode is almost this simple:
x0 = sample_real_data()
t = sample_timestep()
eps = sample_gaussian_noise()
xt = add_noise(x0, t, eps)
eps_pred = model(xt, t)
loss = mse(eps_pred, eps)There is also an important asymmetry between training and generation. During training, we can randomly select any \(t\) and construct \(x_t\) directly, so different time steps can appear in parallel within the same batch. During generation, however, we must begin with \(x_T\) and obtain each subsequent state in the order \(T,T-1,\ldots,1\).
| Stage | Starting point | Time step | Network task | Sequential execution required? |
|---|---|---|---|---|
| Training | Real sample \(x_0\) | Randomly sample \(t\) | Predict noise from \((x_t,t)\) | No; any \(x_t\) can be constructed directly |
| Generation | Gaussian noise \(x_T\) | Move from \(T\) to \(1\) | Provide a prediction for each reverse sampling step | Yes; each step depends on the result of the previous step |
This explains a characteristic feature of diffusion models: training is highly parallelizable, but the original sampling process is sequential. Each \(x_{t-1}\) depends on the newly obtained \(x_t\), so all time steps cannot be computed in parallel as they can during training.
Finally, a note about notation is important. This chapter primarily follows the original DDPM’s common \(\epsilon\)-prediction parameterization, in which the network predicts noise. However, “a diffusion model must predict noise” is not a universal rule. A model may instead predict \(x_0\) directly or predict \(v\), which combines \(x_0\) and \(\epsilon\). For this section, focusing on the classic \(\epsilon\)-prediction parameterization is enough to establish the basic DDPM framework.
Although we have not yet derived all the mathematical details of a DDPM, we can now connect the pieces of its overall logic.
A DDPM first defines a forward diffusion process. Gaussian noise is gradually added to a real sample \(x_0\) over time, eventually making \(x_T\) approach a simple standard Gaussian distribution:
\[ x_0 \xrightarrow{q} \cdots \xrightarrow{q} x_T\approx\mathcal N(0,I) \]
This process does not need to be learned. What the neural network must learn is the reverse direction: given \(x_t\) at a particular noise level, estimate the information required for the reverse transition so that the sample gradually moves toward cleaner regions that better match the real data distribution:
\[ x_T \xrightarrow{p_\theta} \cdots \xrightarrow{p_\theta} x_0 \]
In the classic DDPM parameterization, the network accomplishes this task by predicting the Gaussian noise added to \(x_t\):
\[ \epsilon_\theta(x_t,t)\approx\epsilon \]
Thus, an apparently complex image generation problem is transformed into many denoising-learning problems at different noise levels.
The essential logic of a DDPM can be condensed into four steps:
The crucial shift is that a DDPM does not ask a model to learn how to generate an entire image in one step. Instead, at every noise level, the model learns how to take the next small step toward the data distribution. Connecting many such steps produces the complete generative process.
Several questions remain unanswered. We have repeatedly said that Gaussian noise is “gradually added,” but how exactly is it added at each step? What are \(\beta_t\), \(\alpha_t\), and \(\bar\alpha_t\)? Why can we obtain \(x_t\) directly from \(x_0\) without passing through \(x_1,x_2,\ldots,x_{t-1}\)?
The next section begins with the forward process and expresses these ideas formally as probability distributions and equations. Once we understand forward noising, the reasons why reverse denoising can be written as a Gaussian distribution-and where the final noise-prediction objective comes from-will follow much more naturally.