Diffusion Models Explained: DDPM, Samplers, and Guidance

Diffusion models learn from examples corrupted at different noise levels. A network predicts a denoising quantity, and a sampler uses that prediction to construct progressively less noisy samples. Generation usually starts from Gaussian noise. Learning the reverse distribution does not recover the particular image that once produced a noisy observation; many clean images may be compatible with it.

The forward process

Let x₀ be clean data and t=1,…,T index corruption steps. For a fixed schedule \(0<\beta_t<1\), each step scales the previous state and adds independent Gaussian noise:

\[q(x_t\mid x_{t-1})=\mathcal{N}\left(x_t;\sqrt{1-\beta_t}\,x_{t-1},\;\beta_t I\right)\]

The property that makes this practical is that the composition of these steps has a closed form. With \(\alpha_t=1-\beta_t\) and \(\bar\alpha_t=\prod_{s\le t}\alpha_s\),

\[x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\qquad\varepsilon\sim\mathcal{N}(0,I)\]

This formula samples the marginal distribution at a chosen noise level directly. Training can choose a different t for each example without simulating all preceding steps. It saves repeated noising operations; those operations would be costly, not mathematically impossible. When the final alpha-bar is sufficiently small relative to the data scale, the terminal distribution is close to standard Gaussian, motivating the starting noise used for generation.

import torch
import torch.nn as nn
import torch.nn.functional as F

def cosine_schedule(T, s=0.008):
    if not isinstance(T, int) or T < 1 or not 0 <= s < float("inf"):
        raise ValueError("T must be positive and s finite and nonnegative")
    t = torch.linspace(0, 1, T + 1, dtype=torch.float64)
    f = torch.cos((t + s) / (1 + s) * torch.pi * 0.5).square()
    target_alpha_bar = f / f[0]
    betas = 1 - target_alpha_bar[1:] / target_alpha_bar[:-1]
    return betas.clamp(1e-8, 0.999)

T = 1000
betas = cosine_schedule(T)
alphas = 1 - betas
alpha_bar = alphas.cumprod(0)
for index in (0, 250, 500, 750, 999):
    print(f"index {index:3d} step {index+1:4d} alpha_bar {alpha_bar[index]:.6e} "
          f"noise_std {(1-alpha_bar[index]).sqrt():.4f}")
# index   0 step    1 alpha_bar 9.999587e-01 noise_std 0.0064
# index 250 step  251 alpha_bar 8.458880e-01 noise_std 0.3926
# index 500 step  501 alpha_bar 4.922852e-01 noise_std 0.7125
# index 750 step  751 alpha_bar 1.431786e-01 noise_std 0.9256
# index 999 step 1000 alpha_bar 2.428767e-09 noise_std 1.0000

Array index 0 represents mathematical step 1, so alpha_bar[0] is already slightly below 1. The clean endpoint is represented separately by alpha-bar=1. Clipping beta below 1 avoids an exactly zero cumulative product; use the cumulative product of the clipped betas, as above, rather than the unmodified cosine target. The first printed cumulative alpha is about 0.999959 rather than exactly 1. The schedule sets signal-to-noise ratios. Cosine scheduling improved results in the experiments that introduced it, but its suitability depends on data scaling, resolution, and model design; it is not uniformly better than a linear schedule.

The latent-variable formulation and the reparameterization trick are in Variational Autoencoders: ELBO, Reparameterization, and Latent Space.

The training objective

For Gaussian forward transitions, \(q(x_{t-1}\mid x_t,x_0)\) is exactly Gaussian. The reverse conditional \(q(x_{t-1}\mid x_t)\), which averages over unknown clean data, need not be Gaussian. DDPM models it with a Gaussian transition; sufficiently small steps help motivate that approximation. A network can parameterize its mean through a predicted noise vector. The commonly used simplified training loss is:

\[\mathcal{L}=\mathbb{E}_{t,x_0,\varepsilon}\left[\left\lVert\varepsilon-\varepsilon_\theta(x_t,t)\right\rVert^2\right]\]

The variational bound contains timestep-dependent weights and endpoint terms. The unweighted noise MSE shown here is a related training objective, not an exact algebraic replacement for the full bound. It avoids an adversarial opponent, but training can still become unstable or fit poorly. The target is the noise actually sampled during training; at inference the network estimates noise from the noisy observation and timestep, without access to the original clean example.

def schedule_for(alpha_bar, x):
    ab = alpha_bar.to(device=x.device, dtype=x.dtype)
    if ab.ndim != 1 or ab.numel() == 0 or not torch.isfinite(ab).all():
        raise ValueError("alpha_bar must be a finite nonempty vector")
    if not ((ab > 0) & (ab < 1)).all() or not (ab[1:] < ab[:-1]).all():
        raise ValueError("alpha_bar must decrease strictly inside (0,1)")
    return ab


def diffusion_loss(model, x0, alpha_bar):
    if x0.ndim < 2 or x0.size(0) == 0 or not torch.is_floating_point(x0):
        raise ValueError("x0 must be a nonempty floating-point batch")
    ab_table = schedule_for(alpha_bar, x0)
    N = x0.size(0)
    t = torch.randint(len(ab_table), (N,), device=x0.device)
    noise = torch.randn_like(x0)
    ab = ab_table[t].reshape(N, *([1] * (x0.ndim - 1)))
    x_t = ab.sqrt() * x0 + (1 - ab).sqrt() * noise
    prediction = model(x_t, t)
    if prediction.shape != noise.shape:
        raise ValueError("predicted noise must match the input shape")
    return F.mse_loss(prediction, noise)

The model receives the noisy batch and one integer timestep index per example. In the code, the gathered coefficients have singleton feature axes so each example’s noise level broadcasts over its coordinates. F.mse_loss averages over examples and coordinates, whereas the displayed squared norm sums coordinates; for a fixed input size this is a constant scaling. Timestep information can enter through learned embeddings or sinusoidal features and suitable conditioning layers.

U-Nets are one established denoiser architecture, with skip connections carrying spatial features across resolutions. Diffusion Transformers use token-based processing instead. The objective does not require a U-Net or a particular embedding scheme. The small vector example below uses a multilayer perceptron and a learned timestep embedding.

Sampling: DDPM and DDIM

The basic DDPM sampler traverses every trained timestep, predicting the mean of each reverse transition and adding noise except at the final step. In noise-prediction form, \(\mu_\theta(x_t,t)=\alpha_t^{-1/2}[x_t-\beta_t\varepsilon_\theta(x_t,t)/\sqrt{1-\bar\alpha_t}]\). One variance choice is \(\tilde\beta_t=\beta_t(1-\bar\alpha_{t-1})/(1-\bar\alpha_t)\); other fixed or learned choices are possible. Reduced-step samplers change the update rather than proving that every DDPM-based model needs 1,000 evaluations.

DDIM allows a selected decreasing sequence of noise levels. For a jump from t to an earlier s, first estimate \(\hat x_0=(x_t-\sqrt{1-\bar\alpha_t}\,\hat\varepsilon)/\sqrt{\bar\alpha_t}\), then use:

\[\sigma_{t\to s}=\eta\sqrt{\frac{1-\bar\alpha_s}{1-\bar\alpha_t}}\sqrt{1-\frac{\bar\alpha_t}{\bar\alpha_s}},\qquad x_s=\sqrt{\bar\alpha_s}\hat x_0+\sqrt{1-\bar\alpha_s-\sigma_{t\to s}^2}\,\hat\varepsilon+\sigma_{t\to s}\xi\]

Here xi is fresh standard Gaussian noise. For the final clean endpoint, alpha-bar_s=1 and sigma=0. With eta=1, all adjacent timesteps, and no clipping, this update matches DDPM with the posterior variance tilde-beta above. Skipping timesteps changes that chain. Eta=0 removes newly injected noise but retains randomness in the initial sample.

@torch.no_grad()
def ddim_sample(model, shape, alpha_bar, steps=50, eta=0.0,
                device="cpu", initial_noise=None, clip_range=None):
    if len(shape) < 2 or min(shape) < 1:
        raise ValueError("shape must describe a nonempty batch")
    if not isinstance(steps, int) or not 1 <= steps <= len(alpha_bar):
        raise ValueError("steps must be between 1 and the schedule length")
    if not 0 <= eta <= 1:
        raise ValueError("eta must be in [0,1]")
    if clip_range is not None and not clip_range[0] < clip_range[1]:
        raise ValueError("clip_range needs increasing bounds")
    param = next(model.parameters(), None)
    dtype = param.dtype if param is not None else alpha_bar.dtype
    if initial_noise is None:
        x = torch.randn(shape, device=device, dtype=dtype)
    else:
        if tuple(initial_noise.shape) != tuple(shape):
            raise ValueError("initial noise has the wrong shape")
        x = initial_noise.to(device=device, dtype=dtype).clone()
    ab = schedule_for(alpha_bar, x)
    times = torch.linspace(len(ab)-1, 0, steps).round().long().tolist()
    was_training = model.training
    model.eval()
    try:
        for i, index in enumerate(times):
            ab_t = ab[index]
            ab_prev = ab[times[i+1]] if i+1 < len(times) else ab.new_tensor(1.)
            t = torch.full((shape[0],), index, device=x.device, dtype=torch.long)
            eps = model(x, t)
            if eps.shape != x.shape:
                raise ValueError("predicted noise must match the sample shape")
            x0_pred = (x - (1-ab_t).sqrt()*eps) / ab_t.sqrt()
            if clip_range is not None:
                x0_pred = x0_pred.clamp(*clip_range)
                eps = (x-ab_t.sqrt()*x0_pred) / (1-ab_t).sqrt()
            sigma = eta * ((1-ab_prev)/(1-ab_t) * (1-ab_t/ab_prev)).clamp_min(0).sqrt()
            x = ab_prev.sqrt()*x0_pred + (1-ab_prev-sigma.square()).clamp_min(0).sqrt()*eps
            if eta > 0 and i+1 < len(times):
                x = x + sigma*torch.randn_like(x)
        return x
    finally:
        model.train(was_training)

DDIM constructs a family of forward processes sharing the same noised-data marginals and noise-prediction training objective. Its updates permit fewer selected timesteps, with a speed–quality trade-off that must be measured. Fifty evaluations instead of 1,000 is 20 times fewer model calls, not a guaranteed 20-fold wall-clock speedup or equal image quality. Eta=0 is deterministic given the initial noise, model, conditioning, schedule, and deterministic execution. A seed alone does not guarantee identical results across devices or software versions. Move the model to the requested device before sampling; the helper moves schedule coefficients and initial noise to that device and uses the model’s parameter dtype. Clipping is optional: it should match known data bounds, and this implementation recomputes the noise estimate after clipping to keep the update consistent.

Train and sample a small vector model

This CPU example uses two short intervals centered at −1.5 and +1.5. The model predicts one noise coordinate, conditioned on a learned time embedding. Training and validation draws are separate, and the validation noise is held fixed for the before/after comparison. Clean values lie inside [-2,2], so this example enables that clipping range. Unbounded features and learned image latents should not inherit pixel clipping automatically. Run all code blocks in order.

torch.manual_seed(6)
toy_T = 64
toy_ab = (1-cosine_schedule(toy_T)).cumprod(0).float()

class TinyDenoiser(nn.Module):
    def __init__(self, T):
        super().__init__()
        self.time = nn.Embedding(T, 16)
        self.net = nn.Sequential(nn.Linear(17, 64), nn.SiLU(),
                                 nn.Linear(64, 64), nn.SiLU(), nn.Linear(64, 1))
    def forward(self, x, t):
        return self.net(torch.cat([x, self.time(t)], dim=1))

def draw_clean(n):
    side = 2*torch.randint(2, (n,1)).float()-1
    return 1.5*side + 0.5*(torch.rand(n,1)-0.5)

toy = TinyDenoiser(toy_T)
train_x, valid_x = draw_clean(1024), draw_clean(256)
valid_t = torch.randint(toy_T, (len(valid_x),))
valid_noise = torch.randn_like(valid_x)
a = toy_ab[valid_t, None]
valid_noisy = a.sqrt()*valid_x + (1-a).sqrt()*valid_noise
with torch.no_grad():
    before = F.mse_loss(toy(valid_noisy, valid_t), valid_noise).item()
opt = torch.optim.Adam(toy.parameters(), lr=0.001)
for _ in range(400):
    xb = train_x[torch.randint(len(train_x), (64,))]
    loss = diffusion_loss(toy, xb, toy_ab)
    opt.zero_grad(set_to_none=True)
    loss.backward()
    opt.step()
toy.eval()
with torch.no_grad():
    after = F.mse_loss(toy(valid_noisy, valid_t), valid_noise).item()
initial = torch.randn(128,1)
generated = ddim_sample(toy, initial.shape, toy_ab, steps=32,
                        initial_noise=initial, clip_range=(-2.,2.))
print("validation noise MSE before/after", round(before,4), round(after,4))
print("generated shape", tuple(generated.shape), "finite", torch.isfinite(generated).all().item())
print("first six", [round(v,2) for v in generated[:6,0].tolist()])
# validation noise MSE before/after 0.9619 0.7473
# generated shape (128, 1) finite True
# first six [2.0, -1.24, -1.75, 1.16, -0.9, -1.91]

In this run, validation noise MSE decreases from 0.9619 to 0.7473. Several displayed samples still fall outside the clean intervals [-1.75,-1.25] and [1.25,1.75]. Lower noise loss therefore has not established an accurate generated distribution. It does not establish image quality or accurate mode probabilities. Inspect the generated values and evaluate distribution coverage separately. This is an unconditional model; the next section describes the additional training needed for classifier-free guidance.

Classifier-free guidance

A condition c can be a class label, text embedding, or other information. Classifier-free guidance trains conditional and null-conditioned predictions in the same model, replacing the condition with a designated null representation on a chosen fraction of examples. A dropout probability such as 0.1 is a training choice, not a requirement. Passing an arbitrary zero vector to a model never trained on that null condition does not supply an unconditional predictor.

At sampling time, combine noise predictions at the same noisy input and timestep. Here the model’s time argument is implicit in the notation:

\[\tilde\varepsilon=\varepsilon_\theta(x_t,\varnothing)+w\left(\varepsilon_\theta(x_t,c)-\varepsilon_\theta(x_t,\varnothing)\right)\]

def guided_eps(model, x, t, cond, null_cond, guidance_scale=1.0):
    eps_uncond = model(x, t, null_cond)
    eps_cond = model(x, t, cond)
    return eps_uncond + guidance_scale * (eps_cond - eps_uncond)

With this convention, w=0 selects the unconditional prediction and w=1 selects the ordinary conditional prediction. Values above 1 extrapolate their difference. Stronger guidance can improve condition adherence while reducing diversity or introducing artifacts, but useful scales depend on the model, parameterization, and sampler. Some publications define a different scale whose zero already means conditional generation. This helper makes two model calls; batching both branches or other implementation choices changes wall-clock cost, so latency need not exactly double. To use it with the sampler above, pass a model wrapper whose forward(x,t) calls guided_eps with the chosen condition and null condition.

Latent diffusion

Latent diffusion trains an image autoencoder, encodes images, applies diffusion to the codes, and decodes generated codes back to images. The autoencoder may use KL or other regularization. A spatial factor of eight is one configuration, not part of the definition of latent diffusion. Code scaling and normalization are part of the trained system and must be reproduced during sampling.

For example, a 512×512×3 image encoded as 64×64×4 has 64 times fewer spatial positions but 48 times fewer scalar entries. Neither ratio is the exact compute reduction: denoiser width, attention, the number of steps, and autoencoder cost all matter. Some text-conditioned models use cross-attention with image or latent queries and text-derived keys and values; this is an architecture choice, not a property required by diffusion.

The adversarial alternative and its training difficulties are covered in Generative Adversarial Networks: Minimax, WGAN-GP, and Training Stability.

Comparing objectives and sampling approaches

Iterative inference is one cost of diffusion. Distillation and consistency approaches can reduce the number of evaluations by training for larger jumps or endpoint predictions. Their quality and coverage depend on the teacher, student, training procedure, and sampling budget; they do not establish that all diffusion models can be sampled well in a fixed small number of steps.

Flow matching trains a velocity field for specified probability paths between noise and data. Those paths can include diffusion paths or other choices. Even if conditional training paths are straight lines between paired endpoints, the learned marginal flow trajectories need not be straight. Sampling cost depends on the learned field and numerical solver; fewer steps and simpler training are not automatic consequences of the formulation.

Compare samplers at matched models, resolutions, hardware, and quality targets. Count model evaluations as well as measuring latency; guidance branches and decoder work can make equal step counts cost different amounts.

Exercises

1. The closed-form forward process. Compare 201 Gaussian noising steps with a direct marginal draw. Array index 200 represents the 201st noising step. Report the two sample means and standard deviations alongside their theoretical values.

The two procedures have the same Gaussian marginal distribution, but independent draws will not give identical sample statistics.

import torch
torch.manual_seed(0)
T = 1000
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1 - betas
abar = alphas.cumprod(0)
t = 200
x0 = torch.full((20_000,), 3.0)

x = x0.clone()
for i in range(t + 1):
    x = alphas[i].sqrt() * x + betas[i].sqrt() * torch.randn_like(x)
print(f"iterated  mean {x.mean():.4f}  std {x.std():.4f}")

closed = abar[t].sqrt() * x0 + (1 - abar[t]).sqrt() * torch.randn_like(x0)
print(f"closed    mean {closed.mean():.4f}  std {closed.std():.4f}")
print(f"sqrt(abar[{t}]) {abar[t].sqrt():.4f}")
print(f"theory mean {3*abar[t].sqrt():.4f} std {(1-abar[t]).sqrt():.4f}")
# iterated  mean 2.4301  std 0.5847
# closed    mean 2.4284  std 0.5846
# sqrt(abar[200]) 0.8102
# theory mean 2.4305 std 0.5862

2. The algebra of guidance. Apply the guidance formula to two fixed synthetic prediction vectors and tabulate the output norm from w=0 to w=15. What can this calculation establish without generating any images?

When the difference vector is nonzero, its contribution eventually dominates as the scale grows. The norm need not increase at every small scale.

import torch
torch.manual_seed(0)
eps_u = torch.randn(4096)                     # unconditional prediction
eps_c = eps_u + 0.3 * torch.randn(4096)       # conditional, slightly different
for w in (0, 1, 3, 7.5, 15):
    guided = eps_u + w * (eps_c - eps_u)
    print(f"w={w:5.1f}  norm {guided.norm():8.2f}  "
          f"ratio {guided.norm() / eps_u.norm():.3f}")
# w=  0.0  norm    63.57  ratio 1.000
# w=  1.0  norm    66.43  ratio 1.045
# w=  3.0  norm    86.10  ratio 1.354
# w=  7.5  norm   158.66  ratio 2.496
# w= 15.0  norm   297.73  ratio 4.683

3. Determinism, random paths, and reproducibility. Hold initial noise fixed and compare eta=0 runs, eta=1 runs with different random streams, and eta=1 runs with the same stream. Use the full timestep grid for the posterior-variance DDPM configuration.

Eta=0 uses no new noise. A stochastic sampler can also reproduce its result when the entire random stream and deterministic execution are fixed.

class CheckDenoiser(nn.Module):
    def forward(self, x, t):
        return 0.1*x

check_model = CheckDenoiser()
check_ab = (1-torch.linspace(.02,.1,20)).cumprod(0)
torch.manual_seed(12)
start = torch.randn(4,2)
kwargs = dict(model=check_model, shape=start.shape, alpha_bar=check_ab,
              steps=20, initial_noise=start)
a = ddim_sample(**kwargs, eta=0)
b = ddim_sample(**kwargs, eta=0)
torch.manual_seed(21)
c = ddim_sample(**kwargs, eta=1)
torch.manual_seed(22)
d = ddim_sample(**kwargs, eta=1)
torch.manual_seed(21)
e = ddim_sample(**kwargs, eta=1)
print("eta=0 repeated", torch.equal(a,b))
print("eta=1 different streams", torch.equal(c,d))
print("eta=1 same stream", torch.equal(c,e))
# eta=0 repeated True
# eta=1 different streams False
# eta=1 same stream True

References

• Sohl-Dickstein, Weiss, Maheswaranathan, and Ganguli (2015). Deep Unsupervised Learning Using Nonequilibrium Thermodynamics. ICML.
• Ho, Jain, and Abbeel (2020). Denoising Diffusion Probabilistic Models.
• Song, Meng, and Ermon (2021). Denoising Diffusion Implicit Models.
• Nichol and Dhariwal (2021). Improved Denoising Diffusion Probabilistic Models. ICML.
• Dhariwal and Nichol (2021). Diffusion Models Beat GANs on Image Synthesis. NeurIPS.
• Ho and Salimans (2022). Classifier-Free Diffusion Guidance.
• Rombach, Blattmann, Lorenz, Esser, and Ommer (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR.
• Song, Dhariwal, Chen, and Sutskever (2023). Consistency Models.
• Lipman, Chen, Ben-Hamu, Nickel, and Le (2023). Flow Matching for Generative Modeling.