Generative Adversarial Networks: Minimax, WGAN-GP, and Training Stability

A GAN trains a generator using feedback from a learned discriminator. The adversarial objective is specified in advance, while the discriminator learns which features distinguish generated samples from real ones. This provides a training signal without requiring an explicit generator likelihood. Sharpness and stability depend on the objective, architecture, data, and optimization; likelihood-based training does not inherently produce blurry samples.

The minimax game

The generator G transforms random noise z drawn from a chosen distribution p_z into a sample G(z). The discriminator D outputs a probability that a sample is real. We use equal weighting for real and generated examples. The original objective is:

\[\min_G\max_D\;\mathbb{E}_{x\sim p_{\text{data}}}\left[\log D(x)\right]+\mathbb{E}_{z\sim p_z}\left[\log(1-D(G(z)))\right]\]

D maximizes this binary log-likelihood while G minimizes its generated-sample term. For a fixed generator distribution \(p_g\), an unrestricted optimal discriminator is \(D^*(x)=p_{data}(x)/(p_{data}(x)+p_g(x))\) wherever the denominator is positive. In the idealized distribution-level optimum, \(p_g=p_{data}\) and D*=1/2 on their support. Values outside that support are unconstrained. A network outputting 1/2 is not by itself evidence of distribution matching; it could simply be an untrained discriminator.

Each network minimizes its own implemented loss, and each update changes the other network’s objective. This coupled game can oscillate even in simple examples. General neural GAN training has no automatic convergence guarantee, although convergence results exist under particular assumptions and algorithms. Architecture, gradient flow, optimizer choices, and implementation errors can all affect the outcome.

The optimizers that turn these gradients into parameter updates are compared in Deep Learning Optimizers: Mini-Batch, Momentum, RMSprop, and Adam.

The saturation problem

Write the discriminator probability as \(D=\operatorname{sigmoid}(a)\), where a is its logit. The saturating generator loss has derivative \(\partial\log(1-D)/\partial a=-D\). When D on fakes is near zero, this factor shrinks. Its derivative with respect to D alone, \(-1/(1-D)\), does not vanish; the sigmoid derivative is essential to this explanation.

The non-saturating generator minimizes \(-\log D(G(z))\), whose logit derivative is D−1. For D=0.001, the two derivative magnitudes are 0.001 and 0.999. This avoids that saturation factor, though the full generator gradient still multiplies derivatives through D and G. The objectives share the ideal distribution-matching equilibrium, but their optimization dynamics differ. Non-saturating logistic loss is one common choice; hinge and Wasserstein objectives are other choices.

In code, the discriminator returns logits with one scalar per sample and no final sigmoid. BCEWithLogitsLoss combines sigmoid and binary cross-entropy stably. The discriminator loss adds the mean real and mean fake losses; it does not average those two terms again. A constant zero logit therefore gives 2 log 2 ≈ 1.3863. Run the blocks in order; a small executable example follows the WGAN-GP update.

import torch
import torch.nn as nn
import torch.nn.functional as F

def d_loss_fn(d_real, d_fake):
    """Discriminator: real -> 1, fake -> 0."""
    real = F.binary_cross_entropy_with_logits(d_real, torch.ones_like(d_real))
    fake = F.binary_cross_entropy_with_logits(d_fake, torch.zeros_like(d_fake))
    return real + fake

def g_loss_fn(d_fake):
    """Non-saturating: label fakes as real to get a usable gradient."""
    return F.binary_cross_entropy_with_logits(d_fake, torch.ones_like(d_fake))

The training loop

def train_step(G, D, opt_g, opt_d, real, latent_dim, device):
    G.train()
    D.train()
    real = real.to(device)
    bs = real.size(0)
    opt_g.zero_grad(set_to_none=True)
    with torch.no_grad():
        fake = G(torch.randn(bs, latent_dim, device=device, dtype=real.dtype))
    opt_d.zero_grad(set_to_none=True)
    d_loss = d_loss_fn(D(real), D(fake))
    d_loss.backward()
    opt_d.step()
    opt_d.zero_grad(set_to_none=True)

    flags = [p.requires_grad for p in D.parameters()]
    try:
        for p in D.parameters():
            p.requires_grad_(False)
        z = torch.randn(bs, latent_dim, device=device, dtype=real.dtype)
        g_loss = g_loss_fn(D(G(z)))
        g_loss.backward()
        opt_g.step()
    finally:
        for p, flag in zip(D.parameters(), flags):
            p.requires_grad_(flag)
    return d_loss.item(), g_loss.item()

The discriminator step uses no_grad() to avoid building a generator graph; passing fake.detach() is another way to block that path. Without either, backward() computes generator gradients too, but only an optimizer step changes its parameters. If those extra gradients are discarded before the generator step, they waste computation rather than necessarily corrupting training. Unintentionally stepping G with them would optimize the discriminator’s objective in G.

The generator step keeps differentiation through D’s input while temporarily freezing D’s parameters. Putting this step inside no_grad() would block the needed path. Fresh noise is convenient but not required: a graph-connected fake can be reused if its G graph remains available, followed by a new D forward pass after D’s update. Freezing parameters does not freeze BatchNorm buffers or disable dropout; the small networks below contain neither, and architectures with such state need an explicit mode policy.

Mode collapse

Mode collapse means generating a narrow subset of the data’s patterns, possibly despite varied noise inputs. Against a temporarily fixed discriminator, concentrating on its highest-scoring outputs can be attractive. That is not the global distribution-matching optimum of the adversarial game: a discriminator trained on real data can learn to distinguish an omitted mode or an overrepresented one.

Repeated-looking samples across different noise inputs can reveal collapse, but loss curves alone are not a sufficient diagnosis. Track a fixed batch of noise inputs across checkpoints and also inspect fresh samples. On synthetic data, measure coverage of known modes; on real data, use diversity and coverage measures suited to the domain.

Minibatch discrimination gives D information about sample collections, while unrolled GAN training accounts for some of D’s future response when updating G. Wasserstein objectives offer a different gradient signal. These methods address different aspects of the problem and do not guarantee full mode coverage.

Wasserstein GAN with gradient penalty

Jensen–Shannon (JS) divergence compares each distribution with their equal mixture. With the optimal discriminator substituted into the original minimax objective, its value is \(-\log4+2\,\mathrm{JS}(p_{data}\Vert p_g)\). JS divergence is log 2 for mutually singular distributions, so it cannot distinguish how far apart two point masses are. This motivates studying other distances. It does not say that every finite neural discriminator, or the non-saturating generator objective, is literally optimizing that substituted JS expression.

For distributions with finite first moments, Wasserstein-1 is the minimum expected Euclidean transport cost between them. For point masses at 0 and theta it is |theta|, continuous but not differentiable at zero. Under suitable generator regularity it can vary continuously and be differentiable almost everywhere in parameters, rather than being differentiable everywhere. Its dual optimizes the real-minus-fake score gap over 1-Lipschitz critics: \(|C(x)-C(y)|\le\lVert x-y\rVert_2\). The critic outputs an unrestricted scalar, not a real/fake probability. WGAN-GP uses a soft penalty on interpolated inputs to encourage gradient norms near 1:

\[\mathcal{L}_C=\mathbb{E}\left[C(\tilde x)\right]-\mathbb{E}\left[C(x)\right]+\lambda\,\mathbb{E}\left[\left(\lVert\nabla_{\hat x}C(\hat x)\rVert_2-1\right)^2\right]\]

Here x is real data, \(\tilde x=G(z)\) is generated data, and \(\hat x=\epsilon x+(1-\epsilon)\tilde x\) with a separate uniform \(\epsilon\in[0,1]\) for each example. Lambda sets the penalty weight. The generator minimizes \(\mathcal L_G=-\mathbb E_z[C(G(z))]\), encouraging higher critic scores on generated samples. Neither Wasserstein loss uses sigmoid or binary labels.

def gradient_penalty(C, real, fake, lambda_gp=10.0):
    if real.shape != fake.shape or real.ndim < 2 or real.size(0) == 0:
        raise ValueError("real and fake must share a nonempty batch shape")
    if real.device != fake.device or real.dtype != fake.dtype:
        raise ValueError("real and fake must share device and floating-point dtype")
    if not torch.is_floating_point(real) or not 0 <= lambda_gp < float("inf"):
        raise ValueError("floating-point inputs and a finite nonnegative penalty required")
    bs = real.size(0)
    eps = torch.rand(bs, *([1] * (real.dim() - 1)),
                     device=real.device, dtype=real.dtype)
    x_hat = (eps * real.detach() + (1 - eps) * fake.detach()).requires_grad_(True)
    d_hat = C(x_hat)
    if d_hat.numel() != bs:
        raise ValueError("critic must return one scalar per example")
    grads = torch.autograd.grad(d_hat, x_hat, torch.ones_like(d_hat),
                                create_graph=True)[0]
    norm = grads.flatten(1).norm(2, dim=1)
    return lambda_gp * ((norm - 1) ** 2).mean()


def wgan_gp_step(G, C, opt_g, opt_c, real, latent_dim, lambda_gp=10.0):
    G.train()
    C.train()
    bs = real.size(0)
    opt_g.zero_grad(set_to_none=True)
    with torch.no_grad():
        fake = G(torch.randn(bs, latent_dim, device=real.device, dtype=real.dtype))
    opt_c.zero_grad(set_to_none=True)
    gap = C(real).mean() - C(fake).mean()
    gp = gradient_penalty(C, real, fake, lambda_gp)
    critic_loss = -gap + gp
    critic_loss.backward()
    opt_c.step()
    opt_c.zero_grad(set_to_none=True)
    flags = [p.requires_grad for p in C.parameters()]
    try:
        for p in C.parameters():
            p.requires_grad_(False)
        fake = G(torch.randn(bs, latent_dim, device=real.device, dtype=real.dtype))
        generator_loss = -C(fake).mean()
        generator_loss.backward()
        opt_g.step()
    finally:
        for p, flag in zip(C.parameters(), flags):
            p.requires_grad_(flag)
    return gap.item(), gp.item(), generator_loss.item()

The penalty must be differentiated with respect to critic parameters through the input-gradient computation. create_graph=True retains that derivative graph; without it, the computed input gradients are detached from the critic parameters and the regularizer cannot supply the intended update. This involves mixed derivatives, not just a second derivative with respect to the input. Interpolation endpoints are detached so the critic penalty does not train the generator.

The penalty samples only some interpolated points and pushes their gradient norms toward 1. It does not enforce a global Lipschitz bound, nor is “norm equals 1” identical to “norm at most 1.” Log the real-minus-fake critic gap and the penalty separately: their combination is a training objective, not a Wasserstein distance or a guaranteed quality score. Additional critic steps can improve its estimate, but a fixed count such as five does not establish near-optimality or convergence.

The gradient calculation assumes each critic output depends only on its corresponding input. Training-mode BatchNorm couples examples, so the gradient of summed outputs used here is not the intended per-example quantity. Omit it in this critic; LayerNorm or GroupNorm avoids that particular cross-example coupling. Those choices alone do not guarantee a Lipschitz bound. The update above takes one critic step per generator step; other ratios require their own tuning.

Run both updates on two-dimensional data

Here G maps two noise coordinates to a two-coordinate point, and D maps a point to one scalar. The logistic discriminator treats that scalar as a logit; the Wasserstein critic uses it directly as a score. We start both runs from copies of the same initial weights, but this short run is a check of the training code, not a controlled quality comparison or a claim of convergence. Move both networks and real inputs to the same device and dtype if adapting it.

from copy import deepcopy
torch.manual_seed(9)
G0 = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 2))
D0 = nn.Sequential(nn.Linear(2, 32), nn.LeakyReLU(0.2), nn.Linear(32, 1))
angles = torch.arange(8) * (2 * torch.pi / 8)
centers = torch.stack([angles.cos(), angles.sin()], dim=1)

def real_batch(n):
    return centers[torch.randint(8, (n,))] + 0.05 * torch.randn(n, 2)

fixed_z = torch.randn(16, 2)
for kind in ("logistic", "wgan_gp"):
    G, D = deepcopy(G0), deepcopy(D0)
    opt_g = torch.optim.Adam(G.parameters(), lr=0.001, betas=(0.0, 0.9))
    opt_d = torch.optim.Adam(D.parameters(), lr=0.001, betas=(0.0, 0.9))
    for step in range(50):
        real = real_batch(64)
        if kind == "logistic":
            metrics = train_step(G, D, opt_g, opt_d, real, 2, "cpu")
        else:
            metrics = wgan_gp_step(G, D, opt_g, opt_d, real, 2)
    G.eval()
    with torch.no_grad():
        generated = G(fixed_z)
    print(kind, "last metrics", [round(v, 4) for v in metrics])
    print("generated shape", tuple(generated.shape),
          "finite", torch.isfinite(generated).all().item())
# logistic last metrics [1.3574, 0.7019]
# generated shape (16, 2) finite True
# wgan_gp last metrics [-0.5172, 1.407, -0.5456]
# generated shape (16, 2) finite True

The logistic metrics are discriminator and generator losses. The WGAN-GP metrics are the critic gap before its update, the weighted gradient penalty, and the generator loss after that update. Their magnitudes are not directly comparable across objectives. The negative critic gap in this run (−0.5172) is not a negative distance: this incompletely optimized critic scores fakes higher on that batch. Finite output and successful updates establish that this example runs; sample coverage still requires evaluation.

Architecture and optimization choices

• If using tanh for image output, scale targets to [-1,1]. Unbounded data, as in the two-dimensional example, can use a linear output.
• Resize followed by convolution can reduce artifacts caused by uneven transposed-convolution overlap. Kernel and stride choices matter; neither upsampling method guarantees artifact-free images.
• LeakyReLU keeps a nonzero local derivative on its negative branch. One inactive ReLU need not block every path from discriminator to generator.
• Optimizer settings depend on the formulation. The toy example uses Adam with betas=(0.0,0.9); treat learning rates, momentum, and update ratio as choices to evaluate, not universal stability rules.
• Spectral normalization constrains layer weight norms and can help control sensitivity. Whole-network bounds also depend on the layers and their composition.
• One-sided label smoothing modifies real targets in a logistic discriminator loss. It changes the classification objective and does not apply to an unrestricted Wasserstein critic.
• Inspect a fixed batch of noise samples over time, alongside fresh samples, coverage measurements, and training diagnostics.

Evaluation

Fréchet Inception Distance (FID) compares Gaussian approximations formed from the means and covariances of real and generated Inception features. Lower values indicate closer feature moments under that protocol, not necessarily better human-perceived quality. Use the same feature extractor, preprocessing, reference split, and sample counts for controlled comparisons. Equal sample counts still do not eliminate model-dependent finite-sample bias. FID can also miss memorization, so compare against held-out real data and inspect nearest neighbors when relevant.

Generative precision and recall estimate sample fidelity and coverage using geometry in a chosen feature space. They are not direct measurements of human realism or a count of semantic categories. A collapsed generator can have high precision and low recall if its few outputs are good; poor collapsed outputs may score badly on both. Report metric definitions, reference data, and sample counts rather than treating one scalar as a complete assessment.

The latent-variable formulation and the reparameterization trick are in Variational Autoencoders: ELBO, Reparameterization, and Latent Space.

Choosing adversarial generation

A feed-forward GAN generator can produce an output in one network pass, which is useful when latency matters. Compare actual speed, quality, coverage, and training cost at the target resolution; architecture size and the number of sampling steps in competing methods affect the trade-off. Single-pass generation alone does not make one model best for every application.

Adversarial losses can also be combined with reconstruction or perceptual objectives in restoration, image translation, and learned image autoencoders. The reconstruction terms constrain correspondence to the input while the adversarial term rewards matching output statistics. Their weights determine the balance, including the risk of generating plausible details that were absent from the input.

Diffusion-based generation is introduced in Diffusion Models Explained: DDPM, Samplers, and Guidance.

Exercises

1. Saturating versus non-saturating. Tabulate the generator-loss derivative with respect to discriminator logit a, where D=sigmoid(a), as D goes from 0.5 to 0.001. Explain why differentiating only with respect to D gives an incomplete account.

The sigmoid derivative changes the comparison: the saturating logit derivative tends to zero, and the non-saturating one tends to −1.

for d in (0.5, 0.1, 0.01, 0.001):
    sigmoid_derivative = d * (1 - d)
    sat = (-1 / (1 - d)) * sigmoid_derivative
    non = (-1 / d) * sigmoid_derivative
    print(f"D={d:.3f} abs dL/da: saturating {abs(sat):.3f} non-saturating {abs(non):.3f}")
# D=0.500 abs dL/da: saturating 0.500 non-saturating 0.500
# D=0.100 abs dL/da: saturating 0.100 non-saturating 0.900
# D=0.010 abs dL/da: saturating 0.010 non-saturating 0.990
# D=0.001 abs dL/da: saturating 0.001 non-saturating 0.999

2. Compare distances between point masses. Compute JS divergence and Wasserstein-1 distance between mass at 0 and mass at theta. Include theta=0 and distinguish continuity from differentiability.

You should get: one distance that is constant everywhere except at zero, and one that is linear in the gap.

import math
for theta in (1.0, 0.1, 0.01, 0.001, 0.0):
    js = math.log(2) if theta != 0 else 0.0
    w1 = abs(theta)
    print(f"theta={theta:6.3f}  JS {js:.4f}   W1 {w1:.4f}")
# theta= 1.000  JS 0.6931   W1 1.0000
# theta= 0.100  JS 0.6931   W1 0.1000
# theta= 0.010  JS 0.6931   W1 0.0100
# theta= 0.001  JS 0.6931   W1 0.0010
# theta= 0.000  JS 0.0000   W1 0.0000

3. The same generator, two discriminators. Let real data be uniform over eight point masses and generated data uniform over only two. Compare the loss of a constant discriminator with the exact optimal discriminator, and report mode coverage.

Six of eight modes, or 75%, are missing. A constant discriminator has the chance-level loss even though an optimal discriminator distinguishes the distributions.

import numpy as np
rng = np.random.default_rng(0)
p_real = np.full(8, 1/8)
p_fake = np.array([1/2, 1/2, 0, 0, 0, 0, 0, 0])
real_modes = rng.choice(8, size=2000, p=p_real)
fake_modes = rng.choice(8, size=2000, p=p_fake)
print("observed modes: real", len(np.unique(real_modes)), "fake", len(np.unique(fake_modes)))
d_star = p_real / (p_real + p_fake)
positive = p_fake > 0
optimal_loss = -(p_real * np.log(d_star)).sum() - (p_fake[positive] * np.log1p(-d_star[positive])).sum()
print("constant D loss", round(2*np.log(2), 4))
print("optimal D on generated modes", d_star[:2])
print("optimal D loss", round(optimal_loss, 4))
print("missing fraction", 1 - np.count_nonzero(p_fake)/8)
# observed modes: real 8 fake 2
# constant D loss 1.3863
# optimal D on generated modes [0.2 0.2]
# optimal D loss 0.6255
# missing fraction 0.75

References

• Goodfellow et al. (2014). Generative Adversarial Nets.
• Radford, Metz, and Chintala (2016). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. ICLR.
• Salimans et al. (2016). Improved Techniques for Training GANs. NIPS.
• Arjovsky, Chintala, and Bottou (2017). Wasserstein GAN.
• Gulrajani, Ahmed, Arjovsky, Dumoulin, and Courville (2017). Improved Training of Wasserstein GANs. NIPS.
• Metz, Poole, Pfau, and Sohl-Dickstein (2017). Unrolled Generative Adversarial Networks. ICLR.
• Heusel, Ramsauer, Unterthiner, Nessler, and Hochreiter (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NIPS.
• Miyato, Kataoka, Koyama, and Yoshida (2018). Spectral Normalization for Generative Adversarial Networks. ICLR.
• Chong and Forsyth (2020). Effectively Unbiased FID and Inception Score and Where to Find Them. CVPR.
• Kynkaanniemi, Karras, Laine, Lehtinen, and Aila (2019). Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS.