Fréchet Inception Distance (FID)

FID scores how close a set of generated images is to a set of real images. It does not compare images pair by pair. It passes both sets through a fixed pretrained network, fits a Gaussian to each set of feature vectors, and measures the distance between the two Gaussians. Lower is better, and 0 means the two fitted Gaussians coincide.

Definition

Heusel et al. introduced FID with the pooled 2,048-dimensional features of an Inception-v3 network trained on ImageNet. For features of real images with mean \(\mu_r\) and covariance \(\Sigma_r\), and generated images with \(\mu_g\) and \(\Sigma_g\),

\[ \mathrm{FID}=\lVert\mu_r-\mu_g\rVert^2+\operatorname{Tr}\!\Big(\Sigma_r+\Sigma_g-2\,(\Sigma_r\Sigma_g)^{1/2}\Big). \]

The first term is the squared distance between the average features. The second compares spreads: for one-dimensional Gaussians it is \((\sigma_r-\sigma_g)^2\), the squared difference of standard deviations, and the matrix version also compares the orientations of the covariances. The formula is the squared Fréchet (2-Wasserstein) distance between two Gaussians. A check in one dimension: for \(N(0,1)\) and \(N(1,4)\) it gives \(1 + 1 + 4 – 2\cdot2 = 2\), and 200,000 samples from each give 1.985.

The program below implements the formula (the trace of the matrix square root equals the sum of the square roots of the eigenvalues of \(\Sigma_r\Sigma_g\)) and runs three experiments on synthetic features: the one-dimensional check, two sets from the same Gaussian at growing sample sizes, and a two-cluster dataset against a Gaussian with the same mean and covariance.

import numpy as np
rng = np.random.default_rng(1)
def fd(a, b):
    m1, m2 = a.mean(0), b.mean(0)
    S1, S2 = np.atleast_2d(np.cov(a, rowvar=False)), np.atleast_2d(np.cov(b, rowvar=False))
    ev = np.linalg.eigvals(S1 @ S2).real.clip(0)      # trace of (S1 S2)^(1/2) = sum of sqrt eigenvalues
    return float(((m1 - m2) ** 2).sum() + np.trace(S1) + np.trace(S2) - 2 * np.sqrt(ev).sum())

a = rng.normal(0, 1, (200000, 1)); b = rng.normal(1, 2, (200000, 1))
print("1-D check, expected 2.0, got %.3f" % fd(a, b))
d = 32
for n in (64, 256, 1024, 4096, 16384):                 # same distribution, true distance 0
    v = [fd(rng.normal(size=(n, d)), rng.normal(size=(n, d))) for _ in range(5)]
    print(f"n={n:6d}  FD between two samples of the SAME distribution: {np.mean(v):.3f}")

n = 50000                                              # two well-separated clusters vs a same-moment Gaussian
modes = np.array([[-2., 0], [2., 0]])
real = modes[rng.integers(0, 2, n)] + rng.normal(size=(n, 2)) * 0.3
fake = rng.multivariate_normal(real.mean(0), np.cov(real, rowvar=False), n)
print("mixture vs same-moment Gaussian, FD = %.4f" % fd(real, fake))
print("fraction of fake samples between the modes (|x|<1): %.3f; real: %.3f" % ((np.abs(fake[:, 0]) < 1).mean(), (np.abs(real[:, 0]) < 1).mean()))
one = modes[0] + rng.normal(size=(n, 2)) * 0.3
print("one-mode fake, FD = %.3f" % fd(real, one))

1-D check, expected 2.0, got 1.985
n=    64  FD between two samples of the SAME distribution: 9.539
n=   256  FD between two samples of the SAME distribution: 2.333
n=  1024  FD between two samples of the SAME distribution: 0.588
n=  4096  FD between two samples of the SAME distribution: 0.147
n= 16384  FD between two samples of the SAME distribution: 0.036
mixture vs same-moment Gaussian, FD = 0.0001
fraction of fake samples between the modes (|x|<1): 0.378; real: 0.000
one-mode fake, FD = 6.918

Sample-size bias

The covariance matrices are estimated from a finite sample, and the estimation error adds a positive bias to the distance. In the second experiment, two independent sets come from the same 32-dimensional Gaussian, so the true distance is 0. Averaged over five repeats, FD is 9.54 at 64 samples per set, 2.33 at 256, 0.59 at 1,024, 0.15 at 4,096 and 0.036 at 16,384.

The bias falls roughly in proportion to \(1/n\), and it grows with feature dimension because there are more covariance entries to estimate (Inception features have 2,048 dimensions, not 32). The practical rules are that FID values are comparable only at the same sample count, the same feature network and the same image preprocessing, and that comparisons should account for uncertainty in the difference and model-dependent bias. A common bias can cancel between two estimates, so a difference smaller than an absolute bias is not automatically uninformative. Repeated samples or paired resampling show how stable the difference is.

What it misses

FID sees only the mean and covariance of the features. Two sets with the same first two moments score 0 whatever their shapes. In the third experiment the real data is two well-separated clusters and the fake set is drawn from a single Gaussian with the same mean and covariance, so the fake samples fill the gap between the clusters.

The fake set puts 37.8 percent of its samples where the real data has none, and the distance is still 0.0001. Dropping one of the two clusters is detected (6.9). In real images the Gaussian assumption fails for the same reason: the feature distribution is multimodal and has heavy tails.

Other limits are inherited from the feature network. Inception was trained on ImageNet classification, so it encodes what separates ImageNet classes and tends to be less sensitive to other qualities, such as text rendering or hands, and FID does not use the prompt at all, so it says nothing about whether an image matches its caption. Kernel-based alternatives avoid the Gaussian assumption; CMMD (Jayasumana et al.) compares CLIP embeddings with a maximum mean discrepancy and has an unbiased estimator.

Fidelity versus diversity

A single distance mixes two properties: whether generated images look real (fidelity) and whether they cover the variety of real images (diversity). Raising a diffusion model’s guidance scale usually trades diversity for fidelity up to a point, and an FID sweep over guidance values shows the combined effect of both changes, so FID can move in either direction along it. Because FID is one number built from a mean and a covariance, the sweep alone cannot say how much of a change came from fidelity and how much from diversity. Separating the two needs additional measures, such as precision and recall estimated from nearest neighbors in feature space, or another diversity metric.

References

• Heusel, Ramsauer, Unterthiner, Nessler, Hochreiter, GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium — https://arxiv.org/abs/1706.08500
• Jayasumana et al., Rethinking FID: Toward a Better Evaluation Metric for Image Generation — https://arxiv.org/abs/2401.09603