Energy Distance
Definition
Energy distance is a statistical distance between two probability distributions $P$ and $Q$.
It is based on the idea of comparing expected pairwise distances between samples.
- If energy distance = 0, the two distributions are identical.
- A larger value means the distributions are more different.
It’s often used as a two-sample test (similar to MMD) to check whether two datasets come from the same distribution.
Mathematical Formula
For random variables $X \sim P$, $Y \sim Q$:
$D_E^2(P, Q) = 2 \, \mathbb{E}\|X – Y\| \;-\; \mathbb{E}\|X – X’\| \;-\; \mathbb{E}\|Y – Y’\|$
Where:
- $X, X’$ are i.i.d. samples from $P$.
- $Y, Y’$ are i.i.d. samples from $Q$.
- $\|\cdot\|$ is the Euclidean norm.
Empirical Version (with samples)
Given two samples $X = \{x_1, \dots, x_m\}$ and $Y = \{y_1, \dots, y_n\}$:
$\hat{D}_E^2(P, Q) = \frac{2}{mn} \sum_{i=1}^m \sum_{j=1}^n \|x_i – y_j\| – \frac{1}{m^2} \sum_{i=1}^m \sum_{j=1}^m \|x_i – x_j\| – \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n \|y_i – y_j\|$
Key Properties
- Metric property: Energy distance is always non-negative, symmetric, and equals 0 if $P = Q$.
- No kernel needed: Unlike MMD, it works directly with Euclidean distances (though it can be generalized).
- Good in high dimensions: Like MMD, it can detect subtle differences where simpler tests (e.g., KS test) fail.
Applications
- Two-sample testing: Hypothesis test whether $P = Q$.
- Drift detection: Compare training vs. production distributions.
- GAN evaluation: Like MMD, used to check if generated samples match real data.
- Clustering validation: Related to the “energy statistic” used in clustering and goodness-of-fit tests.
Comparison: MMD vs. Energy Distance
| Aspect | MMD | Energy Distance |
|---|---|---|
| Based on | Kernel methods (RKHS mean embeddings) | Euclidean distances |
| Flexibility | Choice of kernel changes sensitivity | More direct, kernel-free |
| Usage | GAN eval, domain adaptation, drift | Two-sample testing, drift, clustering |
| Similarity | Both are Integral Probability Metrics (IPMs) | Yes, also an IPM |
Example
- Suppose we test whether two customer age distributions are the same.
- Energy distance = 0.15 → distributions are fairly similar.
- Energy distance = 1.2 → distributions differ significantly (maybe younger people dominate the new sample).
Summary:
Energy distance is a distributional distance measure that uses expected pairwise Euclidean distances.
It is kernel-free (unlike MMD), works well in high dimensions, and is widely used for two-sample testing and drift detection.