Jensen–Shannon (JS) Divergence

Jensen–Shannon divergence is KL divergence made symmetric by comparing both distributions against their midpoint:

\[ JS(P \parallel Q) = \tfrac{1}{2}D_{KL}(P \parallel M) + \tfrac{1}{2}D_{KL}(Q \parallel M), \quad M = \tfrac{1}{2}(P + Q) \]

That one change fixes the two things that make KL awkward to use as a score, and introduces a limitation of its own.

What the midpoint buys

import numpy as np

def kl(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    with np.errstate(divide="ignore", invalid="ignore"):
        return float(np.where(p > 0, p * np.log(p / q), 0.0).sum())

def js(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    m = (p + q) / 2
    return 0.5 * kl(p, m) + 0.5 * kl(q, m)

P, Q = [0.5, 0.5], [0.9, 0.1]
print(f"  P = {P}, Q = {Q}, midpoint M = {[(a + b) / 2 for a, b in zip(P, Q)]}")
print(f"    JS(P || Q) = {js(P, Q):.6f}   JS(Q || P) = {js(Q, P):.6f}   symmetric: "
      f"{js(P, Q) == js(Q, P)}")
print(f"    KL(P || Q) = {kl(P, Q):.6f}   KL(Q || P) = {kl(Q, P):.6f}   these differ")
"""Averaging against the midpoint is what buys symmetry, and the midpoint is
   never zero where either side is positive - which is what buys finiteness."""

print(f"\n  KL runs to infinity as the overlap disappears; JS stops at ln 2 = {np.log(2):.6f}:")
print(f"{'Q':>22}{'KL(P || Q)':>14}{'JS(P || Q)':>14}")
for eps in (0.1, 0.01, 0.001, 0.0):
    q = [1 - eps, eps]
    k = kl(P, q)
    print(f"{str([round(v, 4) for v in q]):>22}"
          f"{k if np.isfinite(k) else float('inf'):>14.6f}{js(P, q):>14.6f}")
print("  the last row has KL infinite while JS reads 0.215762 - the two still")
print("  overlap in the first cell, so this is finiteness, not the bound being hit")

print(f"\n  the bound is why JS fails as a training signal for disjoint distributions:")
for gap in (0.0, 0.25, 0.5, 1.0):
    a = np.array([0.5, 0.5, 0.0, 0.0])
    b = np.array([0.0, 0.0, 0.5, 0.5])
    mix = (1 - gap) * a + gap * b
    print(f"    overlap parameter {1 - gap:>4.2f}: JS = {js(a, mix):.6f}")
print("  once two distributions stop overlapping, JS is constant at 0.693147 and")
print("  it stays constant along parameter paths that keep the supports disjoint")

print(f"\n  the square root obeys the triangle inequality, and JS itself does not:")
A, B, C = [0.98, 0.01, 0.01], [0.01, 0.98, 0.01], [0.495, 0.495, 0.01]
d = lambda x, y: np.sqrt(js(x, y))
print(f"    sqrt: d(A,C) + d(C,B) = {d(A, C) + d(C, B):.6f} >= d(A,B) = {d(A, B):.6f}"
      f"   holds: {d(A, C) + d(C, B) >= d(A, B)}")
print(f"    raw:  JS(A,C) + JS(C,B) = {js(A, C) + js(C, B):.6f} vs JS(A,B) = {js(A, B):.6f}"
      f"   holds: {js(A, C) + js(C, B) >= js(A, B)}")
print("  so the metric is the square root; the divergence itself is only a score")
#   P = [0.5, 0.5], Q = [0.9, 0.1], midpoint M = [0.7, 0.3]
#     JS(P || Q) = 0.101749   JS(Q || P) = 0.101749   symmetric: True
#     KL(P || Q) = 0.510826   KL(Q || P) = 0.368064   these differ
#
#   KL runs to infinity as the overlap disappears; JS stops at ln 2 = 0.693147:
#                      Q    KL(P || Q)    JS(P || Q)
#             [0.9, 0.1]      0.510826      0.101749
#           [0.99, 0.01]      1.614463      0.193187
#         [0.999, 0.001]      2.761231      0.212357
#             [1.0, 0.0]           inf      0.215762
#   the last row has KL infinite while JS reads 0.215762 - the two still
#   overlap in the first cell, so this is finiteness, not the bound being hit
#
#   the bound is why JS fails as a training signal for disjoint distributions:
#     overlap parameter 1.00: JS = 0.000000
#     overlap parameter 0.75: JS = 0.095603
#     overlap parameter 0.50: JS = 0.215762
#     overlap parameter 0.00: JS = 0.693147
#   once two distributions stop overlapping, JS is constant at 0.693147 and
#   it stays constant along parameter paths that keep the supports disjoint
#
#   the square root obeys the triangle inequality, and JS itself does not:
#     sqrt: d(A,C) + d(C,B) = 0.874253 >= d(A,B) = 0.793924   holds: True
#     raw:  JS(A,C) + JS(C,B) = 0.382159 vs JS(A,B) = 0.630315   holds: False
#   so the metric is the square root; the divergence itself is only a score

Symmetry is immediate and exact: \(JS(P \parallel Q)\) and \(JS(Q \parallel P)\) are the same floating-point number, where the two KL values differ (0.510826 against 0.368064).

Finiteness follows from the same construction. The midpoint is zero only where both inputs are zero, so no term ever divides by zero. The second table shows KL running to infinity as \(Q\) starves one outcome while JS stays at 0.215762 — still overlapping in the other cell, so this is the finiteness rather than the ceiling being reached.

That property is what makes JS usable on empirical distributions where KL is not. An observed bin with zero count in one sample and positive count in the other breaks KL outright and leaves JS well defined, which is why drift monitoring built on sample histograms tends to use it.

The bound, and the cost of it

JS is bounded above by \(\ln 2 = 0.693147\) in nats, reached exactly when the two distributions share no support. The third block walks a distribution from full overlap to none: 0.000000, 0.095603, 0.215762, 0.693147.

A bounded score is easy to read and easy to threshold, and that same bound is a real limitation when JS is used as a training objective. If a generator parameter can vary locally while the two supports remain disjoint, JS stays at 0.693147 along that path and supplies no gradient there. A path that moves probability mass into overlap need not have zero derivative; disjoint supports alone do not make every possible gradient zero. This is the vanishing-gradient problem that original GANs hit, and the reason later work moved to divergences that keep responding after the overlap is gone.

For measurement rather than optimization the bound is a convenience. A JS value can be read as a fraction of its maximum, and the square root, \(\sqrt{JS}\), ranges from 0 to \(\sqrt{\ln 2}\) and is a genuine metric.

Only the square root is a metric

The last block checks that claim rather than asserting it. For three distributions A, B and C, the square-root version satisfies the triangle inequality — 0.874253 against 0.793924 — while the raw divergence does not: \(JS(A,C) + JS(C,B) = 0.382159\) is smaller than \(JS(A,B) = 0.630315\).

So the distinction is not pedantry. Anything that assumes a metric — clustering on distances, nearest-neighbor search over distributions, a triangle-inequality shortcut in an index — needs \(\sqrt{JS}\), sometimes called the Jensen–Shannon distance. The raw divergence is a comparable score and nothing stronger.

One practical footnote: the value depends on the logarithm base. In nats the maximum is 0.693147; in bits it is exactly 1, which is why some libraries report JS on a 0-to-1 scale. A reported JS figure needs its base stated before it can be compared with another.

References

• NumPy documentation, numpy.log and numpy.sqrt. numpy.org