Kullback-Leibler Divergence

Kullback–Leibler divergence measures how much one distribution differs from another, in a specific and directional sense:

\[ D_{KL}(P \parallel Q) = \sum_x P(x)\log\frac{P(x)}{Q(x)} \]

It is also called relative entropy, and the second name is the more honest one: it is a cost, not a distance.

A cost you can count in bits

import numpy as np

def kl(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    with np.errstate(divide="ignore", invalid="ignore"):
        terms = np.where(p > 0, p * np.log(p / q), 0.0)
    return float(terms.sum())

P, Q = [0.5, 0.5], [0.9, 0.1]
print(f"  P = {P}, Q = {Q}")
print(f"    KL(P || Q) = {kl(P, Q):.6f} nats")
print(f"    KL(Q || P) = {kl(Q, P):.6f} nats")
print(f"  the same pair, two answers - KL is a direction, not a distance")
"""It is the extra cost of encoding samples from the first distribution with a
   code built for the second. Swapping them changes what is being encoded."""

print(f"\n  and the cost interpretation checks out, in bits:")
bits_true = -sum(p * np.log2(p) for p in P)
bits_wrong = -sum(p * np.log2(q) for p, q in zip(P, Q))
print(f"    optimal code for P costs {bits_true:.6f} bits per symbol")
print(f"    a code built for Q costs {bits_wrong:.6f} bits on samples from P")
print(f"    the gap {bits_wrong - bits_true:.6f} equals KL(P || Q) in bits: "
      f"{kl(P, Q) / np.log(2):.6f}")

print(f"\n  a zero in the wrong place makes it infinite:")
for q in ([0.9, 0.1], [0.99, 0.01], [0.999, 0.001], [1.0, 0.0]):
    value = kl(P, q)
    print(f"    Q = {str(q):<16} KL(P || Q) = {value:.6f}" if np.isfinite(value)
          else f"    Q = {str(q):<16} KL(P || Q) = infinite")
print("  any outcome P gives probability to and Q gives zero costs unbounded surprise")

print(f"\n  the asymmetry has a direction, and it decides what a fit is forced to do:")
target = np.array([0.45, 0.10, 0.45])
covers = np.array([0.40, 0.20, 0.40])
ignores = np.array([0.98, 0.01, 0.01])
outside = np.array([0.45, 0.10, 0.45])
strict = np.array([0.5, 0.5, 0.0])
print(f"    target {target.tolist()}")
print(f"{'fit':>34}{'KL(target || fit)':>20}{'KL(fit || target)':>20}")
for name, f in (("covers both modes", covers), ("ignores the right mode", ignores)):
    print(f"{name:>34}{kl(target, f):>20.6f}{kl(f, target):>20.6f}")
print(f"  dropping a mode costs {kl(target, ignores):.4f} under the forward form and"
      f" {kl(ignores, target):.4f} under the reverse.")
print("  minimizing the forward form forces a fit to cover everything the target")
print("  reaches, and the reverse form does not - which is the whole of mode-seeking")

print(f"\n  and each direction blows up on a different kind of zero:")
print(f"    target {strict.tolist()} has an empty third cell")
print(f"    a fit that puts mass there: KL(fit || target) is infinite")
print(f"    the same pair the other way:  KL(target || fit) = {kl(strict, outside):.6f}")
print("  forward KL punishes missing mass; reverse KL punishes inventing it")
#   P = [0.5, 0.5], Q = [0.9, 0.1]
#     KL(P || Q) = 0.510826 nats
#     KL(Q || P) = 0.368064 nats
#   the same pair, two answers - KL is a direction, not a distance
#
#   and the cost interpretation checks out, in bits:
#     optimal code for P costs 1.000000 bits per symbol
#     a code built for Q costs 1.736966 bits on samples from P
#     the gap 0.736966 equals KL(P || Q) in bits: 0.736966
#
#   a zero in the wrong place makes it infinite:
#     Q = [0.9, 0.1]       KL(P || Q) = 0.510826
#     Q = [0.99, 0.01]     KL(P || Q) = 1.614463
#     Q = [0.999, 0.001]   KL(P || Q) = 2.761231
#     Q = [1.0, 0.0]       KL(P || Q) = infinite
#   any outcome P gives probability to and Q gives zero costs unbounded surprise
#
#   the asymmetry has a direction, and it decides what a fit is forced to do:
#     target [0.45, 0.1, 0.45]
#                                fit   KL(target || fit)   KL(fit || target)
#                  covers both modes            0.036690            0.044403
#             ignores the right mode            1.593019            0.701646
#   dropping a mode costs 1.5930 under the forward form and 0.7016 under the reverse.
#   minimizing the forward form forces a fit to cover everything the target
#   reaches, and the reverse form does not - which is the whole of mode-seeking
#
#   and each direction blows up on a different kind of zero:
#     target [0.5, 0.5, 0.0] has an empty third cell
#     a fit that puts mass there: KL(fit || target) is infinite
#     the same pair the other way:  KL(target || fit) = 0.857399
#   forward KL punishes missing mass; reverse KL punishes inventing it

The second block makes the interpretation concrete rather than metaphorical. An optimal code for \(P = (0.5, 0.5)\) costs 1 bit per symbol. A code built for \(Q = (0.9, 0.1)\), applied to samples that really come from \(P\), costs 1.736966 bits. The gap is 0.736966 bits, and that is exactly \(D_{KL}(P \parallel Q)\) converted from nats.

So KL divergence is the price of being wrong about which distribution you are facing, measured in the units information is measured in. That framing also explains why it is never negative and zero only when the two agree everywhere.

The first block is the property that disqualifies it as a distance: \(D_{KL}(P \parallel Q) = 0.510826\) while \(D_{KL}(Q \parallel P) = 0.368064\). The same pair, two answers. Writing “the KL divergence between A and B” is incomplete — which one is the reference matters, and swapping them changes the number.

Zeros, and which direction you minimize

The third block is the practical failure mode. As \(Q\) shrinks toward zero on an outcome \(P\) still gives weight to, the divergence grows without bound — 0.510826, 1.614463, 2.761231, and then infinite. One outcome the reference assigns zero probability to and the data does not is enough.

This is why KL estimated from samples needs care: an empirical distribution has zeros wherever nothing was observed, so a naive plug-in estimate returns infinity on any bin a comparison sample happened to miss. Smoothing or binning is not a refinement here, it is a prerequisite.

The last blocks show what the asymmetry does when KL is used as an objective. Dropping a mode of the target costs 1.5930 under the forward form and 0.7016 under the reverse. And each direction blows up on a different kind of zero: forward KL punishes a fit that misses mass the target has, reverse KL punishes a fit that invents mass the target does not.

That is the whole of the mode-seeking story. In common variational inference, minimizing reverse KL over a restricted family that cannot represent the target’s separated modes can favor concentrating on one mode rather than covering the low-density gap. This is a tendency of the objective and approximation family, not a requirement that every variational approximation choose one mode. If the family contains the full target, that target minimizes either KL direction. The same choice explains why the KL term in a variational autoencoder pulls the latent distribution toward the prior rather than the reverse.

Where it turns up

Cross-entropy loss is KL divergence plus a constant: minimizing the cross-entropy between the true labels and a model's predictions is minimizing \(D_{KL}\) between them, since the label distribution's own entropy does not depend on the model. Much of classification training is therefore KL minimization under another name, as noted under likelihood.

For drift detection the asymmetry and the infinity both matter: which distribution is the reference changes the number, and a production bin with no training data in it breaks the calculation outright. Where a bounded symmetric score is wanted instead, Jensen–Shannon divergence is built from this one for exactly that reason.

References

• NumPy documentation, numpy.log and numpy.errstate. numpy.org