Variational Inference (VI)
At a correlation of 0.99, mean-field reports a standard deviation of 0.141067 where the truth is 1.0.
-
-
Kullback-Leibler Divergence
An optimal code for P costs 1 bit; a code for Q costs 1.736966 on samples from P. The 0.736966 gap is the KL.
-
Jensen–Shannon (JS) Divergence
JS reaches 0.693147 nats for disjoint supports. A parameter path that keeps them disjoint has a constant objective, unlike one that changes overlap.
-
Probability and Information Theory
Every loss function in this series is a probability statement in disguise. Here is the vocabulary.