Variational Inference (VI)

Variational inference turns the problem of computing a posterior into an optimization. Instead of integrating, you pick a family of simpler distributions and find the member closest to the posterior:

\[ q^*(\theta) = \arg\min_q \, \mathrm{KL}\big(q(\theta) \parallel p(\theta \mid x)\big) \]

That objective contains the thing you could not compute, so it is not directly usable. What makes the method work is that minimizing it is the same as maximizing a quantity you can.

The bound, and what the gap is

import numpy as np
from scipy.stats import beta
from scipy.special import betaln, digamma
from scipy.optimize import minimize

k, n = 7, 10
log_evidence = betaln(k + 1, n - k + 1)  # ordered Bernoulli data, uniform prior

def elbo(log_shapes):
    a, b = np.exp(log_shapes)
    elog_p = digamma(a) - digamma(a + b)
    elog_one_minus_p = digamma(b) - digamma(a + b)
    return ((k - a + 1) * elog_p + (n - k - b + 1) * elog_one_minus_p
            + betaln(a, b))

initial = np.log([7.0, 3.0])
fit = minimize(lambda v: -elbo(v), initial, method="BFGS", tol=1e-10)
a, b = np.exp(fit.x)
print(f"  log evidence {log_evidence:.6f}")
for label, shapes in (("initial Beta(7,3)", initial), ("optimized Beta q", fit.x)):
    value = elbo(shapes)
    print(f"  {label}: ELBO {value:.6f}, KL gap {log_evidence - value:.6f}")
print(f"  fitted shapes {a:.4f}, {b:.4f}; exact posterior Beta(8,4)")

print(f"{'correlation':>13}{'true sd':>10}{'mean-field sd':>16}{'understated by':>17}")
for r in (0.0, 0.5, 0.9, 0.99):
    cov = np.array([[1.0, r], [r, 1.0]])
    precision = np.linalg.inv(cov)
    mf = float(1 / np.sqrt(precision[0, 0]))
    print(f"{r:>13}{np.sqrt(cov[0, 0]):>10.4f}{mf:>16.6f}{1 / mf:>16.2f}x")
#   log evidence -7.185387
#   initial Beta(7,3): ELBO -7.237366, KL gap 0.051979
#   optimized Beta q: ELBO -7.185387, KL gap 0.000000
#   fitted shapes 8.0000, 4.0000; exact posterior Beta(8,4)
#   correlation   true sd   mean-field sd   understated by
#           0.0    1.0000        1.000000            1.00x
#           0.5    1.0000        0.866025            1.15x
#           0.9    1.0000        0.435890            2.29x
#          0.99    1.0000        0.141067            7.09x

The Beta variational family has the same support as the Beta posterior, so its ELBO and KL are finite. The code evaluates the expectations analytically with digamma functions. Starting from Beta(7,3), optimization recovers the Beta(8,4) posterior and the gap approaches zero. This family contains the exact answer; a restricted family need not.

That identity is the whole method. Since the log evidence is a constant that does not depend on \(q\), pushing the bound up must push the divergence down — so maximizing the ELBO minimizes the KL divergence without ever computing the marginal likelihood.

It also means the ELBO doubles as a diagnostic in a limited sense: a higher ELBO for the same model means a tighter approximation. Comparing ELBOs across different models is less safe, because the gap differs and a model with a looser bound can have a higher evidence.

The family you choose is the error you get

The first example checks the ELBO identity in a family that can represent the posterior. The next example restricts a correlated Gaussian posterior to independent Gaussian coordinates, making the approximation error visible.

Mean-field variational inference assumes the coordinates are independent. On a correlated two-dimensional Gaussian, its optimum is known exactly, and the standard deviation it reports is \(1/\sqrt{\Lambda_{ii}}\) from the precision matrix: 0.866025 at a correlation of 0.5, 0.435890 at 0.9, 0.141067 at 0.99. The true value is 1.0 in every row.

So at a correlation of 0.99 a mean-field approximation reports a spread seven times too small. And it gets the means exactly right, which is what makes the failure hard to notice: point estimates look fine while every interval derived from them is far too narrow.

For this correlated Gaussian target and factorized Gaussian family, reverse-KL optimization keeps the means and reduces the marginal variances. That is a result for this family and target, not a universal guarantee for variational inference. Other objectives, families and optimization errors can change means and uncertainty; reverse KL does not imply that every uncertainty measure is always underestimated.

When to reach for it

The comparison with MCMC depends on computational budget, mixing and the variational family. VI can be fast, but stochastic VI is not deterministic, and a restricted family or imperfect optimization can introduce bias. Check quantities relevant to the decision rather than assuming that accurate means imply accurate intervals.

That points to where each belongs. Use variational inference when the model is large, the data is large, and the quantity of interest is a point estimate or a ranking — which is why it underlies variational autoencoders and topic models. When interval accuracy is central or a basic mean-field family cannot represent important dependence or multiple modes, consider a richer variational family or sampling with appropriate convergence checks. Compare decision-relevant posterior quantities; correlation alone does not disqualify all VI, and a sampler is not automatically reliable.

Richer families narrow the gap: full-covariance Gaussians capture correlation, and normalizing flows go further at more cost. None of them remove the question, which is always the same one — how much of the posterior can the family you chose actually represent.

References

• SciPy documentation, scipy.optimize.minimize. docs.scipy.org
• Blei, Kucukelbir and McAuliffe, Variational Inference: A Review for Statisticians. Source