Variational Inference (VI)
1. The Problem: Bayesian Inference
In Bayesian statistics, we want to compute the posterior distribution:
$p(\theta \mid x) = \frac{p(x \mid \theta) p(\theta)}{p(x)}$
- $x$: observed data
- $\theta$: latent variables or parameters
- $p(x \mid \theta)$: likelihood
- $p(\theta)$: prior
- $p(x)$: evidence (marginal likelihood)
The challenge: computing $p(x)$ requires integrating over all $\theta$:
$p(x) = \int p(x \mid \theta) p(\theta) \, d\theta$
This integral is intractable in most real-world models.
2. The Idea of Variational Inference
Since computing the exact posterior is too hard, VI approximates it.
- Introduce a family of simpler distributions $q(\theta)$ (e.g., Gaussian family).
- Find the member of this family that is closest to the true posterior $p(\theta \mid x)$.
Formally, we solve:
$q^*(\theta) = \arg \min_q \, \text{KL}(q(\theta) \parallel p(\theta \mid x))$
where $\text{KL}$ is the Kullback-Leibler (KL) divergence.
3. Evidence Lower Bound (ELBO)
Because $p(\theta \mid x)$ is unknown, we can’t minimize KL directly. Instead, we maximize the Evidence Lower Bound (ELBO):
$\mathcal{L}(q) = \mathbb{E}_{q(\theta)} \big[ \log p(x, \theta) – \log q(\theta) \big]$
Properties:
- Maximizing ELBO ≡ Minimizing KL divergence.
- ELBO provides a lower bound on $\log p(x)$.
- Optimization is usually done with gradient-based methods.
4. How It Works in Practice
Steps:
- Choose a variational family $q(\theta; \phi)$, parameterized by $\phi$ (e.g., mean and variance of a Gaussian).
- Optimize $\phi$ to maximize ELBO.
- Use $q(\theta; \phi)$ as an approximation of the posterior.
5. Common Techniques
- Mean-field approximation: Assume all latent variables are independent:
- $q(\theta) = \prod_i q_i(\theta_i)$
- Coordinate ascent VI (CAVI): Update each qiq_iqi one at a time.
- Stochastic VI (SVI): Use stochastic optimization + mini-batches for big data.
- Reparameterization trick:
- Write $\theta = g(\phi, \epsilon)$ with noise $\epsilon$, which allows gradient-based optimization (used in Variational Autoencoders).
6. Applications
- Topic modeling (Latent Dirichlet Allocation)
- Variational Autoencoders (VAEs) in deep learning
- Bayesian neural networks
- Probabilistic graphical models
Summary:
Variational inference is like replacing a hard math problem (computing the true posterior) with an easier optimization problem (finding the best approximate distribution within a simpler family). Instead of exact answers, you get a fast, good-enough approximation.