1. What “unknown normalizing factor” means

In many Bayesian models, the likelihood $p(y \mid \theta)$ and prior $p(\theta)$ are known up to a single normalizing constant (the marginal likelihood $p(y)$). That constant does not depend on $\theta$, so it does not interfere with posterior inference.

However, in some models—especially conditional or spatial models—the likelihood itself contains a normalizing factor that depends on θ. This creates a major computational problem.

Such models take the form:

$p(y \mid \theta) = \frac{1}{z(\theta)}\, q(y \mid \theta)$

where:

  • $q(y \mid \theta)$ is an unnormalized density we can compute,
  • $z(\theta) = \int q(y \mid \theta)\, dy$ is an unknown normalizing factor depending on θ.

Because $z(\theta)$ varies with θ, it cannot be ignored in posterior computation.

The posterior then becomes:

$p(\theta \mid y) \propto p(\theta)\, \frac{1}{z(\theta)} \, q(y \mid \theta)$

which requires evaluating $z(\theta)$ (or its approximation) for every needed θ-value.

2. Why this problem is serious

Ignoring $z(\theta)$ leads to incorrect posteriors.

Example:
If a normal likelihood were incorrectly treated as

$p(y\mid \mu,\sigma) \propto e^{-(y-\mu)^2/(2\sigma^2)}$

missing the factor $1/\sigma$, then the resulting posterior would be mathematically wrong.

Thus, when $z(\theta)$ depends on θ, standard MCMC or optimization methods break unless this factor is accounted for.

3. Unknown normalizing factors in hierarchical models

The same issue arises when a population distribution in a hierarchical model is only known up to normalization:

$q(\gamma \mid \phi) = z(\phi)\,p(\gamma \mid \phi)$

Then the joint posterior becomes:

$p(\gamma, \phi \mid y) \propto p(\phi)\, \frac{1}{z(\phi)}\, q(\gamma \mid \phi)\, p(y \mid \gamma)$

Again, $z(\phi)$ must be evaluated because it depends on hyperparameters.

If the likelihood also has its own unknown normalizing factor, both must be included.

4. General computational strategy

When $z(\theta)$ cannot be computed analytically, the following approach is used:

Step 1 — Get an analytic approximation

Use an approximate method such as Laplace’s method centered around a rough estimate of θ.
This provides a crude estimate of $z(\theta)$.

Step 2 — Build an approximate posterior

Construct a posterior approximation (e.g., Gaussian/Laplace, variational, EP, mixture approximation).
This allows quick evaluation of the posterior shape.

Step 3 — Perform more exact computation

Whenever posterior evaluation requires a new θ-value, compute or estimate $z(\theta)$ numerically.

This treats $z(\theta)$ as an expensive but “known” function.

5. Grid strategies

When θ is low-dimensional (1–2 dimensions):

  • Compute $z(\theta)$ on a grid.
  • Interpolate between points.

This avoids computing it everywhere.

Approximate posterior computations (Steps 1–2) help identify which parts of the θ-space actually matter, so we avoid wasting computation in regions with negligible posterior probability.

6. Computing $z(\theta)$ using numerical integration

6.1 Direct numerical integration

Apply quadrature or Laplace approximation directly to $z(\theta) = \int q(y\mid \theta)\,dy$.

This is the simplest approach if q has moderate dimension.

7. Importance sampling for normalizing constants

Use:

$z(\theta) = \int \frac{q(y\mid \theta)}{g(y)} g(y)\,dy = \mathbb{E}_g \left[ \frac{q(y\mid \theta)}{g(y)} \right]$

Estimate as:

$\hat{z}(\theta) = \frac{1}{S} \sum_{s=1}^S \frac{q(y_s \mid \theta)}{g(y_s)}, \quad y_s \sim g(y)$

Key practical notes

  • Use the same sample $y_s$​ to evaluate $z(\theta)$ for multiple θ-values to save computation.
  • g(y) does not need to be normalized, because posterior inference uses ratios.
  • g(y) must approximate q(y|θ) well in regions where θ is plausible.

8. Bridge sampling and path sampling

These advanced methods outperform ordinary importance sampling when a family of posteriors must be evaluated, for different φ or θ.

8.1 Bridge sampling

Useful when distributions vary too much for a single importance density to cover.

Simulate from two “anchor” distributions:

  • $p(\gamma\mid \phi_0, y)$
  • $p(\gamma\mid \phi_1, y)$

Then construct a weighted combination (“bridge”) to estimate normalization for intermediate φ.

8.2 Path sampling

The continuous-limit version of bridge sampling.

Uses identity:

$\frac{d}{d\phi} \log p(\phi \mid y) = \mathbb{E} \left[ U(\gamma,\phi,y) \mid \phi,y \right]$

where:

$U(\gamma,\phi,y) = \frac{d}{d\phi} \log p(\gamma,\phi,y)$

Integrating this expectation over φ gives relative log-normalizing constants.

9. Relation to parallel tempering

Bridge/path sampling resemble tempering, where sampling is done across a family of distributions indexed by a temperature-like parameter.

Tempering is used for mixing efficiency,
while path sampling is used for computing normalizing constants.

Final Notes

  • Unknown normalizing factors make posterior inference fundamentally harder because the likelihood cannot be used directly.
  • Many specialized tools exist (Laplace, grid interpolation, importance sampling, bridge sampling, path sampling).
  • The best method depends on the dimensionality and structure of the model.