1. The Concept of Nuisance Parameters
In many Bayesian models, the parameter vector $θ$ (originally denoted as ωωω) can be divided into two parts:
$θ = (θ_1, θ_2)$
where:
- $θ_1$: parameter(s) of interest,
- $θ_2$: parameter(s) considered nuisance parameters (not of direct interest but necessary for modeling).
For example, in a normal model:
$y \mid μ, σ^2 \sim N(μ, σ^2)$
we might care primarily about the mean $μ$ (i.e., $θ_1 = μ$), while the variance $σ^2$ (i.e., $θ_2 = σ^2$) is a nuisance parameter.
2. The Joint and Marginal Posterior
The joint posterior for all parameters is given by Bayes’ theorem:
$p(θ_1, θ_2 \mid y) \propto p(y \mid θ_1, θ_2) \, p(θ_1, θ_2)$
However, if our goal is only to infer $θ_1$, we can average out the nuisance parameter $θ_2$ to obtain the marginal posterior:
$p(θ_1 \mid y) = \int p(θ_1, θ_2 \mid y) \, dθ_2.$. (3.0)
This integration process effectively removes $θ_2$ from the analysis by marginalization.
3. The Mixture Interpretation
We can rewrite the marginal posterior using the factorization identity:
$p(θ_1, θ_2 \mid y) = p(θ_1 \mid θ_2, y) \, p(θ_2 \mid y)$
Substituting into equation (3.0), we get:
$p(θ_1 \mid y) = \int p(θ_1 \mid θ_2, y) \, p(θ_2 \mid y) \, dθ_2.$ (3.1)
This reveals that $p(θ_1 \mid y)$ is a mixture distribution:
- Each component is the conditional posterior $p(θ_1 \mid θ_2, y)$,
- Weighted by $p(θ_2 \mid y)$, the posterior distribution of the nuisance parameter.
So, our posterior for the parameter of interest is literally a weighted average of possible conditional posteriors, averaged over plausible values of the nuisance parameter.
4. Interpretation of the Weights
The weight function $p(θ_2 \mid y)$ represents how much support the data and prior together assign to each possible value of $θ_2$.
That means the “importance” of each conditional posterior $p(θ_1 \mid θ_2, y)$ is automatically determined by the posterior uncertainty in $θ_2$.
This concept generalizes beyond continuous parameters — $θ_2$ could even represent discrete choices such as different submodels, in which case the integral becomes a weighted sum.
5. Practical Computation: Simulation Approach
In real Bayesian computation, we rarely evaluate the integral (3.1) analytically.
Instead, we use simulation-based methods such as Gibbs sampling or MCMC.
The idea is simple:
- Draw samples of the nuisance parameter $θ_2^{(s)}$ from its posterior distribution $p(θ_2 \mid y)$.
- For each draw $θ_2^{(s)}$, draw $θ_1^{(s)}$ from $p(θ_1 \mid θ_2^{(s)}, y)$.
- The set of pairs $(θ_1^{(s)}, θ_2^{(s)})$ forms samples from the joint posterior, and the $θ_1^{(s)}$ alone form samples from the marginal posterior $p(θ_1 \mid y)$.
This way, integration is replaced by Monte Carlo averaging:
$p(θ_1 \mid y) \approx \frac{1}{S} \sum_{s=1}^{S} p(θ_1 \mid θ_2^{(s)}, y)$
where the averaging is done implicitly through the random draws.
6. Canonical Example
A classic example is again the normal model with unknown mean and variance:
$y_i \mid μ, σ^2 \sim N(μ, σ^2)$
where $μ$ is of interest and $σ^2$ is the nuisance parameter.
In this case, equation (3.1) becomes:
$p(μ \mid y) = \int p(μ \mid σ^2, y) \, p(σ^2 \mid y) \, dσ^2$
which means:
- $p(μ \mid σ^2, y)$ is normal,
- $p(σ^2 \mid y)$ is scaled inverse-$χ^2$,
so the marginal posterior for $μ$ is a Student-$t$ distribution.
7. Summary Insight
| Concept | Description |
|---|---|
| Parameter of interest | $θ_1$ (e.g., the mean $μ$) |
| Nuisance parameter | $θ_2$ (e.g., variance $σ^2$) |
| Marginalization | $p(θ_1 \mid y) = \int p(θ_1, θ_2 \mid y) dθ_2$ |
| Mixture interpretation | $p(θ_1 \mid y) = \int p(θ_1 \mid θ_2, y) p(θ_2 \mid y) dθ_2$ |
| Computation | Simulate $θ_2$ from $p(θ_2 \mid y)$, then $θ_1$ from $p(θ_1 \mid θ_2, y)$ |
| Example | Normal model → $μ$ interest, $σ^2$ nuisance → marginal $p(μ \mid y)$ is $t$-distribution |
8. Intuitive Summary
- Marginalization removes uninteresting parameters by averaging over their uncertainty.
- The posterior for parameters of interest becomes a mixture of conditional posteriors weighted by how plausible each nuisance value is.
- Computation-wise, we simulate rather than integrate.
- This principle underlies most modern Bayesian computation, especially Gibbs sampling, MCMC, and hierarchical model inference.
In short:
To get $p(θ_1 \mid y)$, average over everything you don’t care about — and simulation does this automatically.