Label switching and posterior computation

1. What is label switching, and why is it a problem?

In finite mixture models, label switching (or label ambiguity) is an identifiability issue that arises because:

• The likelihood does not care which label is assigned to which mixture component.
• You can permute the labels of the components and obtain exactly the same likelihood.

Example with a two-component mixture:

• Suppose we have two components with
  • $(\pi_1, \theta_1) = (0.2, 0)$,
  • $(\pi_2, \theta_2) = (0.8, -1)$.
• This is likelihood-equivalent to
  • $(\pi_1, \theta_1) = (0.8, -1)$,
  • $(\pi_2, \theta_2) = (0.2, 0)$.

So the model cannot distinguish “component 1” from “component 2” just from the data; only the set of components matters, not their labels.

More generally, if the EM algorithm converges to parameter estimates

• $(\hat{\pi}_1,\dots,\hat{\pi}_H)$,
• $(\hat{\theta}_1,\dots,\hat{\theta}_H)$,
• and possibly some global parameter $\hat{\phi}$,

then for any permutation $(\kappa_1,\dots,\kappa_H)$ of ${1,\dots,H}$, the permuted parameter set

• $(\hat{\pi}{\kappa_1},\dots,\hat{\pi}{\kappa_H})$,
• $(\hat{\theta}{\kappa_1},\dots,\hat{\theta}{\kappa_H})$,
• $\hat{\phi}$,

has the same likelihood. The likelihood surface has multiple symmetric modes corresponding to relabelings of the components.

---

2. Bayesian viewpoint: exchangeable priors and consequences

In a Bayesian mixture model, we specify a joint prior for:

• the mixture weights $(\pi_1,\dots,\pi_H)$,
• the component-specific parameters $(\theta_1,\dots,\theta_H)$,
• and possibly additional global parameters $\phi$.

A common and natural choice is an exchangeable prior over the components:

• $(\pi_1,\dots,\pi_H) \sim \text{Dirichlet}(a,\dots,a)$,
• $\theta_h \sim P_0$ independently for $h=1,\dots,H$,

where $P_0$ is a common prior distribution for component-specific parameters. (In your notation preference, you can view each $\theta_h$ as part of a larger parameter vector $\theta$; the key idea is that all components are treated symmetrically.)

If the prior is exchangeable over the component labels ${1,\dots,H}$, then:

• The posterior is also exchangeable in these labels.
• The marginal posterior distribution of $\theta_h$ is identical for all $h$.

This has an important implication:

You cannot meaningfully talk about “component $h$” as a distinct, labeled subpopulation unless you break this symmetry somehow (e.g., by constraints or postprocessing).

So if your scientific goal is to estimate “the posterior of component 1,” you must define what “component 1” means beyond the symmetric mixture model.

---

3. A concrete Gaussian mixture setup

To illustrate computation, the text focuses on a univariate Gaussian location–scale mixture:

• For each observation $y_i$, $$y_i \mid z_i \sim \text{Normal}(\mu_{z_i}, \tau_{z_i}^2), \quad \Pr(z_i = h) = \pi_h,\quad i = 1,\dots,n,$$is the latent component index.

The prior is as in the exchangeable setup above, with $P_0$ chosen to be conditionally conjugate:

• For each component $h = 1,\dots,H$, $$\mu_h \mid \tau_h^2 \sim \text{Normal}(\mu_0, \kappa \tau_h^2),$$ $$\tau_h^2 \sim \text{Inv-Gamma}(a_\tau, b_\tau).$$

So for each component $h$, the pair $(\mu_h,\tau_h^2)$ has a Normal–Inverse-Gamma prior.

There are no additional global parameters $\phi$ in this simplified case; everything is captured by $(\pi_h,\mu_h,\tau_h^2)$ and ${z_i}$.

---

4. Gibbs sampler with data augmentation (using $z_i$)

Given this conjugate structure, the posterior can be explored via a Gibbs sampler that alternates between:

1. Updating the latent component labels $z_i$.
2. Updating $(\mu_h,\tau_h^2)$ for each component $h$.
3. Updating the mixture weights $(\pi_1,\dots,\pi_H)$.

Step 1: Update the allocation variables $z_i$

Given current values of $(\pi_h,\mu_h,\tau_h^2)$, the conditional distribution of $z_i$ is multinomial over ${1,\dots,H}$ with probabilities proportional to:$$\Pr(z_i = h \mid -) = \frac{\pi_h \, N(y_i \mid \mu_h, \tau_h^2)} {\sum_{l=1}^H \pi_l \, N(y_i \mid \mu_l, \tau_l^2)}, \quad h = 1,\dots,H.$$

So for each $i$, you compute these probabilities and randomly assign $z_i$ accordingly.

Step 2: Update component-specific parameters $(\mu_h,\tau_h^2)$

Given the current allocations ${z_i}$ and mixture weights $(\pi_h)$, you compute for each component $h$:

• The number of observations in component $h$: $$n_h = \sum_{i=1}^n 1_{\{z_i = h\}},$$
• The sample mean within component $h$: $$\bar{y}_h = \frac{1}{n_h} \sum_{i:z_i = h} y_i.$$

Then the conjugate Normal–Inverse-Gamma posterior for $(\mu_h,\tau_h^2)$ is:

• Posterior precision factor: $$\kappa_h = \left(\kappa^{-1} + n_h\right)^{-1}.$$
• Posterior mean of $\mu_h$: $$\mu_h^{\text{post}} = \kappa_h\left(\kappa^{-1}\mu_0 + n_h \bar{y}_h\right).$$
• Posterior shape for $\tau_h^2$: $$\hat{a}_{\tau h} = a_\tau + \frac{n_h}{2}.$$
• Posterior scale for $\tau_h^2$: $$\hat{b}_{\tau h} = b_\tau + \frac{1}{2} \left[ \sum_{i:z_i = h} (y_i – \bar{y}_h)^2 + \frac{n_h}{1 + \kappa n_h} (\bar{y}_h – \mu_0)^2 \right].$$

Then:

• $\tau_h^2 \mid – \sim \text{Inv-Gamma}(\hat{a}{\tau h}, \hat{b}{\tau h})$,
• $\mu_h \mid \tau_h^2, – \sim \text{Normal}(\mu_h^{\text{post}}, \kappa_h \tau_h^2)$.

Step 3: Update mixture weights $(\pi_1,\dots,\pi_H)$

Given the current allocations ${z_i}$, the posterior for $(\pi_1,\dots,\pi_H)$ is again Dirichlet:$$(\pi_1,\dots,\pi_H) \mid – \sim \text{Dirichlet}(a + n_1,\dots,a + n_H).$$

This is a straightforward conjugate update.

---

5. Using the Gibbs output to estimate the density

Define the mixture density at a point $y$ as:$$g(y) = \sum_{h=1}^H \pi_h \, N(y \mid \mu_h, \tau_h^2).$$

From the Gibbs sampler, we get $S$ posterior draws:

• $(\pi_1^{(s)},\dots,\pi_H^{(s)})$,
• $(\mu_1^{(s)},\dots,\mu_H^{(s)})$,
• $(\tau_1^{2,(s)},\dots,\tau_H^{2,(s)})$,

for $s = 1,\dots,S$.

A natural Bayesian density estimate is the posterior mean of $g(y)$, approximated by:$$\hat{g}(y) \approx \frac{1}{S} \sum_{s=1}^S \sum_{h=1}^H \pi_h^{(s)} \, N\big(y \mid \mu_h^{(s)}, \tau_h^{2\,(s)}\big).$$

The key point is that this estimated density can have very good statistical properties, even if the component labels themselves show label switching.

---

6. Why label switching harms component-wise inference but not density estimation

Because the mixture components are exchangeable in the prior and posterior:

• The marginal posterior for each pair $(\mu_h,\tau_h)$ is identical across $h$.
• The posterior for each $(\mu_h,\tau_h)$ is typically multimodal, with modes corresponding to the different mixture components’ locations.

Consider a simple two-component example where:

• One component is centered at $\mu = 0$,
• The other is centered at $\mu = -1$,
• And $H = 2$ is fixed.

Then:

• The posterior distribution of $\mu_1$ has two modes: one near $0$ and one near $-1$.
• The same is true for $\mu_2$.

If the Gibbs sampler mixes perfectly, the chain for $\mu_1$ would frequently jump between values near $0$ and near $-1$, and similarly for $\mu_2$, because the sampler should explore both labelings:

• For some iterations, “component 1” is at $0$ and “component 2” at $-1$.
• For others, “component 1” is at $-1$ and “component 2” at $0$.

In practice, however:

• When the components are well-separated, there is a region of very low probability between the modes.
• The chain tends to get stuck in one labeling:
  • For example, for the first $5000$ iterations, $\mu_1$ may stay near $0$ and $\mu_2$ near $-1$.
  • Suddenly, the labels might switch, and then $\mu_1$ stays near $-1$ and $\mu_2$ near $0$.

From the perspective of component-wise parameters, this looks like very poor mixing: long stretches in one mode and maybe zero or very few switches.

However, if you focus on the induced density $g(y)$:

• $g(y)$ does not care which label is 1 or 2, only about the set of components.
• Even if labels never switch, the sampler can still explore the correct posterior for the density, because the density is invariant to label permutations.

So:

Label switching is a serious issue for inference about mixture component-specific parameters, but often not a problem for density estimation using $g(y)$.

---

7. Practical performance of Bayesian finite mixtures for density estimation

Based on extensive simulation and real data experience, the authors note:

• For sufficiently large $H$ and a specific prior choice like $a = 1/H$ in the Dirichlet prior,
• The Bayesian density estimate constructed from such a finite mixture often has excellent frequentist properties.
• In particular, it tends to have lower mean integrated squared error (MISE) than standard kernel density estimators.

Advantages:

• Because each component has its own variance $\tau_h^2$, the model induces a locally adaptive bandwidth:
  • In regions of the data that need more flexibility, smaller variances and more components can be used.
  • In smoother regions, fewer or broader components suffice.
• The Bayesian framework automatically incorporates a penalty for model complexity:
  • As items are spread across more components, the marginal likelihood tends to decrease because more parameters must be integrated over.
  • This implicit penalty helps control overfitting without cross-validation.

Thus, a finite mixture with a reasonable prior can be seen as a powerful, adaptive density estimator.

---

8. Choosing the base prior $P_0$ carefully

A crucial practical point is the choice of the base prior $P_0$ for component-specific parameters:

• You should avoid:
  • Improper priors,
  • Very diffuse (extremely high-variance) proper priors.

Such choices can make the results sensitive and unstable, since the prior places mass on unrealistic regions.

Instead:

• Choose $P_0$ so that typical mixture components lie near the support of the data.

Two strategies are highlighted:

1. Normalize the data first:
   • For example, transform the data to have mean $0$ and unit scale.
   • Then choose default hyperparameters like:
     • $\mu_0 = 0$,
     • $\kappa = 1$,
     • $a_\tau = 2$,
     • $b_\tau = 4$.
   • After sampling the density on the normalized scale, apply the inverse transformation to map back to the original scale.
2. Direct elicitation from prior knowledge:
   • Use prior information about plausible location and scale of the data to set $(\mu_0,\kappa,a_\tau,b_\tau)$ directly.

Good performance relies on $P_0$ being informative enough to restrict components to plausible regions, without being too restrictive.

---

9. Making inferences on component-specific parameters despite label switching

Suppose you really want posterior inferences about individual components, e.g. $(\mu_h,\tau_h)$ for “component $h$”.

Because of label switching:

• Directly averaging over MCMC draws of $(\mu_h,\tau_h)$ does not make sense.
• For example, in the two-component example, with good mixing you would obtain the same posterior mean for $\mu_1$ and $\mu_2$, which doesn’t reflect the fact that one component is around $0$ and the other around $-1$.

To address this, you can use postprocessing relabeling methods:

1. Run the MCMC sampler, ignoring label switching.
2. Collect $S$ posterior draws.
3. For each draw $s$, consider a permutation $\sigma^{(s)}$ of ${1,\dots,H}$ that reassigns labels.
4. Choose permutations $\sigma^{(1)},\dots,\sigma^{(S)}$ to minimize a loss function $L(a,\theta)$, where:
   • $a$ is some action summarizing the parameters (e.g. cluster centers),
   • $\theta$ denotes the full parameter collection.
5. Iterate between:
   • Choosing $a$ given current labelings to minimize expected loss.
   • Choosing each $\sigma^{(s)}$ to minimize the loss given $a$.

After relabeling, you can interpret:

• “component 1” as consistently the component near $\mu = 0$,
• “component 2” as the one near $\mu = -1$,

and then compute standard posterior summaries for each component.

---

10. Alternative approach: identifiability constraints in the prior (and their problems)

Instead of using exchangeable priors plus postprocessing, you might try to break the symmetry directly in the prior. For example, in the univariate Gaussian mixture you could impose:

• An ordering constraint such as $\mu_1 < \mu_2 < \dots < \mu_H$.

This would, in theory, assign a unique label to each component (e.g. “the component with smallest mean,” “second smallest mean,” etc.).

However, this approach has serious drawbacks:

1. Multivariate extension is unclear:
   • For multivariate data, there is no obvious, generally appropriate way to impose such ordering constraints.
   • Which dimension should you order on? What if components overlap in some directions?
2. Even in univariate mixtures, ordering may not solve the real problem:
   • The model may need multiple components with similar means but different variances to fit the data.
   • If the means are close, label switching can effectively still occur, even with $\mu_1 < \mu_2 < \dots < \mu_H$, because components may almost overlap.
3. Bias and poor mixing:
   • Constraints like $\mu_1 < \mu_2 < \dots < \mu_H$ can:
     • Make the sampler mix poorly.
     • Induce a bias that pushes the means apart artificially due to the ordering prior.
   • This can cause overestimation of differences between components when the posterior uncertainty is large.

Because of these issues, strict identifiability constraints in the prior are often not recommended in complex, realistic applications.

---

11. Summary

• Label switching is a fundamental property of mixture models with exchangeable priors: the labels of components are not identifiable from the data alone.
• This causes:
  • Serious issues for inference on component-specific parameters $(\mu_h,\tau_h)$,
  • But usually not a problem for functions of the mixture that are invariant to relabeling, such as the density $g(y)$.
• A finite Gaussian mixture with a well-chosen prior and sufficiently large $H$ yields:
  • Highly competitive density estimates with excellent frequentist properties.
  • Automatic, locally adaptive smoothing via component-specific variances.
• For component-wise inference, you should:
  • Either use postprocessing relabeling methods to define consistent labels,
  • Or be very careful with any identifiability constraints in the prior, recognizing their limitations and potential biases.