Marginal Likelihood (also called The Model Evidence or Integrated Likelihood)

1. Definition

• The marginal likelihood (also called the model evidence or integrated likelihood) is the probability of the observed data under a model, averaging over all possible parameter values.
• It is the denominator in Bayes’ theorem:

$P(\theta \mid D) = \frac{P(D \mid \theta) \cdot P(\theta)}{P(D)}$

where:

$P(D) = \int P(D \mid \theta) P(\theta) \, d\theta$

• $P(D)$ = marginal likelihood.
• It ensures that posterior probabilities normalize to 1.

In words: The marginal likelihood is the weighted average likelihood of the data, where the weights come from the prior.

---

2. Role in Bayes’ Theorem

• Prior: $P(\theta)$ = belief about parameters before data.
• Likelihood: $P(D \mid \theta)$ = probability of data given parameter values.
• Marginal likelihood: $P(D)$ = normalizing constant to make posterior a valid probability distribution.
• Posterior: $P(\theta \mid D)$ = updated belief after data.

---

3. Example – Coin Toss

Suppose we toss a coin 10 times and observe 7 heads.

• Prior for $p$ (probability of heads): Uniform = Beta(1,1).
• Likelihood:

$P(D \mid p) = {10 \choose 7} p^7 (1-p)^3$

• Marginal likelihood:

$P(D) = \int_0^1 {10 \choose 7} p^7 (1-p)^3 \cdot 1 \, dp$

• This integral gives the overall probability of observing “7 heads out of 10,” averaged across all possible ppp.

---

4. Interpretation

• Answers: “How likely is it to see this data under this model, considering all possible parameter values consistent with the prior?”
• Used to compare models (via Bayes Factors).

---

5. Applications

1. Posterior Normalization
   • Without $P(D)$, the posterior formula would not be a valid probability distribution.
2. Model Comparison (Bayes Factors)
   • Compare models $M_1, M_2$​ by their marginal likelihoods:
     • $BF = \frac{P(D \mid M_1)}{P(D \mid M_2)}$
     • Higher marginal likelihood → model better explains the data.
3. Bayesian Machine Learning
   • In model selection, hyperparameter tuning, and evidence approximation (Laplace, Variational Bayes, MCMC).

---

6. Key Takeaways

• Marginal likelihood = probability of data given the model, integrating out parameters.
• It’s the normalizing constant in Bayes’ theorem.
• Crucial for posterior calculation and model comparison (Bayes factors).

---

In short:
The marginal likelihood (a.k.a. model evidence) is the probability of the observed data under a model, obtained by averaging the likelihood across all possible parameter values with respect to the prior. It normalizes the posterior and is central in Bayesian model comparison.