Bayes factors compare models by giving each model a prior probability and then updating those probabilities using how well each model predicts the observed data. The updating piece is the marginal likelihood (also called the integrated likelihood): the likelihood averaged over the model’s prior for its parameters. The ratio of these marginal likelihoods is the Bayes factor. Multiply a Bayes factor by the prior odds for the two models, and the result is the posterior odds for the two models.

Formally, for two models $H_1$​ and $H_2$​,

$\frac{p(H_2 \mid y)}{p(H_1 \mid y)} \;=\; \frac{p(H_2)}{p(H_1)} \times \frac{p(y \mid H_2)}{p(y \mid H_1)},$

and the second ratio,

$\frac{p(y \mid H_2)}{p(y \mid H_1)},$

is the Bayes factor. Each marginal likelihood has the form

$p(y \mid H_i) = \int p(\theta_i \mid H_i)\, p(y \mid \theta_i, H_i)\, d\theta_i,$

so for Bayes factors to be defined, those integrals must be proper (finite).

The material makes two main points:

  1. there are situations where this is a clean and useful way to think;
  2. in a lot of applied modeling, it ends up being fragile and not worth the trouble.

1. When Bayes factors work well

Bayes factors are at their best when the competing models are genuinely discrete and scientifically distinct, and when each model has a proper prior.

A genetic example shows this clearly. Consider two hypotheses:

  • $H_1$​: a woman is a carrier/affected
  • $H_2$​: a woman is not a carrier/not affected

These are natural alternatives; nothing in between is needed. Suppose two unaffected sons are observed. Under the “unaffected” hypothesis that outcome has probability 1; under the “affected” hypothesis it has probability 0.25. So the Bayes factor in favor of “unaffected” over “affected” is $1/0.25 = 4$. With equal prior odds, the posterior odds become 4:1. The calculation is transparent and meaningful because:

  • both models make sense as-is,
  • both induce proper marginal likelihoods,
  • and the question itself is discrete.

In that kind of problem, representing learning as a product of prior odds × data-supported odds is natural.

2. When Bayes factors become a distraction

Now contrast that with a problem like the 8-schools example. Two simple models can be written down:

  • No pooling: each school has its own effect $\theta_j$, all priors flat.
  • Complete pooling: all school effects are equal, $\theta_1 = \cdots = \theta_J = \theta$, prior flat.

Those two can be treated as two “models” $H_1$​ and $H_2$​. But a Bayes factor needs proper priors. The usual flat priors in that example are improper, so the marginal likelihoods are of the form 0/0 and the Bayes factor is undefined. To proceed, proper priors must be introduced, say

$\theta_j \sim N(0, A^2)$

with a large $A$. Once that is done, the marginal likelihoods can be computed and a Bayes factor can be formed.

But now a problem appears: the Bayes factor becomes highly sensitive to the arbitrary scale $A$. As $A \to \infty$, the Bayes factor moves toward favoring complete pooling. That happens even though the data didn’t change; the only thing that changed was how wide the prior was made. That kind of sensitivity to an arbitrary scale is not acceptable in most applied settings.

A second problem shows up if the same setup is applied not to 8 schools but to, say, 80 similar schools. With more groups, the same prior choice can push the Bayes factor even more strongly toward one of the discrete models, even though, scientifically, one still wouldn’t want to assert “exactly no variation” or “infinite variation.” So the result becomes sensitive to details of the problem (number of groups, exact prior scale) that are not scientifically central.

This shows why treating “no pooling” and “complete pooling” as competing discrete models is awkward. Those two are really endpoints of a sensible continuous family:

$y_j \sim N(\theta_j, \sigma_j^2), \quad \theta_j \sim N(\mu, \tau^2)$

with $\mu$ flat and $\tau \in [0, \infty)$

  • $\tau = 0$ → complete pooling
  • $\tau \to \infty$ → no pooling
  • intermediate $\tau$ → partial pooling

Once that continuous model is in place, assigning positive probability to exactly $\tau = 0$ or exactly $\tau = \infty$ is not especially meaningful. The continuous model already expresses the scientific story better. In that setting, Bayes factors between the two extremes are solving the wrong problem.

3. Core criticisms from the discussion

  1. Sensitivity to the prior for parameters
    Bayes factors integrate the likelihood over the entire prior. Diffuse or arbitrarily scaled priors lead to arbitrarily small or large marginal likelihoods. That means the comparison of models can be driven more by the tail behavior of chosen priors than by the data.
  2. Improper priors break marginal likelihoods
    A lot of applied Bayesian work uses flat or weak priors. These are fine for posterior inference but not fine for computing marginal likelihoods. As soon as Bayes factors are introduced, priors must be made proper, and the result can depend strongly on how that was done.
  3. Discrete comparison can be the wrong framing
    In many applied examples, models that at first look “competing” are better understood as special cases of a richer, continuous model. In such situations, the question is not “which of these two is true?” but “where in this continuous space does the data push us?” A Bayes-factor-based choice between two extremes can mislead.
  4. Dimension dependence
    In the 8-schools illustration, posterior inferences that depend heavily on how many schools happen to be in the dataset are considered scientifically unappealing. That dependence comes from forcing a discrete comparison with priors that were not set up to be compared through marginal likelihoods.

4. Bottom-line guidance from the discussion

  • Bayes factors are fine when the problem itself is naturally discrete and when proper priors arise naturally (genetic carrier status, competing mechanistic models, mutually exclusive data-generating stories).
  • For typical applied hierarchical or regression modeling, it is better to expand to a continuous model and then evaluate or compare models using predictive criteria (LOO, WAIC) or posterior predictive checks, instead of trying to force a discrete choice with Bayes factors.
  • The main practical reason for not recommending Bayes factors in those settings is not philosophical but stability: marginal likelihoods are too sensitive to aspects of the model that are otherwise benign and untestable from the data.