1. Overview

This section introduces the binomial model as a simple yet foundational example for Bayesian inference, mirroring Thomas Bayes’s original 1763 analysis.
The goal is to estimate an unknown population proportion — that is, the probability of success (denoted by θ) — based on n Bernoulli trials $y_1, y_2, …, y_n$​, each taking a value of 0 or 1.

Each trial represents a “success” or “failure,” and because the trials are exchangeable (having no identifiable order or difference in context), the data can be summarized by the total number of successes $y = \sum_{i=1}^n y_i$​.
This makes the binomial distribution a natural model for such problems.

2. The Binomial Model

If θ represents the probability of success (or the proportion of successes in the population), then the likelihood function for observing y successes in n trials is:

$p(y|\theta) = \text{Bin}(y|n, \theta) = \binom{n}{y} \, \theta^y (1 – \theta)^{n – y}$

Here, n is treated as fixed — part of the experimental design — so the analysis is conditional on n.

3. Example: Estimating the Probability of a Female Birth

To illustrate, the text considers estimating the proportion of female births (θ) in a population.

  • Each birth is modeled as an independent Bernoulli trial with probability θ of being female.
  • The number of female births among n observed births is y.
  • Historically, data have shown that the proportion of female births is slightly below 0.5, typically around θ ≈ 0.485 in large European populations.

An alternative parameterization is sometimes used:

$\phi = \frac{1 – \theta}{\theta}$

which represents the male-to-female birth ratio.

This setup assumes conditional independence of births given θ — that is, once θ is fixed, each birth is independent of others. This assumption is justified by exchangeability, since no further explanatory variables (like family or multiple births) are used.

4. Prior and Posterior Distributions

Prior

A uniform prior distribution is assumed:

$p(\theta) = \text{Uniform}(0, 1)$

This reflects total prior ignorance about the proportion θ.

Posterior

Applying Bayes’ theorem gives:

$p(\theta|y) \propto p(y|\theta) \, p(\theta)$

Substituting the likelihood and prior:

$p(\theta|y) \propto \theta^y (1 – \theta)^{n – y}$

The combinatorial constant $\binom{n}{y}$ is independent of θ and can be ignored in proportional form.

This posterior distribution is recognized as a Beta distribution:

$\theta | y \sim \text{Beta}(y + 1, n – y + 1)$

Thus, the posterior mean, variance, and credible intervals can be derived directly from Beta distribution properties.

5. Interpretation

  • The posterior $\text{Beta}(y + 1, n – y + 1)$ represents updated beliefs about θ after observing y successes in n trials.
  • Larger n leads to a sharper (less uncertain) posterior, as the data provide stronger evidence.
  • The figure referenced (Figure 2.1) shows the unnormalized posterior density for θ for several (n, y) combinations, each with the same proportion y/n but different sample sizes — illustrating how increasing n tightens the posterior.

6. Historical Note: Bayes and Laplace

Early Developments

  • Early mathematicians such as Jacob Bernoulli focused on forward (pre-data) probabilities:
    Given θ, what is the probability distribution of y?
    Bernoulli’s weak law of large numbers established that as n → ∞, the sample proportion y/n converges to θ with high probability.

Thomas Bayes (1763)

Bayes was the first to invert the probability relationship, seeking:

$P(\theta \in [\theta_1, \theta_2] | y)$

He used a physical analogy of a billiard table:

  1. A “true” value θ is determined by a ball W landing uniformly along a table (the prior).
  2. Another ball O is thrown n times — y times it lands to the right of W (the data).

This leads to the posterior integral:

$P(\theta \in [\theta_1, \theta_2] | y) = \frac{ \int_{\theta_1}^{\theta_2} \binom{n}{y} \theta^y (1 – \theta)^{n – y} d\theta }{ p(y) }$

where

$p(y) = \int_0^1 \binom{n}{y} \theta^y (1 – \theta)^{n – y} d\theta$

and Bayes showed that

$p(y) = \frac{1}{n + 1}, \quad \text{for } y = 0, 1, \dots, n$

implying all outcomes of y are equally likely a priori.

However, computing the numerator required incomplete Beta integrals, which were difficult to evaluate by hand in Bayes’s time.

Pierre-Simon Laplace

Laplace independently rediscovered and extended Bayes’s theorem, introducing analytic methods like Taylor expansion and the normal approximation around $\theta = y/n$ to approximate the incomplete Beta integral.

He applied the method to estimate the female birth probability in Paris (1745–1770):

  • 241,945 girls
  • 251,527 boys
    Thus, $n = 493,472$ and $y = 241,945$.
    He computed:

$P(\theta \ge 0.5 | y, n) \approx 1.15 \times 10^{-42}$

and concluded he was “morally certain” that $\theta < 0.5$.

7. Prediction (Posterior Predictive Inference)

Bayesian analysis not only updates beliefs about θ but also predicts future outcomes.

Prior Predictive Distribution

From Bayes’s earlier result $p(y) = \frac{1}{n + 1}$​, all outcomes y = 0, 1, …, n are equally likely a priori under the uniform prior.

Posterior Predictive Distribution

Suppose we want the probability that the next birth (a new Bernoulli trial) is female, denoted $\tilde{y} = 1$.
Then:

$P(\tilde{y} = 1 | y) = \int_0^1 P(\tilde{y} = 1 | \theta, y) p(\theta | y) \, d\theta = \int_0^1 \theta p(\theta | y) \, d\theta = E(\theta | y)$

For a Beta(y + 1, n − y + 1) posterior, the mean is:

$E(\theta | y) = \frac{y + 1}{n + 2}$

This is known as Laplace’s Law of Succession.

8. Laplace’s Law of Succession

This rule provides the predictive probability of success in the next trial under a uniform prior:

$P(\tilde{y} = 1 | y) = \frac{y + 1}{n + 2}$

At the extremes:

  • If no successes yet (y = 0), then $P(\tilde{y} = 1 | y) = \frac{1}{n + 2}$​.
  • If all successes (y = n), then $P(\tilde{y} = 1 | y) = \frac{n + 1}{n + 2}$​.

Thus, even with extreme observations, the posterior predictive probability never collapses to 0 or 1 — it remains slightly uncertain due to the prior, ensuring a regularized prediction.

9. Key Takeaways

  • The binomial model forms the foundation of Bayesian inference for discrete data.
  • The uniform prior leads to a Beta posterior, combining prior beliefs and observed data coherently.
  • Laplace’s Law of Succession shows how Bayesian reasoning prevents overconfidence in small samples.
  • Historically, Bayes and Laplace established the principles of inference about parameters rather than data, introducing the posterior concept that underlies all modern Bayesian statistics.
  • This example demonstrates the essence of Bayesian thinking — updating uncertainty about θ and predicting future events coherently through probability.