1. Overview
In most Bayesian models, the posterior density $p(θ \mid y)$ has no closed-form expression because the normalizing constant $p(y)$ involves an intractable integral.
Therefore, much of formal Bayesian analysis focuses on cases where closed-form solutions are available.
Although these models are sometimes oversimplified, they provide useful analytical foundations and often serve as building blocks for more realistic hierarchical or mixture models.
The most common single-parameter models are:
- Binomial (for counts of successes)
- Normal (for continuous additive variables)
- Poisson (for counts over time or space)
- Exponential (for waiting times between events)
Each of these standard distributions has a natural conjugate prior family, allowing analytical derivations of the posterior distribution.
2. Normal Distribution with Known Mean but Unknown Variance
This model is less about direct application and more about providing a foundation for modeling scale parameters, especially as a building block for the normal model with both unknown mean and variance.
Let the data be $y_1, \dots, y_n$, where:
$y_i \mid θ, σ^2 \sim N(θ, σ^2)$
and here the mean θ is known, but the variance σ² is unknown.
Likelihood
$p(y \mid σ^2) \propto (σ^2)^{-n/2} \exp\left[-\frac{1}{2σ^2}\sum_{i=1}^n (y_i – θ)^2\right]$
Define the sufficient statistic:
$v = \frac{1}{n}\sum_{i=1}^n (y_i – θ)^2$
Conjugate Prior
The conjugate prior for $σ^2$ is an inverse-gamma (or equivalently, a scaled inverse-χ²) distribution:
$σ^2 \sim \text{Inv-}\chi^2(ν_0, σ_0^2)$
with prior degrees of freedom $ν_0$ and scale $σ_0^2$.
Posterior
Combining the likelihood and prior gives:
$σ^2 \mid y \sim \text{Inv-}\chi^2\!\left(ν_0 + n,\; \frac{ν_0 σ_0^2 + n v}{ν_0 + n}\right)$
This posterior can be interpreted as if the prior contributes information equivalent to $ν_0$ prior observations with average squared deviation $σ_0^2$.
3. Poisson Model
The Poisson model naturally describes count data—for example, the number of disease cases, defects, or arrivals in a fixed time period.
$p(y \mid θ) = \frac{θ^y e^{-θ}}{y!}, \quad y = 0, 1, 2, \dots$
For $n$ i.i.d. observations $y_1, \dots, y_n$:
$p(y \mid θ) \propto θ^{\sum_{i=1}^n y_i} e^{-nθ}$
with the sufficient statistic $t(y) = \sum_{i=1}^n y_i$.
Conjugate Prior
Since the likelihood has the form $θ^{a} e^{-bθ}$, the conjugate prior is a Gamma distribution:
$θ \sim \text{Gamma}(\alpha, \beta), \quad p(θ) \propto θ^{\alpha-1} e^{-\beta θ}$
Posterior
$θ \mid y \sim \text{Gamma}\left(\alpha + \sum_{i=1}^n y_i,\; \beta + n\right)$
Interpretation:
- $\alpha – 1$: prior number of “counts”
- $\beta$: prior total “exposure” or baseline period
4. Negative Binomial as a Poisson–Gamma Mixture
Combining a Poisson likelihood with a Gamma prior gives a negative binomial marginal (prior predictive) distribution.
$p(y) = \int \text{Poisson}(y \mid θ) \, \text{Gamma}(θ \mid \alpha, \beta) \, dθ$
This mixture interpretation shows that the negative binomial arises from Poisson data with rate θ that varies according to a Gamma distribution:
$\text{NegBin}(y \mid \alpha, \beta) = \int \text{Poisson}(y \mid θ)\, \text{Gamma}(θ \mid \alpha, \beta)\, dθ$
Thus, the negative binomial distribution can be viewed as a robust alternative to the Poisson, capable of handling over-dispersion (variance > mean).
5. Poisson Model with Rate and Exposure
In practical applications, observation units often have different exposure levels $x_i$, such as varying observation times or population sizes.
The extended model is:
$y_i \mid θ \sim \text{Poisson}(x_i θ), \quad x_i > 0$
where:
- $θ$: rate per unit of exposure
- $x_i$: exposure (known)
The likelihood becomes:
$p(y \mid θ) \propto θ^{\sum y_i} \exp\left[-θ \sum x_i\right]$
and thus the Gamma prior remains conjugate.
Prior and Posterior
If
$θ \sim \text{Gamma}(\alpha, \beta)$
then
$θ \mid y \sim \text{Gamma}\left(\alpha + \sum_{i=1}^n y_i,\; \beta + \sum_{i=1}^n x_i\right)$
Example: Asthma Mortality Rate
- City population: 200,000
- Deaths in one year: $y = 3$
- Exposure: $x = 2.0$ (because rate is per 100,000 persons)
$y \mid θ \sim \text{Poisson}(2θ)$
Prior belief:
$θ \sim \text{Gamma}(3.0, 5.0)$
with mean $3/5 = 0.6$ and mode 0.4.
Posterior:
$θ \mid y = 3 \sim \text{Gamma}(6.0, 7.0)$
Posterior mean:
$E(θ \mid y) = \frac{6}{7} \approx 0.86$
The posterior mean shows shrinkage toward the prior mean of 0.6.
More Data Example
If 10 years of data are available, with $y = 30$ deaths and total exposure $x = 20$:
$θ \mid y = 30 \sim \text{Gamma}(33.0, 25.0)$
Posterior mean:
$E(θ \mid y) = \frac{33}{25} = 1.32$
The posterior is much narrower and now more influenced by the data, while still reflecting the prior.
6. Exponential Model
The exponential distribution is used for waiting times or survival times, where the variable is positive and continuous.
$p(y \mid θ) = θ e^{-θ y}, \quad y > 0$
Here, $θ$ is the rate parameter, and $1/θ = E(y \mid θ)$ is the mean waiting time.
Conjugate Prior
The conjugate prior for $θ$ is again Gamma(α, β).
$θ \sim \text{Gamma}(\alpha, \beta)$
For a single observation $y$:
$θ \mid y \sim \text{Gamma}(\alpha + 1, \beta + y)$
Multiple Observations
For nnn independent exponential samples $y_1, \dots, y_n$:
$p(y \mid θ) = θ^n e^{-θ \sum_{i=1}^n y_i}$
With the same Gamma prior:
$θ \mid y \sim \text{Gamma}\left(\alpha + n,\; \beta + \sum_{i=1}^n y_i\right)$
Interpretation:
- $\alpha – 1$: prior number of “events”
- $\beta$: prior total waiting time
7. Key Takeaways
| Model | Likelihood | Conjugate Prior | Posterior |
|---|---|---|---|
| Normal (σ² unknown) | $(σ^2)^{-n/2} e^{-nv/2σ^2}$ | Inv-$χ^2(ν_0, σ_0^2)$ | Inv-$χ^2(ν_0+n, (ν_0σ_0^2+nv)/(ν_0+n))$ |
| Poisson | $θ^{\sum y_i} e^{-nθ}$ | $\text{Gamma}(α, β)$ | $\text{Gamma}(α+\sum y_i, β+n)$ |
| Poisson w/ exposure | $θ^{\sum y_i} e^{-θ\sum x_i}$ | $\text{Gamma}(α, β)$ | $\text{Gamma}(α+\sum y_i, β+\sum x_i)$ |
| Exponential | $θ^n e^{-θ\sum y_i}$ | $\text{Gamma}(α, β)$ | $\text{Gamma}(α+n, β+\sum y_i)$ |
Summary Insight
- The conjugate families (Normal–Normal, Poisson–Gamma, Exponential–Gamma, Normal–Inv-χ²) make inference analytically simple.
- Each posterior combines prior information and data evidence by adding information (degrees of freedom or counts).
- Poisson–Gamma mixtures yield negative binomial distributions, offering a robust alternative to Poisson for overdispersed data.
- These foundational models demonstrate how simple conjugate structures form the basis for complex Bayesian hierarchical systems.