1. Overview
The posterior probability distribution $p(\theta | y)$ contains all available information about the unknown parameter $\theta$ after observing the data $y$.
In principle, one could report the entire posterior distribution—for instance, through a graphical display such as a histogram or density plot .
For multivariate posteriors, contour plots and scatterplots help visualize parameter relationships.
A key advantage of the Bayesian framework, particularly when combined with simulation-based methods, is its flexibility:
posterior results can be summarized and transformed in any way, even for highly complex models.
2. Numerical Summaries of the Posterior Distribution
Although the full posterior distribution provides complete information, in practice, we often summarize it numerically.
Common Summaries of Location (Center)
- Mean — the posterior expectation $E(\theta | y)$, representing the average value of $\theta$ given the data.
- Median — the value dividing the posterior probability into two equal halves.
- Mode(s) — the most probable value(s) of $\theta$, i.e., where $p(\theta | y)$ is maximized.
Each of these measures conveys different aspects:
- The mean is useful for expected values and decision analysis.
- The mode (also called the maximum a posteriori or MAP estimate) represents the most likely value given data and prior.
- The median is robust against asymmetry or skewness in the distribution.
Summaries of Dispersion (Spread)
To represent uncertainty or variability, we often report:
- Standard deviation — measures the typical deviation from the posterior mean.
- Interquartile range (IQR) — the range between the 25th and 75th percentiles.
- Other quantiles — particularly the 2.5th and 97.5th percentiles for a 95% credible interval.
In Bayesian analysis, such summaries directly describe the posterior uncertainty about the parameter, not the sampling variability under repeated experiments.
3. Closed-Form Posterior Summaries for Simple Models
When the posterior distribution has a closed form (e.g., a Beta posterior in the binomial model), summary statistics can often be obtained analytically.
For the Beta posterior derived earlier:
$\theta | y \sim \text{Beta}(y + 1, n – y + 1)$
From standard properties of the Beta distribution:
- Posterior mean: $E(\theta | y) = \frac{y + 1}{n + 2}$
- Posterior mode: $\text{Mode}(\theta | y) = \frac{y}{n}$ (This equals the maximum likelihood estimate (MLE) and the minimum variance unbiased estimator (MVUE) for $\theta$.)
Hence, the Bayesian posterior incorporates both data evidence (y/n) and prior information, while still maintaining interpretability through familiar estimators.
4. Posterior Quantiles and Intervals
Bayesian inference places strong emphasis on quantifying uncertainty through intervals or regions derived from the posterior.
(a) Posterior Intervals
A posterior interval (or credible interval) is the range containing a specified proportion of the posterior probability.
For example, a 100(1 − α)% posterior interval contains all parameter values with cumulative probability between $\alpha/2$ and $1 – \alpha/2$.
That is:
$P(\theta_L < \theta < \theta_U | y) = 1 – \alpha$
For example, a 95% posterior interval corresponds to the range between the 2.5th and 97.5th percentiles of $p(\theta | y)$.
These intervals can be:
- Computed analytically (from cumulative distribution functions for simple models like Beta or Normal), or
- Estimated by simulation, by generating random draws $\theta^{(1)}, \dots, \theta^{(S)}$ from the posterior and taking the empirical quantiles.
(b) Highest Posterior Density (HPD) Region
An HPD region is another way to summarize posterior uncertainty.
It is defined as the set of parameter values with the highest posterior density that together contain a given probability mass $1 – \alpha$.
Formally:
$R_{\text{HPD}} = \{ \theta : p(\theta | y) \ge c \}, \quad \text{where } \int_{R_{\text{HPD}}} p(\theta | y) \, d\theta = 1 – \alpha$
Properties:
- Every point inside the HPD region has higher posterior density than any point outside it.
- For unimodal and symmetric posteriors (like the normal distribution), the HPD region and central posterior interval coincide.
- For skewed or multimodal posteriors, they can differ substantially.
5. Comparison: Central Interval vs HPD Region
- Central Posterior Interval:
Includes the middle probability mass (e.g., between the 2.5th and 97.5th percentiles).
It’s easy to compute (especially using simulation) and widely used in practice. - Highest Posterior Density (HPD) Region:
Contains only the most probable values and excludes low-probability gaps.
It is more informative for asymmetric or bimodal distributions, but harder to compute.
Example:
For a bimodal posterior, the 95% central interval might include a central region where the probability density is actually zero, while the 95% HPD region consists of two disjoint intervals — each surrounding a mode.
In such cases, the HPD region better reflects the true uncertainty structure, but summarizing with a single interval may be misleading.
6. Practical Recommendations
- Use plots (histograms, density curves, contour plots) to visualize the posterior distribution.
- Report posterior means, medians, or modes as point estimates, depending on the context.
- Provide posterior intervals or HPD regions to convey uncertainty.
- When posteriors are asymmetric or multimodal, graphical summaries are often more informative than a single numerical interval.
7. Summary Table
| Concept | Definition / Expression | Interpretation |
|---|---|---|
| Posterior Mean | $E(\theta \mid y)$ | Expected value of the parameter given the data (average belief). |
| Posterior Mode | $\text{argmax}_{\theta} \, p(\theta \mid y)$ | Most likely parameter value (MAP estimate). |
| Posterior Median | $P(\theta < \tilde{\theta} \mid y) = 0.5$ | The middle value that divides the posterior probability equally. |
| Posterior Interval (Credible Interval) | Between posterior quantiles $\alpha/2$ and $1 – \alpha/2$ | Contains a specified fraction (e.g., 95%) of the posterior probability. |
| HPD Region | Smallest region $R$ such that $\int_R p(\theta \mid y) \, d\theta = 1 – \alpha$ | Contains the most probable parameter values (highest posterior density). |
| Symmetry Relation | Central interval = HPD if posterior is symmetric | When the posterior is skewed or multimodal, HPD provides a clearer representation. |
8. Key Insight
Bayesian inference provides full probabilistic information about parameters through the posterior distribution.
Summarizing it involves selecting the most meaningful combination of:
- Point estimates (mean, mode, median) for central tendency,
- Interval or density regions (credible intervals, HPD) for uncertainty, and
- Visualizations for interpretation, especially when distributions are complex or non-symmetric.
This flexibility—both in computation and in interpretation—is one of the defining strengths of the Bayesian approach.