1. Direct simulation from the posterior
In simple Bayesian models, especially those with conjugate priors, we can draw samples directly from the posterior $p(\theta | y)$.
- Example: In a normal–normal or beta–binomial model, the posterior has a known closed form, so drawing samples is straightforward.
When models get more complex:
- We may factorize the posterior into simpler conditional parts, e.g.:
- Sample hyperparameters from their marginal posterior.
- Then sample lower-level parameters given those hyperparameters and the data.
This “factorized simulation” approach was used in hierarchical examples from earlier chapters (like the rat-tumor model).
If some parts of the model allow for analytic integration, it’s efficient to integrate those parts out and simulate the rest.
2. Direct approximation using a grid
For very low-dimensional problems (say, 1 or 2 parameters):
Grid-based approach:
- Choose evenly spaced points $\theta_1, \ldots, \theta_N$ covering the range of plausible parameter values.
- Compute the (possibly unnormalized) posterior density $q(\theta_i | y)$ at each grid point.
- Normalize to get a discrete approximation: .
- $P_i = \frac{q(\theta_i | y)}{\sum_{j=1}^N q(\theta_j | y)}$
- To draw a random sample:
- Draw a uniform random number $U \sim \text{Uniform}(0,1)$,
- Use the inverse CDF method to map $U$ to one of the grid points.
Works well when:
- Posterior is unimodal and one-dimensional.
- Grid is fine enough and covers all important regions.
Fails when:
- The parameter space is high-dimensional (computationally infeasible to evaluate on dense grids).
3. Simulating from posterior predictive distributions
Once you have posterior draws $\theta^{(s)}$ from $p(\theta | y)$, you can simulate future or unobserved data easily:
- For each posterior sample $\theta^{(s)}$, draw a new value:
- $\tilde{y}^{(s)} \sim p(\tilde{y} | \theta^{(s)})$
- The collection of $\tilde{y}^{(s)}$ values represents samples from the posterior predictive distribution:
- $p(\tilde{y} | y) = \int p(\tilde{y} | \theta) p(\theta | y) \, d\theta$
This is a fundamental step for model checking and prediction, as discussed.
4. Rejection Sampling
Goal:
Draw samples from a target density $p(\theta | y)$ (or an unnormalized version $q(\theta | y)$.
Requirements:
We need a simpler proposal function $g(\theta)$ satisfying:
- We can easily draw samples from a distribution proportional to $g(\theta)$.
- There exists a finite constant $M$ such that:
- $p(\theta | y) \le M \, g(\theta) \quad \text{for all } \theta$
The algorithm
- Draw $\theta^\*$ from the proposal distribution proportional to $g(\theta)$.
- Accept $\omega^\*$ with probability:
- $\text{Accept } \theta^\* \text{ with prob } \frac{p(\theta^\* | y)}{M \, g(\theta^\*)}$ Otherwise, reject and go back to step 1.
- Repeat until you have enough accepted samples.
Why it works
- The accepted draws come from the correct posterior $p(\theta | y)$.
- The boundedness condition ensures the acceptance probability is ≤ 1.
Efficiency considerations
- Ideal case: $g(\theta) \propto p(\theta | y)$
→ Every sample is accepted (100% efficiency). - Poor case: $g(\theta)$ doesn’t match $p(\theta | y)$ well
→ $M$ must be large, and most samples get rejected → inefficiency.
The acceptance rate itself tells you how well $g$ approximates $p$ —
so rejection sampling is self-monitoring.
Practical uses
- Works best for univariate or low-dimensional problems.
- Common in:
- Sampling from standard distributions (fast methods for univariate cases).
- Sampling from truncated distributions, if the truncated region is not too large.
Visualization
- Top curve: $M g(\theta)$ — the bounding envelope.
- Bottom curve: $p(\theta | y)$ — target distribution.
- Randomly sample under the top curve.
- Accept points that fall under the bottom curve.
If the two curves are close → high acceptance rate.
If not → most samples get rejected.
Key Takeaways
| Method | Idea | When useful | Pros | Cons |
|---|---|---|---|---|
| Direct simulation | Sample directly from posterior if available | Simple conjugate models | Exact, fast | Only works for simple forms |
| Grid approximation | Evaluate posterior on grid, normalize | 1D or 2D problems | Simple conceptually | Impossible for high dimensions |
| Predictive simulation | Draw future data given posterior samples | Any model | Enables model checking & prediction | Requires posterior draws first |
| Rejection sampling | Accept/reject based on bounding density | Small, well-behaved problems | Always correct | Very inefficient in high dimensions |
In summary:
Direct simulation and rejection sampling are foundational Bayesian computation techniques—conceptually simple and exact—but in practice, they are limited to low-dimensional problems. For complex hierarchical or high-dimensional models, more sophisticated algorithms (like importance sampling and MCMC) are needed.