This section reviews key probability concepts and notational conventions used throughout Bayesian analysis. The focus is on the manipulation of joint, conditional, and marginal distributions; expectations and variances (including conditional forms); and transformations of variables. These are essential tools for understanding and performing Bayesian inference.
1. Fundamental Notation and Concepts
We denote joint probability distributions using $p(u, v)$, where $u$ and $v$ may each represent scalar or vector quantities.
For example:
- Joint density: $p(u, v)$
- Conditional density: $p(u|v)$
- Marginal density: $p(u) = \int p(u, v)\,dv$
It is often useful to factor joint densities:
$p(u, v, w) = p(u|v, w) \, p(v|w) \, p(w)$
This decomposition expresses complex joint structures in terms of simpler conditional relationships.
Some authors use distinct notations for parameters and data (e.g., $\pi(\theta)$, $f(y|\theta)$), but in Bayesian inference all probability distributions have equal logical status — they all represent uncertainty.
However, conditioning must always be explicit:
$p(y|\theta) \neq p(y)$
When necessary, one can make assumptions explicit:
$p(\theta, y | H) = p(\theta | H)\, p(y | \theta, H)$
where $H$ denotes the set of model assumptions or hypotheses.
In practice, we often suppress this conditioning (and also conditioning on known covariates $x$) to simplify notation.
2. Expectation and Variance
We use $E(\cdot)$ and $var(\cdot)$ to denote mean and variance:
$E(u) = \int u\, p(u)\,du, \qquad var(u) = \int (u – E(u))^2 p(u)\,du$
For a vector $u$:
$var(u) = \int (u – E(u))(u – E(u))^T p(u)\,du$
This defines the covariance matrix (also called the variance matrix).
When expectations are conditional:
- $E(u|v)$: expectation of $u$ given $v$
- $E(u)$: marginal expectation, averaging over both $u$ and $v$
Thus, any variable not explicitly listed as “conditioned on” is integrated out.
3. Conditional and Hierarchical Modeling
Complex probability models are often built conditionally or hierarchically rather than as single unconditional distributions.
Example – Student Heights:
- Suppose $y$ = height of a randomly chosen university student.
- The marginal distribution $p(y)$ is roughly a mixture of two normal distributions (one for males, one for females).
- We can model this more usefully by specifying:
- $p(\text{male}) \approx p(\text{female}) \approx \frac{1}{2},$, and:
- $p(y|\text{female}) \sim N(160, \sigma_f^2), \quad p(y|\text{male}) \sim N(175, \sigma_m^2)$
- If the conditional variances are small, the marginal distribution $p(y)$ will appear bimodal.
This demonstrates that it is often simpler and more interpretable to model relationships through conditional structure, introducing additional (possibly unobserved) variables.
This idea is foundational to hierarchical and mixture models, discussed in depth in later chapters.
4. Means and Variances of Conditional Distributions
Two key results relate unconditional and conditional expectations and variances.
(a) Law of Total Expectation
$E(u) = E(E(u|v)) \tag{1.8}$
That is, the overall mean equals the expectation of the conditional mean.
It can be derived from the joint distribution $p(u, v)$:
$E(u) = \int u\, p(u, v)\,du\,dv = \int E(u|v)\,p(v)\,dv$
(b) Law of Total Variance
$var(u) = E(var(u|v)) + var(E(u|v)) \tag{1.9}$
The total variance of uuu equals:
- The expected value of the conditional variance (within-group variability), plus
- The variance of the conditional mean (between-group variability).
Derivation:
$E(var(u|v)) + var(E(u|v)) = E(E(u^2|v) – (E(u|v))^2) + E((E(u|v))^2) – (E(E(u|v)))^2 = E(u^2) – (E(u))^2 = var(u)$
These identities also hold for vector-valued uuu, in which case $E(u)$ is a vector and $var(u)$ a covariance matrix.
5. Transformation of Variables
When transforming a random variable, we adjust its probability density accordingly.
(a) Discrete case
If $u$ is discrete and $v = f(u)$ is one-to-one:
$p_v(v) = p_u(f^{-1}(v))$
If $f$ is many-to-one, sum over all possible inverse branches:
$p_v(v) = \sum_{u: f(u)=v} p_u(u)$
(b) Continuous case
If $u$ is continuous and $v = f(u)$ is one-to-one:
$p_v(v) = |J| \, p_u(f^{-1}(v))$
where $J$ is the Jacobian matrix of the inverse transformation $u = f^{-1}(v)$:
$J = \frac{\partial u_i}{\partial v_j}, \quad |J| = |\det(J)|$
If $f$ is many-to-one, then $p_v(v)$ sums or integrates over all branches.
(c) Common transformations
- Log transformation (for positive variables):
- $v = \log(u), \quad u > 0 \Rightarrow v \in (-\infty, \infty)$
- Logistic (logit) transformation (for probabilities):
- $\text{logit}(u) = \log\frac{u}{1-u}, \quad \text{logit}^{-1}(v) = \frac{e^v}{1 + e^v}$
- Probit transformation:
- $v = \Phi^{-1}(u)$ where Φ is the standard normal cumulative distribution function, also mapping (0, 1) → (−∞, ∞).
These transformations are especially useful in Bayesian modeling to handle bounded parameters (e.g., probabilities, rates) by mapping them onto an unbounded scale for more convenient analysis and computation.
6. Summary of Key Identities
| Concept | Formula | Description |
|---|---|---|
| Joint distribution factorization | p(u,v,w)=p(u∣v,w)p(v∣w)p(w) | Expresses dependencies hierarchically |
| Marginalization | p(u)=∫p(u,v)dv | Integrates out variables |
| Total expectation | E(u)=E(E(u∣v)) | Mean equals mean of conditional means |
| Total variance | var(u)=E(var(u∣v))+var(E(u∣v)) | Total variance = within-group + between-group variance |
| Change of variables (continuous) | $p_v(v) = \left| J \right| \, p_u\left(f^{-1}(v)\right)$ | Adjusts the probability density using the Jacobian determinant when transforming variables. |
| Logit transform | $\text{logit}(u) = \log\frac{u}{1-u}$ | Maps (0, 1) → (−∞, ∞) |
| Probit transform | $v = \Phi^{-1}(u)$ | Maps (0, 1) → (−∞, ∞) |
7. Core Message
- Bayesian inference relies heavily on joint, conditional, and marginal probability manipulations.
- Complex models are most effectively represented using conditional or hierarchical formulations.
- Expectations and variances can always be decomposed via conditioning, clarifying sources of variation.
- Transformations (log, logit, probit) are crucial for handling parameters with restricted domains.
Together, these principles form the mathematical foundation for Bayesian modeling, computation, and interpretation in the rest of the text.