This section explains how numerical prior information can be incorporated into a Bayesian linear regression model. The key idea is that normal and inverse-χ² priors behave like extra data points, so we can express the prior as an “augmented regression” with synthetic observations.

For the regression model:

$y \mid X, \beta, \sigma^{2} \sim N(X\beta,\ \sigma^{2} I)$

the unknown parameters are:

$\theta = (\beta,\ \sigma^{2})$

---

1. Prior Information on a Single Coefficient $\beta_j$​

Suppose we have prior knowledge about one coefficient:

$\beta_j \sim N(\beta_{j0},\ \sigma_{\beta_j}^{2})$

This has density:

$p(\beta_j) \propto \exp\!\left( -\frac{(\beta_j – \beta_{j0})^{2}}{2\sigma_{\beta_j}^{2}} \right)$

This is mathematically identical to treating the prior as an additional “pseudo-observation”:

• “observed value” = $\beta_{j0}$
• “predictor row” = a row vector with 1 in the j-th position and 0 elsewhere $x^{*} = (0,\dots,0,1,0,\dots,0)$
• “observation variance” = $\sigma_{\beta_j}^{2}$

Thus we append to the regression:

$y^{*} = \begin{pmatrix} y \\ \beta_{j0} \end{pmatrix}, \qquad X^{*} = \begin{pmatrix} X \\ e_j^\top \end{pmatrix}, \qquad \Sigma_y^{*} = \begin{pmatrix} \sigma^{2}I & 0 \\ 0 & \sigma_{\beta_j}^{2} \end{pmatrix}$

Then the posterior for $\beta$ is simply the weighted least-squares posterior using
$y^{*}, X^{*}, \Sigma_y^{*}$​.

---

Edge cases

1. No prior information

If $\sigma_{\beta_j}^{2} \to \infty,$

the extra observation becomes infinitely noisy → negligible influence.

2. Perfect prior information

If $\sigma_{\beta_j}^{2} = 0,$

variance = 0 → $\beta_j$​ is forced to equal $\beta_{j0}$​.

---

2. Prior Information on All Coefficients $\beta$

Assume a multivariate normal prior:

$\beta \sim N(\beta_0,\ \Sigma_{\beta})$

This can be written as k pseudo-observations, one for each element of β.

Augmented form:

$y^{*} = \begin{pmatrix} y \\ \beta_0 \end{pmatrix}, \qquad X^{*} = \begin{pmatrix} X \\ I_k \end{pmatrix}$

$\Sigma^{*} = \begin{pmatrix} \Sigma_y & 0 \\ 0 & \Sigma_{\beta} \end{pmatrix}$

Interpretation

Each prior constraint contributes a synthetic observation that nudges the estimate of β.

Note

If some β components have noninformative priors (variance = ∞), they are omitted from the augmented block to avoid infinite terms, or handled at the precision-matrix level.

---

3. Prior Information About the Variance $\sigma^{2}$

The conjugate prior is a scaled inverse-χ²:

$\sigma^{2} \sim \text{Inv-}\chi^{2}(n_{0},\ \sigma_{0}^{2})$

This behaves like $n_{0}$​ pseudo-observations whose sample variance is $\sigma_{0}^{2}$​.

Posterior

Given sample size nnn and sample variance $s^{2}$:

$\sigma^{2} \mid y \sim \text{Inv-}\chi^{2} \left( n_0 + n,\; \frac{n_0\sigma_0^{2} + n s^{2}}{n_0 + n} \right)$

If prior information about β was also incorporated

Use:

• $y^{*}, X^{*}$ from the augmented regression
• replace $s^{2}$ with the augmented residual variance
• replace $n$ with the length of $y^{*}$

---

4. Prior Information on Several Variance Components

If the model has several variance parameters:

$y_i \mid \beta, \sigma_{g(i)}^{2} \sim N(x_i^\top \beta,\ \sigma_{g(i)}^{2})$

with groups $g=1,\dots,I$, the joint posterior is:

$p(\beta, \sigma_1^{2}, \ldots, \sigma_I^{2} \mid y) \propto \prod_{i=1}^{I} (\sigma_i^{2})^{-n_i/2} \exp\!\left( -\frac{1}{2}(y – X\beta)^\top \Sigma_y^{-1} (y – X\beta) \right)$

Inference typically uses Gibbs sampling because:

• $\beta \mid \sigma_1^2,\dots,\sigma_I^2$​ is normal
• each $\sigma_i^2 \mid \beta$ is inverse-χ²

---

5. Prior Information as Inequality Constraints

Examples:

$\beta_1 \ge 0, \qquad \beta_2 \le \beta_3 \le \beta_4$

Approach:

1. Draw samples from the unconstrained posterior.
2. Reject all draws that violate the constraints.

This produces the correct constrained posterior.

When rejection is inefficient

If constraints remove most samples → the model structure contradicts the data → hierarchical modeling or more informative priors are needed.

---

6. Why Treating Priors as Pseudo-Data Is Powerful

1. Computational simplicity

Posterior computation becomes:

$\hat{\beta} = \left(X^{*\top} \Sigma^{*-1} X^{*}\right)^{-1} X^{*\top} \Sigma^{*-1} y^{*}$

Weighted regression machinery handles everything.

---

2. Unified and modular framework

Any combination of:

• coefficient priors
• variance priors
• constraints
• group variances

all integrate naturally into the augmented regression.

---

3. Foundation for hierarchical models

Hierarchical normal models:

$\beta_j \mid \mu, \tau^{2} \sim N(\mu,\ \tau^{2})$

are exactly equivalent to normal priors discussed here → pseudo-data viewpoint extends naturally.

---

Summary

Numerical prior information can be integrated into Bayesian regression in four main ways:

(1) Normal priors on β

$\beta \sim N(\beta_0,\ \Sigma_\beta)$

→ equivalent to adding pseudo-observations → use augmented regression.

---

(2) Inverse-χ² priors on σ²

$\sigma^{2} \sim \text{Inv-}\chi^{2}(n_0,\ \sigma_0^2)$

→ equivalent to $n_0$​ prior observations with variance $\sigma_0^2$​.

---

(3) Multiple variance components

Handled by Gibbs sampling using blockwise normal / inverse-χ² updates.

---

(4) Parameter constraints

Handled via post-hoc rejection of unconstrained posterior draws.