A major improvement in estimating regression coefficients in hierarchical models comes from imposing structure on how predictors are grouped. In applications such as election forecasting, different kinds of predictors (for example, national indicators, regional indicators, quantitative covariates) should be organized into clearly defined batches. Structuring predictors hierarchically allows Bayesian inference to pool information more effectively across related coefficients.
Analysis of variance (Anova) embodies this idea. Traditionally, Anova organizes predictors into factors and interactions, each forming a batch of coefficients. In hierarchical regression, the same concept applies: each row of an Anova table becomes a batch of exchangeable coefficients whose variability is itself a parameter of interest. Thus, Anova provides a natural conceptual framework for analyzing complex regression models with many grouped predictors.
1. Notation and Model Setup
Suppose the linear model consists of $M$ batches of predictors. Each batch $m$ contains $J_m$ coefficients:
- Coefficients are written as $\beta^{(m)}_j$, for $j = 1, \ldots, J_m$.
- The vector for batch $m$ is $\beta^{(m)} = \left( \beta^{(m)}1, \ldots, \beta^{(m)}{J_m} \right)$.
Classical least-squares estimates for this batch, written $\hat{\beta}^{(m)}$, satisfy $c_m$ linear constraints:
- The constraint matrix $C^{(m)}$ has dimension $c_m \times J_m$ and rank $c_m$.
- The constraints satisfy $C^{(m)} \hat{\beta}^{(m)} = 0$.
Thus the batch has $(\text{df})_m = J_m – c_m$ degrees of freedom.
The overall linear model for observation $i$ can be written as
$\displaystyle y_i = \sum_{m=0}^M \beta^{(m)}_{j^{(m)}_i}.$
Each observation pulls exactly one coefficient from each batch. This corresponds to a design matrix consisting entirely of zeros and ones.
A more general regression formulation replaces the above structure with:
$\displaystyle y_i = \sum_{m=0}^M \sum_{j=1}^{J_m} x^{(m)}_{ij} \beta^{(m)}_j,$
which allows arbitrary design matrices $X^{(m)}$. The distinguishing feature of the Anova-style representation is the hierarchical batching of coefficients $\beta^{(m)}_j$.
2. Hierarchical Priors for Batches
Each batch of coefficients is modeled as exchangeable:
$\displaystyle \beta^{(m)}_j \sim N(0, \sigma_m^2), \quad j = 1, \ldots, J_m.$
The mean can be set to zero without loss of generality; if the batch needs a nonzero mean, that mean would be represented as a separate predictor.
Variances for each batch have scaled inverse-$\chi^2$ priors:
$\displaystyle \sigma_m^2 \sim \text{Inv-}\chi^2(\nu_m, \sigma_{0m}^2).$
A noninformative prior corresponds to $(\nu_m = -1, \sigma_{0m} = 0)$, which implies a prior uniform on $\sigma_m$.
Interpretation:
- When $J_m$ is large, $\sigma_m$ is well estimated from the data.
- When $J_m$ is small, the weak prior allows $\sigma_m$ to be large, reducing shrinkage.
3. Computation Using the Gibbs Sampler
The posterior distribution of $(\beta, \sigma)$ can be sampled using a simple Gibbs sampler:
- Update $\beta$ using linear regression conditional on current $\sigma$.
- Update each $\sigma_m$ from its inverse-$\chi^2$ conditional posterior given $\beta^{(m)}$.
A difficulty arises if some $\sigma_m$ becomes close to zero, which can trap the Gibbs sampler.
4. Parameter Expansion for Improved Mixing
To improve efficiency, a parameter-expanded representation introduces new parameters:
$\displaystyle \beta^{(m)}_j = \zeta_m \gamma^{(m)}_j,$
$\displaystyle \sigma_m = |\zeta_m| \tau_m.$
In this reparameterization:
- $\gamma^{(m)}_j \sim N(0, \tau_m^2)$,
- $\tau_m^2 \sim \text{Inv-}\chi^2(\nu_m, \sigma_{0m}^2)$,
- $\zeta_m$ has a uniform prior.
The model becomes:
- $y = X (\zeta \gamma)$,
- update $\gamma$ by linear regression with $n$ observations and $\sum_m J_m$ predictors,
- update $\zeta$ by linear regression with $M$ predictors,
- update $\tau_m$ using inverse-$\chi^2$ distributions.
After each iteration, the original parameters are reconstructed as:
- $\beta^{(m)}_j = \zeta_m \gamma^{(m)}_j$,
- $\sigma_m = |\zeta_m| \tau_m$.
Parameter expansion prevents the sampler from sticking at $\sigma_m \approx 0$ and typically improves convergence dramatically.
5. Finite-Population vs. Superpopulation Standard Deviations
For each batch, two kinds of variability are meaningful:
Finite-population standard deviation (existing coefficients)
Defined as
$\displaystyle
s_m = \sqrt{ \frac{1}{(\text{df})_m} ,
\beta^{(m)T} \left[ I – C^{(m)T} \left( C^{(m)} C^{(m)T} \right)^{-1} C^{(m)} \right] \beta^{(m)} }.
$
This describes variation among the actual $J_m$ coefficients.
Superpopulation standard deviation ($\sigma_m$)
This describes variation for new, hypothetical coefficients drawn from the same batch distribution.
Key distinction
- $s_m$ can be well estimated even when $J_m$ is very small (e.g., $J_m = 2$).
- $\sigma_m$ still depends on a $\chi^2$ estimate with few degrees of freedom, making it uncertain.
Interpretation
The classical distinction between “fixed effects” and “random effects” becomes clearer in this framework:
- Finite-population inferences ($s_m$) describe the variation in the observed categories.
- Superpopulation inferences ($\sigma_m$) describe a broader family of possible categories.
Depending on the scientific question—e.g., U.S. states (finite) vs. individual subjects in psychology (superpopulation)—one measure may be more relevant.
6. Example: Five-Way Factorial Analysis of Web Connect Times
Data included:
- 45 locations (“from”),
- 4 destinations (“to”),
- 2 companies,
- 25 hours,
- Two separate weeks.
This produces a full factorial structure with no replication; the five-way interaction becomes the residual error.
A Bayesian Anova display plots the finite-population standard deviations $s_m$ with 50% and 95% posterior intervals. Interpretation:
- The lowest-level interaction contributes large variability.
- The main effect of destination (“to”) is also large.
- Company effects and its interactions with “to” and “from” exhibit substantial variation.
Because the model was fit on log(connect time), $s_m = 0.20$ corresponds to a coefficient of variation of approximately $20%$ on the original scale, meaning $\exp(\beta)$ indicates multiplicative differences.
Anova displays such as this are exploratory tools: they reveal which sources of variation are important and guide refinement of the regression model or future data collection.