1. Identifiability and Collinearity
Regression parameters cannot be uniquely estimated if:
- There are more parameters than data points, or
- The columns of $X$ are linearly dependent or nearly dependent.
When columns of $X$ are not linearly independent, the data are collinear, meaning the likelihood does not uniquely identify the coefficients $\beta$. Even with a Bayesian prior, collinearity creates very large posterior variances and unstable estimates.
Geometric interpretation
- Think of the $n$ data points in a $k$-dimensional predictor space.
- If the points lie on a lower-dimensional hyperplane, then $X$ is collinear.
- If points are nearly on a plane, the regression is nearly collinear—leading to weak identifiability and high uncertainty.
Incumbency example
- Suppose incumbents running for reelection always had 70% in the previous election.
- Suppose open seats always occurred in districts where the previous vote was 60%.
Then:
$\text{Previous vote} = 0.6 + 0.1 R_i$
Thus:
- Constant term
- Incumbency indicator $R_i$
- Previous vote
are perfectly collinear → impossible to estimate all three coefficients uniquely.
A nearly collinear dataset would also cause inflated posterior variances and make estimates extremely sensitive to modeling assumptions (such as linearity of $E(y \mid x)$).
2. Nonlinear Relationships
Once variables are chosen, they may need transformation to achieve approximate linearity between predictors $x$ and outcome $y$.
Common transformations
- logarithms
- square roots
- logits
Transformations improve model fit but alter interpretation:
- Coefficient now represents change in transformed outcome per unit change in transformed predictor.
Allowing flexible shapes
To capture nonlinear effects, one can include several transformed versions of the same variable:
- Include both $x_j$ and $x_j^2$ → quadratic relationship
- Higher-order polynomials
- Splines or nonparametric functions
If $y$ is discrete, generalized linear models (GLMs) may be more appropriate.
3. Indicator Variables
Categorical variables must be encoded into the design matrix using indicator (dummy) variables.
Encoding rules
- Binary category → one 0/1 indicator
- $k$ categories → create $k-1$ indicators plus a constant term
- This avoids linear dependence between indicators and intercept (full-rank requirement)
Indicator coefficients are often incorporated into hierarchical models to allow partial pooling across categories.
4. Ordered and Grouped Variables
Ordered categorical variables
If categories have natural order (e.g., grades A, B, C, D):
- It might be appropriate to code as numerical values (e.g., 4, 3, 2, 1).
- This approach captures a linear trend across categories.
Common in epidemiology → “trend analysis.”
Grouping continuous variables
Sometimes researchers group continuous variables into categories to:
- Simplify interpretation
- Explore nonlinearities by comparing group means
However, grouping loses information and should be done carefully.
5. Interaction Terms
Interactions capture situations where the effect of one variable depends on the level of another.
Example
If the effect of $x_i$ on $y$ depends on $x_j$, include:
$(x_i – \bar{x}_i)(x_j – \bar{x}_j)$
Centering variables prevents unnecessary correlation with the intercept.
Notes
- Interactions involving two continuous variables can be hard to interpret.
- Often better to categorize at least one variable to improve interpretability.
- Nonparametric models can capture interactions without specifying functional form.
6. Controlling for Irrelevant or Weird Variables
Regression sometimes highlights variables that look “silly” but appear statistically predictive.
Example:
- The length of the candidate’s last name might correlate with vote share.
- It might simply be a proxy for ethnicity.
Even if such variables do not harm causal inference (because they are pre-treatment), they should not be kept without understanding their underlying meaning.
Researchers should replace such variables with more meaningful ones when possible.
7. Selecting Explanatory Variables
Goal
Include all variables that could reasonably help predict the outcome or satisfy conditional ignorability for causal inference.
Challenges
- Real studies often have many potential predictors
- Some predictors have weak or no effect
- Including “too many variables” can hurt classical regression models (not Bayesian hierarchical models)
Classical regression constraints
- With a noninformative prior, require $k < n$
- Too many predictors → small residual degrees of freedom
- Variance estimates become unstable → classic overfitting problem
Bayesian benefit
Hierarchical priors or regularizing priors largely solve overfitting issues, allowing many predictors without destabilizing the model.
Summary
Constructing the explanatory variable matrix $X$ involves decisions about variable selection, encoding, transformation, and interaction specification. The modeling choices must balance identifiability, interpretability, and alignment with the analytic goals (prediction vs. causal inference). Issues such as collinearity, nonlinear relationships, categorical encoding, and irrelevant predictors can severely impact estimation accuracy, predictive power, and causal validity. Bayesian methods, especially hierarchical modeling, offer robustness to many classical problems—particularly overfitting—by introducing informative structure through priors.