Regression analysis serves three major purposes:
(1) understanding how an outcome variable changes as a function of explanatory variables,
(2) predicting outcomes for new observations, and
(3) performing causal inference to understand how an intervention or manipulation would change the outcome.
1. Understanding how $y$ varies as a function of $x$
This is the classical goal of regression.
A regression model describes the conditional distribution:
$p(y \mid x, \theta)$
where
- $y$ = outcome (response)
- $x$ = explanatory variables
- $\theta$ = parameters such as $\beta$ and $\sigma$
This goal focuses on explaining patterns, associations, linear trends, or determining which predictors help explain variability in the data.
For example, in congressional elections, researchers may ask:
“Which factors help predict the Democratic vote share in a district?”
Regression answers this by quantifying how changes in explanatory variables correspond to changes in the mean of $y$.
2. Predicting $y$ for new observations
Once parameters have been estimated, regression models allow predictions for future or unobserved data:
$\tilde{y} \sim p(\tilde{y} \mid \tilde{x}, y)$
Prediction requires assuming the new and existing cases are exchangeable given $x$.
Key idea: Exchangeability conditional on x
We assume:
- If two units have identical $x$, then their outcomes come from the same conditional distribution.
- All relevant information distinguishing units must be included in $x$.
If something important is not included in $x$, predictions may be biased.
Example: Predicting test scores
Suppose:
- $y_i$: reading score at end of second grade
- $x_i$: reading score at beginning of second grade + constant term
After fitting a regression on 100 students, we can produce a posterior predictive distribution for:
$\tilde{y} \mid \tilde{x}$
for a new student.
If the new student differs from past students
Prediction becomes less reliable.
If the original 100 students came from School A and the new student is from School B, and “school” is not in $x$, then exchangeability fails.
We must include school indicators in $x$ to restore exchangeability.
When predictors are missing for the new observation
If parts of $\tilde{x}$ are unknown, the missing components themselves must be modeled probabilistically — leading to more complex Bayesian models.
3. Causal inference
Causal inference is more subtle because it requires interpreting regression coefficients as representing effects of hypothetical interventions.
Treatment vs. Control variables
- Treatment variable: variable whose causal effect is of interest
- Control variables (covariates): variables that affect both treatment assignment and outcome, required for ignorability
The goal is to estimate:
$\text{Causal effect} = \mathbb{E}(y \mid \text{do}(x=1)) – \mathbb{E}(y \mid \text{do}(x=0))$
Regression achieves this only if treatment assignment is ignorable given the control variables.
Do NOT control for post-treatment variables
A critical rule:
You must not adjust for variables that occur after the treatment is assigned.
Example: incumbency advantage
- Treatment: incumbent runs or not
- Outcome: vote share
- Wrong control variable: total campaign spending (because spending occurs after the incumbent decides to run)
Controlling for a post-treatment variable would block part of the causal effect and bias the estimate downward.
Acceptable control variables
- Previous vote share
- Partisan lean of district
- Any variable measured before the treatment decision
These help satisfy conditional ignorability.
Putting it all together: Goals of regression in Bayesian terms
Understanding
Posterior for parameters:
$p(\theta \mid X, y)$
allows exploring how each predictor affects $y$.
Prediction
Posterior predictive distribution:
$\tilde{y} \sim p(\tilde{y} \mid X, y) = \int p(\tilde{y} \mid \theta) \, p(\theta \mid X, y)\, d\theta$
captures both:
- Sampling uncertainty (noise)
- Parameter uncertainty (finite sample)
Causal inference
Requires reinterpreting regression through:
- treatment assignment mechanism
- conditional ignorability
- valid control variables
Regression estimates become approximations to:
$\mathbb{E}(y^1 – y^0)$
when model assumptions are satisfied.
Summary
Regression analysis aims to (1) understand relationships between variables, (2) predict future outcomes, and (3) conduct causal analysis. Predictive use assumes new units are exchangeable with old ones given predictor variables; this requires including all relevant information in the explanatory variables. If units differ along omitted important dimensions, predictions become unreliable, and additional covariates must be used.
For causal inference, regression must be interpreted through the lens of treatment and control variables. The treatment variable is the manipulable factor whose effect is of interest (e.g., incumbency), while control variables adjust for confounding factors. However, only pre-treatment control variables should be included; controlling for variables that occur after the treatment decision biases the causal effect by conditioning on consequences of the treatment itself.
Bayesian regression formalizes these ideas via posterior distributions for parameters and posterior predictive distributions, enabling rich uncertainty quantification. Understanding, prediction, and causation all rely on the same mathematical backbone but differ in interpretation, assumptions, and how covariates are selected.