Introduction to Regression: Improving Predictions with Additional Information

Regression is one of the most widely used techniques in statistics. Its central purpose is simple: use information about one or more variables to predict or explain variation in another variable.

The method can be applied to problems involving prediction, scientific relationships, business forecasting, risk analysis, and policy evaluation. Its versatility also creates potential pitfalls, so regression should always be accompanied by diagnostics and careful interpretation.

Prediction Without Additional Information

Suppose a dataset contains the heights of 928 adult sons. Let the height of a randomly selected son be denoted by\[ Y. \]

Assume the average height in the dataset is\[ \bar{y}=68.1\text{ inches}. \]

If we must predict the height of a randomly selected son but have no other information about him, a natural prediction is\[ \hat{y}=68.1\text{ inches}. \]

This prediction will not be exactly correct for most individuals. Some sons will be shorter, and others will be taller. Nevertheless, the mean has an important optimality property.

Why the Mean Is the Best Constant Predictor

Suppose we use the same constant \(c\) to predict every observed height. The residual for person \(i\) is\[ e_i=y_i-c. \]

Under squared-error loss, the total prediction error is\[ SSE(c) = \sum_{i=1}^{n}(y_i-c)^2. \]

The value of \(c\) that minimizes this sum is the sample mean:\[ \boxed{c=\bar{y}}. \]

To see this, differentiate with respect to \(c\):\[ \frac{d}{dc} \sum_{i=1}^{n}(y_i-c)^2 = -2\sum_{i=1}^{n}(y_i-c). \]

Set the derivative equal to zero:\[ \sum_{i=1}^{n}(y_i-c)=0. \]

Therefore,\[ \sum_{i=1}^{n}y_i-nc=0, \]

so\[ c = \frac{1}{n} \sum_{i=1}^{n}y_i = \bar{y}. \]

Thus, if prediction error is measured by squared deviations, no other constant prediction performs better on the observed data.

The mean is the best constant predictor under squared-error loss.

“Best” Depends on the Loss Function

The mean is not universally the best predictor under every definition of error.

Squared Error

If the loss is\[ (y-c)^2, \]

the optimal prediction is the mean.

Absolute Error

If the loss is\[ |y-c|, \]

the optimal prediction is a median.

Asymmetric Costs

If underprediction and overprediction have different consequences, an appropriate quantile may be better.

Therefore, the statement that the mean is the best predictor should always be understood relative to a loss function. Ordinary least-squares regression uses squared error.

Population Version

At the population level, suppose \(Y\) is a random variable and we choose a constant \(c\) to minimize the expected squared prediction error:\[ E[(Y-c)^2]. \]

Write\[ Y-c=(Y-\mu)+(\mu-c), \]

where\[ \mu=E(Y). \]

Expanding gives\[ E[(Y-c)^2] = E[(Y-\mu)^2] + 2(\mu-c)E(Y-\mu) + (\mu-c)^2. \]

Because\[ E(Y-\mu)=0, \]

we obtain\[ E[(Y-c)^2] = \operatorname{Var}(Y) + (\mu-c)^2. \]

The variance does not depend on \(c\), and the second term is minimized when\[ c=\mu. \]

Therefore,\[ \boxed{ \arg\min_c E[(Y-c)^2]=E(Y) }. \]

What the Mean Does Not Use

Predicting every son’s height with 68.1 inches ignores any information that might distinguish one son from another.

The same prediction is given to:

• Sons of short fathers
• Sons of average-height fathers
• Sons of tall fathers

If another variable is related to the response, it may help produce more accurate predictions.

This is the central motivation for regression.

Adding the Father’s Height

Suppose each son is paired with his father, giving observations\[ (x_1,y_1),(x_2,y_2),\ldots,(x_{928},y_{928}), \]

where:

• \(x_i\) is the father’s height.
• \(y_i\) is the son’s height.

A scatterplot of these pairs displays how sons’ heights vary with fathers’ heights.

If the plot shows a positive association, taller fathers tend to have taller sons on average. The relationship will not be perfect, but the father’s height contains predictive information.

Now suppose we know that a particular father is 72 inches tall. Instead of using the overall mean of all sons, we can estimate the average height among sons whose fathers are approximately 72 inches tall.

Conceptually, the relevant prediction becomes\[ E(Y\mid X=72), \]

rather than\[ E(Y). \]

Marginal and Conditional Means

The overall mean son’s height is the marginal mean:\[ E(Y). \]

It does not condition on the father’s height.

The average height of sons whose fathers have a particular height is a conditional mean:\[ E(Y\mid X=x). \]

Regression estimates how this conditional mean changes with \(x\).

The comparison is:

Available information	Squared-error prediction
No predictor information	\(E(Y)\)
Predictor value \(X=x\) is known	\(E(Y\mid X=x)\)

If \(X\) contains useful information about \(Y\), then conditional prediction can reduce expected squared error.

Why the Conditional Mean Is Optimal

Suppose \(X=x\) is known and we choose a prediction \(g(x)\). The conditional expected squared error is\[ E\left[ (Y-g(x))^2\mid X=x \right]. \]

This quantity is minimized by\[ \boxed{ g(x)=E(Y\mid X=x) }. \]

Therefore, the theoretically optimal squared-error prediction function is the conditional mean:\[ m(x)=E(Y\mid X=x). \]

Regression attempts to estimate this function from data.

The Linear Regression Model

Simple linear regression approximates the conditional mean with a straight line:\[ E(Y\mid X=x) \approx \beta_0+\beta_1x. \]

The fitted version is\[ \boxed{ \hat{y}=b_0+b_1x }. \]

Here:

• \(b_0\) is the fitted intercept.
• \(b_1\) is the fitted slope.
• \(x\) is the known predictor value.
• \(\hat{y}\) is the predicted response.

For a father who is 72 inches tall, the predicted son’s height is\[ \hat{y} = b_0+b_1(72). \]

The actual value depends on the fitted coefficients calculated from the father–son data.

Interpreting the Slope

The slope \(b_1\) describes the predicted change in the son’s height associated with a one-unit increase in the father’s height.

If\[ b_1=0.4, \]

then an additional inch of father’s height is associated with an increase of 0.4 inches in predicted son’s height:\[ \Delta\hat{y}=0.4\Delta x. \]

This is a predictive or associational statement. It does not mean that increasing a father’s height through an intervention would cause his son to become taller.

The Regression Line as a Better Predictor

Without knowing the father’s height, the prediction is\[ \hat{y}=\bar{y}=68.1. \]

With the father’s height, the prediction becomes\[ \hat{y}=b_0+b_1x. \]

If \(x=72\) is above the average father’s height and the association is positive, the regression prediction will generally be above the average son’s height.

The prediction incorporates the fact that sons associated with taller fathers tend to be located toward the higher end of the son-height distribution.

Better Does Not Mean Perfect

Even if father’s height improves the prediction, sons of 72-inch fathers do not all have the same height.

Their heights vary because of:

• The other parent’s height
• Genetic recombination
• Nutrition
• Health
• Environmental conditions
• Measurement variation
• Other unmeasured factors

The fitted value describes a conditional center, not an exact outcome.

An individual response can be written as\[ Y=\hat{Y}+e, \]

where \(e\) represents the residual deviation from the fitted prediction.

Explained and Residual Variation

Using the overall mean produces errors\[ y_i-\bar{y}. \]

The total squared error of this constant predictor is\[ SST = \sum_{i=1}^{n}(y_i-\bar{y})^2. \]

Using the regression line produces residuals\[ e_i=y_i-\hat{y}_i \]

and residual sum of squares\[ SSE = \sum_{i=1}^{n}(y_i-\hat{y}_i)^2. \]

Because the intercept-only model \(\hat{y}=\bar{y}\) is one candidate available to least squares,\[ SSE\leq SST \]

on the training data.

The proportional reduction in squared error is\[ R^2 = 1-\frac{SSE}{SST}. \]

In simple regression with an intercept,\[ R^2=r^2. \]

Regression Toward the Mean

Suppose the correlation between fathers’ and sons’ heights is positive but less than 1. In standardized units, the regression prediction is\[ z_{\hat{y}}=rz_x. \]

If a father is two standard deviations above the father-height mean and\[ r=0.5, \]

then the predicted son’s height is\[ z_{\hat{y}}=0.5(2)=1 \]

standard deviation above the son-height mean.

The son is predicted to be taller than average, but less extreme in standardized units than the father.

This is regression toward the mean.

It does not mean that every tall father has a shorter son. It describes the expected height of a group selected for an extreme father height.

Why Regression Can Improve Prediction

The law of total variance gives a deeper explanation:\[ \operatorname{Var}(Y) = \operatorname{Var}\left(E(Y\mid X)\right) + E\left[\operatorname{Var}(Y\mid X)\right]. \]

The first term measures variation in conditional means across predictor values. The second measures residual variation remaining within predictor groups.

Predicting with \(E(Y\mid X)\) uses the variation associated with \(X\), leaving only the conditional variation as irreducible error.

More generally,\[ E\left[ (Y-E(Y))^2 \right] – E\left[ (Y-E(Y\mid X))^2 \right] = \operatorname{Var}(E(Y\mid X)). \]

Thus, conditional prediction cannot be worse than predicting the overall mean at the population level when the true conditional mean is known.

An estimated regression model can still perform poorly on new data if it is overfit or misspecified, so out-of-sample validation remains important.

Prediction Versus Causal Explanation

Regression can be useful even when the relationship is not causal.

Father’s height may predict a son’s height because it contains information about shared genetics and family characteristics. But a regression coefficient does not isolate a causal effect automatically.

More generally, regression associations may reflect:

• Direct causation
• Reverse causation
• Confounding
• Selection effects
• Measurement processes
• Shared underlying causes

Prediction asks:

Does knowing \(X\) help forecast \(Y\)?

Causal inference asks:

What would happen to \(Y\) if we intervened and changed \(X\)?

These are different questions.

Regression Inference

A fitted regression line is based on a sample and therefore contains sampling uncertainty.

Regression inference addresses questions such as:

• Is the population slope plausibly zero?
• What values of the slope are compatible with the data?
• How precisely is the mean response estimated?
• What range is plausible for a new individual response?
• Will the observed relationship generalize beyond the sample?

Common inferential tools include:

• Standard errors for coefficients
• Confidence intervals
• Hypothesis tests
• Confidence bands
• Prediction intervals

These procedures require assumptions about sampling, independence, model form, and error behavior.

Regression Diagnostics

Before trusting a regression model, examine whether its assumptions and structure are reasonable.

Important diagnostics include:

• Scatterplots
• Residual-versus-fitted plots
• Residual-versus-predictor plots
• Normal Q–Q plots
• Leverage values
• Studentized residuals
• Cook’s distance
• Case-deletion comparisons

Diagnostics can reveal:

• Curvature
• Heteroscedasticity
• Outliers
• Influential observations
• Dependence
• Missing predictors
• Separate subgroups

A fitted line should not be interpreted in isolation.

Common Regression Pitfalls

Regression is versatile, but several mistakes are common.

Extrapolation

Predictions outside the observed predictor range may be unreliable because the relationship can change.

Confusing Association with Causation

A fitted slope does not automatically measure a causal effect.

Ignoring Curvature

A straight line can conceal an important nonlinear relationship.

Ignoring Unequal Variance

Changing residual spread can invalidate conventional standard errors.

Allowing One Observation to Dominate

High-leverage or influential observations can substantially alter the fitted line.

Overfitting

Adding many predictors can improve training fit while harming future predictions.

Ignoring Study Design

A sophisticated regression cannot repair a fundamentally biased sample or uncontrolled study automatically.

Key Takeaway

Without information about an individual son, the mean height\[ \bar{y}=68.1\text{ inches} \]

is the best constant prediction under squared-error loss.

When the father’s height \(X=x\) is known, a more informative prediction is the conditional mean\[ E(Y\mid X=x). \]

Simple linear regression estimates that conditional mean with\[ \hat{y}=b_0+b_1x. \]

For a father who is 72 inches tall, the regression prediction uses the observed father–son relationship rather than assigning the overall average to every son.

Regression can improve prediction substantially, but it does not eliminate individual variation or establish causation. Reliable use requires model diagnostics, uncertainty estimates, careful sampling, and predictions within a defensible range.