Regression Prediction, Standardized Scores, and Regression Toward the Mean

Simple linear regression uses an observed explanatory value \(x\) to predict a response \(y\). The prediction can be calculated from five summary statistics:

• Mean of \(x\): \(\bar{x}\)
• Mean of \(y\): \(\bar{y}\)
• Standard deviation of \(x\): \(s_x\)
• Standard deviation of \(y\): \(s_y\)
• Correlation between \(x\) and \(y\): \(r\)

These quantities determine the least-squares regression line completely.

The Regression Prediction Formula

The regression line for predicting \(y\) from \(x\) is\[ \hat{y}=b_0+b_1x, \]

where the slope is\[ \boxed{ b_1=r\frac{s_y}{s_x} } \]

and the intercept is\[ \boxed{ b_0=\bar{y}-b_1\bar{x} }. \]

An equivalent and often more intuitive form is\[ \boxed{ \hat{y} = \bar{y} + r\frac{s_y}{s_x}(x-\bar{x}) }. \]

This formula starts at the average response \(\bar{y}\) and adjusts it according to how far \(x\) is from its own average.

Regression in Standardized Units

Define the standardized explanatory value as\[ z_x=\frac{x-\bar{x}}{s_x}. \]

Define the standardized predicted response as\[ z_{\hat{y}} = \frac{\hat{y}-\bar{y}}{s_y}. \]

The regression formula becomes\[ \boxed{ z_{\hat{y}}=rz_x }. \]

This is one of the simplest ways to understand regression:

If \(x\) is \(z_x\) standard deviations from its mean, regression predicts \(y\) to be \(r z_x\) standard deviations from its mean.

The correlation therefore controls how much of the standardized deviation is carried from the predictor into the prediction.

Example: Predicting a Final Score

Suppose the summary statistics are:\[ \bar{x}=49.5 \]

for midterm scores,\[ \bar{y}=69.1 \]

for final-exam scores,\[ s_x=10.2, \qquad s_y=11.8, \]

and\[ r=0.67. \]

A student scored\[ x=41 \]

on the midterm. We want to predict that student’s final score.

Method 1: Standardized-Score Reasoning

The student’s midterm score is\[ 41-49.5=-8.5 \]

points relative to the midterm average.

Standardizing gives\[ z_x = \frac{41-49.5}{10.2} = \frac{-8.5}{10.2} \approx -0.833. \]

Thus, the student’s midterm score is approximately 0.83 standard deviations below the midterm mean.

Regression predicts a standardized final score of\[ z_{\hat{y}} = rz_x. \]

Therefore,\[ z_{\hat{y}} = 0.67(-0.833) \approx -0.558. \]

The predicted final score is approximately 0.558 standard deviations below the final-exam average.

Convert this back to the original scale:\[ \hat{y} = \bar{y}+z_{\hat{y}}s_y. \]

Substituting the values,\[ \hat{y} = 69.1+(-0.558)(11.8). \]

Therefore,\[ \hat{y} \approx 69.1-6.59 \approx 62.51. \]

The predicted final score is\[ \boxed{\hat{y}\approx62.5}. \]

Method 2: Slope and Intercept

The slope is\[ b_1 = r\frac{s_y}{s_x} = 0.67\frac{11.8}{10.2}. \]

Thus,\[ b_1\approx0.775. \]

The intercept is\[ b_0 = \bar{y}-b_1\bar{x}. \]

Therefore,\[ b_0 \approx 69.1-(0.775)(49.5) \approx 30.74. \]

The fitted regression line is approximately\[ \hat{y} = 30.74+0.775x. \]

At \(x=41\),\[ \hat{y} = 30.74+0.775(41) \approx 62.5. \]

Both methods give the same prediction.

Why Regression Improves on the Overall Mean

If the midterm score were unavailable, a natural prediction for the final score would be the overall final-exam average:\[ \hat{y}=\bar{y}=69.1. \]

Knowing that the student scored below average on the midterm provides additional information. Since the correlation is positive, the final score is also predicted to be below average.

The regression prediction is therefore adjusted downward:\[ 69.1\longrightarrow62.5. \]

However, the predicted final is not as far below its mean in standardized units as the midterm score was below its mean.

The midterm score is\[ -0.833 \]

standard deviations from its mean, while the predicted final score is only\[ -0.558 \]

standard deviations from its mean.

This is regression toward the mean.

Regression Toward the Mean

When\[ |r|<1, \]

the standardized prediction\[ rz_x \]

is closer to zero than \(z_x\).

For example,\[ z_x=-0.833 \]

and\[ r=0.67 \]

give\[ rz_x=-0.558. \]

Thus, an unusually low midterm score predicts a below-average final score, but one that is less unusual relative to the final-score distribution.

Similarly, an unusually high midterm score predicts an above-average final score, but usually not one equally far above average in standardized units.

Regression toward the mean is a consequence of imperfect correlation. It does not mean that every individual necessarily moves closer to the average.

Important Properties of the Regression Line

The regression line always passes through the point of averages:\[ \boxed{ (\bar{x},\bar{y}) }. \]

If\[ x=\bar{x}, \]

then\[ \hat{y} = \bar{y} + r\frac{s_y}{s_x}(\bar{x}-\bar{x}) = \bar{y}. \]

Thus, an average value of the predictor produces an average predicted response.

The direction of the slope is determined by \(r\):

• If \(r>0\), the line slopes upward.
• If \(r<0\), the line slopes downward.
• If \(r=0\), the line is horizontal.
• If \(|r|=1\), the observations lie perfectly on a straight line.

Reversing the Prediction

Now suppose a student scored 89 on the final exam, and we want to predict the student’s midterm score.

This is a different regression problem.

Previously:

• Predictor: midterm score
• Response: final score

Now:

• Predictor: final score
• Response: midterm score

The variables must exchange roles.

A Numerical Correction

The final-exam mean is\[ 69.1. \]

Therefore, a final score of 89 is\[ 89-69.1=19.9 \]

points above the mean—not 9.9 points above it.

This arithmetic correction materially changes the prediction.

The standardized final score is\[ z_{\text{final}} = \frac{89-69.1}{11.8} = \frac{19.9}{11.8} \approx 1.686. \]

Thus, the final score is approximately 1.69 standard deviations above average.

The predicted standardized midterm score is\[ z_{\widehat{\text{midterm}}} = r z_{\text{final}}. \]

Therefore,\[ z_{\widehat{\text{midterm}}} = 0.67(1.686) \approx 1.130. \]

Convert this back to the midterm scale:\[ \widehat{\text{midterm}} = 49.5+(1.130)(10.2). \]

Thus,\[ \widehat{\text{midterm}} \approx 49.5+11.53 \approx 61.03. \]

The corrected prediction is\[ \boxed{ \widehat{\text{midterm}}\approx61.0 }. \]

The result 55.2 comes from incorrectly treating 89 as only 9.9 points above 69.1.

The Reverse Regression Formula

For predicting \(x\) from \(y\), the regression equation is\[ \hat{x} = \bar{x} + r\frac{s_x}{s_y}(y-\bar{y}). \]

The reverse-regression slope is\[ \boxed{ b_{x\mid y} = r\frac{s_x}{s_y} }. \]

Using the exam data,\[ b_{x\mid y} = 0.67\frac{10.2}{11.8} \approx 0.579. \]

The intercept is\[ a_{x\mid y} = \bar{x}-b_{x\mid y}\bar{y}. \]

Therefore,\[ a_{x\mid y} \approx 49.5-(0.579)(69.1) \approx 9.49. \]

The reverse regression line is approximately\[ \hat{x} = 9.49+0.579y. \]

At \(y=89\),\[ \hat{x} = 9.49+0.579(89) \approx 61.0. \]

Why the Original Regression Line Cannot Simply Be Reversed

The original regression of final score on midterm score is\[ \hat{y} = \bar{y} + r\frac{s_y}{s_x}(x-\bar{x}). \]

Its slope is\[ b_{y\mid x} = r\frac{s_y}{s_x}. \]

If we solve this equation algebraically for \(x\), the resulting slope is\[ \frac{1}{b_{y\mid x}} = \frac{s_x}{rs_y}. \]

But the actual slope for predicting \(x\) from \(y\) is\[ b_{x\mid y} = r\frac{s_x}{s_y}. \]

These are not equal unless\[ r^2=1. \]

Indeed,\[ \frac{s_x}{rs_y} \neq r\frac{s_x}{s_y} \]

when \(|r|<1\).

The difference occurs because ordinary least squares minimizes residuals in the response direction:

• Regression of \(y\) on \(x\) minimizes vertical squared residuals in \(y\).
• Regression of \(x\) on \(y\) minimizes horizontal squared residuals when drawn on the original axes.

These are different optimization problems.

Comparing the Two Regression Lines

The regression of final score on midterm score has slope\[ b_{y\mid x} = 0.67\frac{11.8}{10.2} \approx 0.775. \]

The regression of midterm score on final score has slope\[ b_{x\mid y} = 0.67\frac{10.2}{11.8} \approx 0.579. \]

If the second line is rewritten on the original axes, it is not the same as the first line. Both pass through their corresponding point of averages, but they serve different predictive purposes.

A reliable organizational rule is:

Put the predictor on the horizontal axis and the variable being predicted on the vertical axis.

The Role of Positive and Negative Correlation

The standardized prediction rule is\[ z_{\hat{y}}=rz_x. \]

The sign of the prediction depends on both \(r\) and \(z_x\):

For example, if\[ z_x=1.2 \]

and\[ r=-0.5, \]

then\[ z_{\hat{y}}=(-0.5)(1.2)=-0.6. \]

An \(x\)-value 1.2 standard deviations above average predicts a \(y\)-value 0.6 standard deviations below average.

Prediction Is Not Certainty

The fitted value\[ \hat{y}=62.5 \]

is a point prediction. It is not a guarantee that the student will receive exactly 62.5.

An individual outcome can be written as\[ y_i=\hat{y}_i+e_i, \]

where \(e_i\) is the residual.

Students with identical midterm scores can have different final scores because of:

• Study effort after the midterm
• Health or attendance
• Differences in preparation
• Measurement variation
• Random performance fluctuations
• Factors absent from the model

A prediction interval is needed to represent the uncertainty for an individual student.

A Correlation of 0.67 Leaves Considerable Variation

In simple regression,\[ R^2=r^2. \]

For\[ r=0.67, \]

we have\[ R^2=0.67^2=0.4489. \]

Thus, approximately 44.9% of the sample variation in final scores is accounted for by the fitted linear relationship with midterm scores.

The remaining fraction is\[ 1-R^2 = 1-0.4489 = 0.5511. \]

Therefore, approximately 55.1% of the squared sample variation remains in the residuals.

This explains why the point prediction should not be treated as an exact outcome.

Predicting in R

If the data are stored in a data frame named scores, the model can be fitted with:

model <- lm(final ~ midterm, data = scores)

A final score can then be predicted for a midterm score of 41:

predict(model, newdata = data.frame(midterm = 41))

To obtain a prediction interval for an individual student:

predict(
  model,
  newdata = data.frame(midterm = 41),
  interval = "prediction"
)

To predict midterm scores from final scores, fit a different model:

reverse_model <- lm(midterm ~ final, data = scores)

Then predict using:

predict(
  reverse_model,
  newdata = data.frame(final = 89)
)

The response and predictor must be exchanged in the model formula. The original model should not simply be solved backward.

Summary of the Two Predictions

The first calculation in the transcript is correct. The reverse calculation requires correcting\[ 89-69.1 \]

to\[ 19.9, \]

which changes the predicted midterm score from the stated 55.2 to approximately 61.0.

Key Takeaway

The regression line for predicting \(y\) from \(x\) is\[ \hat{y} = \bar{y} + r\frac{s_y}{s_x}(x-\bar{x}). \]

In standardized units, this becomes\[ z_{\hat{y}}=rz_x. \]

For a midterm score of 41, the predicted final score is approximately\[ \boxed{62.5}. \]

When reversing the prediction, a new regression line must be fitted:\[ \hat{x} = \bar{x} + r\frac{s_x}{s_y}(y-\bar{y}). \]

Because 89 is 19.9 points above the final-exam mean of 69.1, the corrected predicted midterm score is approximately\[ \boxed{61.0}. \]

Regression of \(y\) on \(x\) and regression of \(x\) on \(y\) solve different prediction problems. Except under perfect correlation, one line cannot be obtained by simply solving the other line backward.