Normal Approximation Around a Regression Line

A regression line predicts the center of the response variable \(Y\) for a given value of the explanatory variable \(X\). Under additional assumptions, regression can also describe how individual \(Y\)-values vary around that predicted center.

This makes it possible to estimate probabilities such as:

Among students with a particular midterm score, what percentage are expected to score above a specified value on the final exam?

The Regression Line Predicts a Conditional Center

The simple linear regression equation can be written as\[ \hat{y} = \bar{y} + r\frac{s_y}{s_x}(x-\bar{x}), \]

where:

• \(\bar{x}\) and \(\bar{y}\) are the sample means.
• \(s_x\) and \(s_y\) are the sample standard deviations.
• \(r\) is the sample correlation.
• \(\hat{y}\) is the predicted response for a given \(x\).

The regression line gives the estimated conditional mean:\[ \hat{y}(x)\approx E(Y\mid X=x). \]

For example, suppose the model predicts that a student with a midterm score of 41 will have a final-exam score of\[ \hat{y}=62.5. \]

This does not mean every student with a midterm score near 41 will score 62.5 on the final. It means that 62.5 is the predicted center of their final scores.

The Conditional Distribution of \(Y\)

To make probability statements about individual responses, we need more than the regression line. We also need a model for the variation around it.

A common regression model is\[ Y=\beta_0+\beta_1X+\varepsilon, \]

with\[ \varepsilon\mid X=x\sim N(0,\sigma_e^2). \]

Under this model,\[ Y\mid X=x \sim N\left( \beta_0+\beta_1x,\, \sigma_e^2 \right). \]

The conditional distribution has:

• Center \(\beta_0+\beta_1x\)
• Standard deviation \(\sigma_e\)

Using fitted values, this becomes approximately\[ Y\mid X=x \approx N\left( \hat{y}(x),\, s_e^2 \right), \]

where \(s_e\) estimates the residual standard deviation.

Conditions for Normal Approximation

A vaguely football-shaped scatterplot can support a linear regression analysis, but the visual shape alone does not guarantee that conditional normal approximation is valid.

The usual model assumes:

1. Linearity\[ E(Y\mid X=x)=\beta_0+\beta_1x. \]
2. Constant conditional variance\[ \operatorname{Var}(Y\mid X=x)=\sigma_e^2. \]
3. Approximately normal conditional errors\[ \varepsilon\mid X=x\sim N(0,\sigma_e^2). \]
4. Independence across observationsOne observation’s error should not determine another’s.

A football-shaped scatterplot is consistent with these assumptions when it resembles an elliptical cloud with roughly constant vertical spread. Residual plots and normal quantile–quantile plots provide more direct diagnostic evidence.

A regression line can still be computed when these assumptions fail. The assumptions determine whether the associated probability calculations and conventional inference are reliable.

The Regression Standard Deviation

In simple linear regression, the residual spread is related to the response standard deviation and the correlation.

A descriptive version of the regression standard deviation is\[ \boxed{ s_y\sqrt{1-r^2} }. \]

This quantity measures the typical vertical variation around the regression line.

Because\[ R^2=r^2 \]

in simple linear regression with an intercept, the unexplained fraction of the response variation is\[ 1-r^2. \]

The corresponding residual scale is therefore approximately\[ s_e\approx s_y\sqrt{1-r^2}. \]

Why the Square Root Appears

The total variation in \(Y\) is measured by \(s_y^2\). In simple regression, the proportion of sample variation accounted for by the line is \(r^2\). The unexplained proportion is\[ 1-r^2. \]

Thus, the residual variance is proportional to\[ (1-r^2)s_y^2. \]

Taking the square root gives the residual standard deviation:\[ s_e\approx\sqrt{(1-r^2)s_y^2} = s_y\sqrt{1-r^2}. \]

Exact Degrees-of-Freedom Adjustment

There is a small distinction between the descriptive residual scale and the conventional residual standard error.

For ordinary least squares with an intercept,\[ SSE=(1-r^2)SST, \]

where\[ SST=\sum_{i=1}^{n}(y_i-\bar{y})^2. \]

If the sample standard deviation is defined by\[ s_y^2=\frac{SST}{n-1}, \]

then the conventional residual standard error is\[ s_e = \sqrt{\frac{SSE}{n-2}}. \]

Substituting \(SSE=(1-r^2)SST\) gives\[ \boxed{ s_e = s_y \sqrt{ (1-r^2)\frac{n-1}{n-2} } }. \]

For a large sample,\[ \sqrt{\frac{n-1}{n-2}}\approx1, \]

so\[ s_e\approx s_y\sqrt{1-r^2}. \]

The simpler formula is therefore a close large-sample approximation.

Example: Midterm and Final Scores

Suppose:\[ r=0.67, \]\[ s_y=11.8, \]

and a midterm score of 41 gives the predicted final score\[ \hat{y}=62.5. \]

We want to estimate the percentage of students with midterm scores around 41 whose final scores exceed 60.

Step 1: Find the Conditional Center

The regression line gives\[ E(Y\mid X=41)\approx62.5. \]

Thus, the conditional final-score distribution is centered at 62.5.

Step 2: Estimate the Conditional Spread

Using the simplified regression standard deviation,\[ s_e \approx s_y\sqrt{1-r^2}. \]

Substitute \(r=0.67\) and \(s_y=11.8\):\[ s_e \approx 11.8\sqrt{1-0.67^2}. \]

Since\[ 0.67^2=0.4489, \]

we have\[ 1-0.4489=0.5511. \]

Therefore,\[ s_e \approx 11.8\sqrt{0.5511}. \]

Since\[ \sqrt{0.5511}\approx0.7424, \]

the residual standard deviation is approximately\[ s_e\approx11.8(0.7424)\approx8.76. \]

The approximate conditional model is\[ Y\mid X\approx41 \sim N(62.5,8.76^2). \]

Step 3: Standardize the Threshold

We want\[ P(Y>60\mid X\approx41). \]

Standardize 60:\[ z = \frac{60-62.5}{8.76}. \]

Thus,\[ z \approx \frac{-2.5}{8.76} \approx -0.285. \]

The value 60 is approximately 0.285 residual standard deviations below the predicted center.

Step 4: Find the Upper-Tail Probability

We need\[ P(Z>-0.285). \]

Using the symmetry of the standard normal distribution,\[ P(Z>-0.285) = \Phi(0.285). \]

This is approximately\[ 0.612. \]

Therefore,\[ \boxed{ P(Y>60\mid X\approx41)\approx0.61 }. \]

Under the fitted model, approximately 61% of students with midterm scores around 41 are expected to score above 60 on the final exam.

Small differences such as 61.2% versus 61.4% can result from rounding the correlation, predicted value, residual scale, or normal-table entries.

Visual Interpretation

The conditional normal curve is centered at\[ 62.5. \]

The threshold 60 lies slightly to the left of the center. Because more than half of a normal distribution lies above a value below its mean, the resulting probability must be greater than 50%.

The standardized distance is only about\[ 0.29 \]

standard deviations below the mean, so an answer near 61% is reasonable.

“At \(X=41\)” Versus “Around \(X=41\)”

For a continuous predictor, the probability of observing exactly \(X=41\) may be conceptually different from examining a neighborhood around 41.

The regression model defines a conditional distribution at\[ X=41. \]

An empirical analysis may instead group students whose midterm scores fall in a small interval, such as\[ 40.5\leq X<41.5. \]

If the regression relationship and residual variance do not change materially across that interval, the distribution for students “around 41” can be approximated using the fitted value at 41.

If the interval is wide, the group combines different conditional means and may have greater spread than the model’s distribution at exactly \(X=41\).

Individual Outcomes Versus Mean Responses

There are two different prediction questions.

Where Is the Mean Response?

The first question asks for the average final score among all students with \(X=x_0\):\[ E(Y\mid X=x_0). \]

The fitted estimate is\[ \hat{y}_0=b_0+b_1x_0. \]

Uncertainty about this estimated mean is described by a confidence interval.

Where Will One Individual Response Fall?

The second question asks about the final score of one particular student with \(X=x_0\).

This requires a prediction interval, which is wider because it includes:

1. Uncertainty in the estimated regression line
2. Individual variation around the line

The lecture-style calculation using\[ s_y\sqrt{1-r^2} \]

primarily describes individual conditional variation around the regression line. It is not merely the uncertainty in the estimated mean response.

Confidence Interval for the Mean Response

The estimated standard error of the mean response at \(x_0\) is\[ SE(\hat{y}_0) = s_e \sqrt{ \frac{1}{n} + \frac{(x_0-\bar{x})^2} {\sum_{i=1}^{n}(x_i-\bar{x})^2} }. \]

A confidence interval takes the form\[ \hat{y}_0 \pm t^*SE(\hat{y}_0). \]

This interval estimates the average response at \(x_0\).

It is narrowest near \(\bar{x}\) and becomes wider as \(x_0\) moves away from the center of the observed predictor values.

Prediction Interval for an Individual

For a new individual at \(x_0\), the prediction standard error is\[ SE_{\text{pred}} = s_e \sqrt{ 1+ \frac{1}{n} + \frac{(x_0-\bar{x})^2} {\sum_{i=1}^{n}(x_i-\bar{x})^2} }. \]

A prediction interval takes the form\[ \hat{y}_0 \pm t^*SE_{\text{pred}}. \]

The extra 1 inside the square root represents individual variation around the regression line.

Consequently,\[ SE_{\text{pred}}>SE(\hat{y}_0). \]

A prediction interval for one student is always wider than a confidence interval for the average score of many comparable students.

The Role of Correlation

The residual scale\[ s_y\sqrt{1-r^2} \]

shows how correlation affects prediction.

If \(r=0\)

Then\[ s_y\sqrt{1-r^2}=s_y. \]

The regression line does not reduce the response uncertainty relative to predicting from \(\bar{y}\) alone.

If \(|r|\) Is Close to 1

Then\[ 1-r^2 \]

is close to zero, so the residual spread is small. The predictor accounts for much of the sample variation in \(Y\).

The Sign Does Not Affect Residual Spread

Both \(r=0.8\) and \(r=-0.8\) produce\[ r^2=0.64. \]

They have slopes in opposite directions, but the same proportion of variation accounted for by the simple linear model.

What \(1-r^2\) Means

In simple ordinary least-squares regression with an intercept,\[ R^2=r^2. \]

Therefore,\[ 1-r^2 \]

is the fraction of total sample variation remaining in the squared residuals.

For \(r=0.67\),\[ r^2=0.4489. \]

Thus, approximately 44.9% of the sample variation in final scores is accounted for by the linear relationship with midterm scores, while approximately\[ 1-0.4489=0.5511 \]

or 55.1% remains in the residual variation.

This does not mean that midterm scores causally determine 44.9% of final-score variation. It is a mathematical decomposition of variation in the fitted sample.

Diagnostics Before Using Conditional Normal Approximation

Before calculating conditional probabilities, examine whether the regression assumptions are reasonable.

Scatterplot

Look for:

• Approximate linearity
• A reasonably consistent vertical spread
• Outliers
• Clusters
• Gaps
• Unsupported extrapolation

Residual Plot

Residuals should generally form a horizontal band around zero. Curvature suggests an inadequate mean specification, while a fan shape suggests heteroscedasticity.

Normal Q–Q Plot

A Q–Q plot helps determine whether the residual distribution is approximately normal. Major curvature or extreme tail departures weaken exact normal probability calculations.

Influence Diagnostics

Check whether a small number of observations dominate the line using:

• Leverage
• Studentized residuals
• Cook’s distance
• Case-deletion analysis

Study Design

Even an excellent-looking regression model does not correct biased sampling, confounding, measurement error, or dependence among observations.

What If the Conditional Distribution Is Not Normal?

If the residuals are not approximately normal, possible alternatives include:

• Transforming the response
• Using a generalized linear model
• Modeling conditional quantiles
• Applying bootstrap methods
• Using robust regression
• Estimating the conditional distribution nonparametrically
• Using heteroscedastic models when spread changes with \(x\)

The fitted regression line can still estimate a conditional mean without normal errors, but normal-curve probability calculations may be inaccurate.

Key Takeaway

A regression line estimates the center of the response distribution at a given predictor value:\[ E(Y\mid X=x)\approx\hat{y}(x). \]

Under a linear, constant-variance, approximately normal conditional model,\[ Y\mid X=x \approx N\left( \hat{y}(x), s_e^2 \right), \]

with the simplified residual scale\[ s_e\approx s_y\sqrt{1-r^2}. \]

For the exam-score example,\[ \hat{y}=62.5, \qquad r=0.67, \qquad s_y=11.8, \]

so\[ s_e\approx8.76. \]

The standardized value for a final score of 60 is approximately\[ z=-0.285, \]

giving\[ P(Y>60\mid X\approx41)\approx0.61. \]

Thus, the model estimates that about 61% of students with midterm scores around 41 will score above 60 on the final. This conclusion depends on the adequacy of the linearity, constant-variance, normal-error, and independence assumptions.