Residual Plots, Nonlinearity, Heteroscedasticity, and Transformations

Residuals measure the differences between observed and predicted response values. Examining their behavior is one of the most important ways to determine whether a regression model adequately represents the data.

Observed and Predicted Values

In simple linear regression, the fitted model is\[ \hat{y}_i=b_0+b_1x_i, \]

where:

• \(x_i\) is the explanatory-variable value for observation \(i\).
• \(y_i\) is its observed response.
• \(\hat{y}_i\) is its predicted response.
• \(b_0\) is the fitted intercept.
• \(b_1\) is the fitted slope.

The fitted value \(\hat{y}_i\) is the point on the regression line corresponding to \(x_i\).

Residuals

The residual for observation \(i\) is\[ \boxed{e_i=y_i-\hat{y}_i}. \]

It is the vertical difference between the observed point and the fitted regression line.

• If \(e_i>0\), the observation lies above the line.
• If \(e_i<0\), the observation lies below the line.
• If \(e_i=0\), the observed and predicted responses are identical.

For example, suppose a model predicts a final-exam score of 82, while the observed score is 87. The residual is\[ e_i=87-82=5. \]

If another student’s observed score is 76 while the model predicts 81, then\[ e_i=76-81=-5. \]

A positive residual means underprediction; a negative residual means overprediction.

Errors Versus Residuals

The theoretical regression model is often written as\[ Y_i=\beta_0+\beta_1x_i+\varepsilon_i, \]

where \(\varepsilon_i\) is the unobserved random error.

After fitting the model, we obtain\[ \hat{Y}_i=b_0+b_1x_i \]

and calculate\[ e_i=Y_i-\hat{Y}_i. \]

Errors and residuals are related but not identical:

• The error \(\varepsilon_i\) is an unobserved deviation from the true population regression function.
• The residual \(e_i\) is an observed deviation from the fitted sample regression line.

Residuals are used as estimates of the unobserved errors.

The Residual Plot

A residual plot is a scatter plot with:

• Residuals \(e_i\) on the vertical axis
• Predictor values \(x_i\), or fitted values \(\hat{y}_i\), on the horizontal axis
• A horizontal reference line at zero

Both versions are useful:\[ (x_i,e_i) \]

or\[ (\hat{y}_i,e_i). \]

A residual-versus-fitted plot is especially common in multiple regression, where there is more than one predictor.

What a Good Residual Plot Looks Like

For a well-specified linear regression model, the residual plot should generally resemble a random horizontal band around zero.

Desirable features include:

• No systematic curvature
• Approximately constant vertical spread
• No distinct clusters
• No strong time pattern
• No extreme isolated residuals
• Rough balance above and below zero

“No pattern” does not mean the residuals must look perfectly uniform. Some irregularity is expected because residuals are random.

If the regression includes an intercept, the ordinary least-squares residuals satisfy\[ \sum_{i=1}^{n}e_i=0. \]

Thus, their sample average is zero:\[ \bar{e}=0. \]

This mathematical property alone does not establish that the model is appropriate. The residuals can average to zero while still exhibiting curvature, unequal variance, dependence, or outliers.

Example: Midterm and Final Scores

Suppose midterm scores are used to predict final-exam scores:\[ \widehat{\text{Final}} = b_0+b_1(\text{Midterm}). \]

If the scatterplot follows an approximately linear pattern and the residual plot shows an unstructured horizontal band around zero, a linear model may be reasonable.

The residual plot allows us to evaluate the model more directly than the original scatterplot because it removes the fitted linear trend. What remains should look like random variation if the model has captured the systematic relationship adequately.

Curvature in a Residual Plot

Suppose income is regressed on years of education:\[ \widehat{\text{Income}} = b_0+b_1(\text{Education}). \]

If the actual relationship is curved, a straight line may systematically:

• Overpredict at low education levels
• Underpredict in the middle
• Overpredict again at high education levels

The residual plot would show a curved pattern rather than random scatter.

For example, it might resemble:\[ +\quad+\quad0\quad-\quad-\quad0\quad+\quad+ \]

as \(x\) increases.

This indicates that the conditional mean of \(Y\) is not adequately represented by a straight line.

The correct conclusion is not that regression in general is impossible. Instead:

The current linear specification is inadequate for the observed relationship.

Possible alternatives include:

• Transforming \(x\)
• Transforming \(y\)
• Including polynomial terms
• Using regression splines
• Using a generalized additive model
• Fitting a model motivated by subject-matter theory

Polynomial Regression

A curved relationship can sometimes be modeled by adding powers of \(x\):\[ Y = \beta_0+\beta_1x+\beta_2x^2+\varepsilon. \]

Although this model is curved as a function of \(x\), it is still called a linear regression model because it is linear in the coefficients \(\beta_0,\beta_1,\beta_2\).

A cubic model would be\[ Y = \beta_0+\beta_1x+\beta_2x^2+\beta_3x^3+\varepsilon. \]

Polynomial terms should be chosen carefully. High-degree polynomials can behave erratically, especially near the edges of the data and during extrapolation.

Transforming the Response

If income is strongly right-skewed or its variability increases with its level, transforming the response may improve the model.

Common transformations include:\[ Y^*=\sqrt{Y} \]

and\[ Y^*=\log(Y). \]

The regression then becomes\[ Y^* = \beta_0+\beta_1x+\varepsilon. \]

For a log transformation,\[ \log(Y) = \beta_0+\beta_1x+\varepsilon. \]

The transformation may:

• Reduce right skew
• Stabilize variance
• Reduce the effect of extreme observations
• Turn multiplicative relationships into additive relationships
• Make a curved relationship more nearly linear

Interpreting a Log-Response Model

Suppose\[ \log(Y)=\beta_0+\beta_1x+\varepsilon. \]

Then increasing \(x\) by one unit changes the predicted log response by \(\beta_1\). On the original scale, the fitted median or geometric-mean response is multiplied by\[ e^{\beta_1}. \]

The exact percentage change associated with a one-unit increase in \(x\) is\[ 100\left(e^{\beta_1}-1\right)\%. \]

When \(|\beta_1|\) is small, this is approximately\[ 100\beta_1\%. \]

For example, if\[ \beta_1=0.04, \]

then the exact multiplicative change is\[ e^{0.04}\approx1.0408, \]

corresponding to an increase of approximately 4.08%.

Transforming the Predictor

A transformation of \(x\) may be useful when the response changes rapidly at low values of \(x\) and more slowly at high values.

Possible models include\[ Y=\beta_0+\beta_1\log(X)+\varepsilon \]

or\[ Y=\beta_0+\beta_1\sqrt{X}+\varepsilon. \]

A logarithmic predictor model means that equal proportional changes in \(X\), rather than equal absolute changes, are associated with equal changes in \(Y\).

For example, moving from \(X=10\) to \(X=20\) has the same change in \(\log(X)\) as moving from \(X=20\) to \(X=40\):\[ \log(20)-\log(10) = \log(40)-\log(20) = \log(2). \]

Log–Log Regression

If both variables are positive, a log–log model may be useful:\[ \log(Y) = \beta_0+\beta_1\log(X)+\varepsilon. \]

Here, \(\beta_1\) is an elasticity. A 1% increase in \(X\) is associated with an approximate \(\beta_1\)% change in \(Y\), holding other variables fixed.

This form is useful for multiplicative and scaling relationships.

Back-Transformation Requires Care

After fitting\[ \log(Y)=\beta_0+\beta_1x+\varepsilon, \]

it may be tempting to predict the original response as\[ \hat{Y} = \exp(\hat{\beta}_0+\hat{\beta}_1x). \]

This quantity estimates a conditional median under common log-error assumptions, not necessarily the conditional mean.

Because the exponential function is nonlinear,\[ E(e^Z)\neq e^{E(Z)} \]

in general.

If\[ \varepsilon\sim N(0,\sigma^2), \]

then\[ E(Y\mid X=x) = \exp\left( \beta_0+\beta_1x+\frac{\sigma^2}{2} \right). \]

A retransformation adjustment may therefore be needed when estimating the mean response on the original scale. Alternatives include:

• A lognormal correction
• Duan’s smearing estimator
• Direct modeling with an appropriate generalized linear model

Transforming back is not always as simple as exponentiating the fitted value.

Restrictions on Transformations

A logarithm requires positive inputs:\[ Y>0. \]

If the data contain zeros or negative values, \(\log(Y)\) is undefined. Adding an arbitrary constant can materially change the model and should not be done without justification.

A square-root transformation generally requires\[ Y\geq0. \]

Transformations should be selected based on the data-generating process and interpretability, not solely to make a diagnostic plot look attractive.

Heteroscedasticity

A regression model is homoscedastic when the conditional error variance is constant:\[ \operatorname{Var}(\varepsilon_i\mid X_i=x) = \sigma^2 \]

for every relevant value of \(x\).

Heteroscedasticity occurs when this variance changes with \(x\):\[ \operatorname{Var}(\varepsilon_i\mid X_i=x) \]

is not constant.

A common residual pattern is a fan or funnel shape:

• Small residual spread at low fitted values
• Large residual spread at high fitted values

The relationship may still be approximately linear in its conditional mean, but the uncertainty changes across the predictor range.

Why Heteroscedasticity Matters

Under appropriate exogeneity conditions, heteroscedasticity does not by itself make ordinary least-squares coefficient estimates biased. However, it can cause several problems:

• Conventional standard errors may be incorrect.
• Confidence intervals may have incorrect coverage.
• Hypothesis tests may be misleading.
• Ordinary least squares may be inefficient.
• Prediction uncertainty may vary substantially across observations.

Thus, a linear mean relationship can remain useful even when constant variance fails, but the inference procedure should be adjusted.

Responding to Heteroscedasticity

Possible approaches include:

Transforming the Response

A log or square-root transformation can sometimes stabilize variance when the response’s spread increases with its level.

Heteroscedasticity-Consistent Standard Errors

Robust standard errors modify the estimated coefficient covariance matrix without changing the fitted coefficients.

These are often called:

• Huber–White standard errors
• Sandwich standard errors
• HC standard errors

Common versions include HC0, HC1, HC2, and HC3. HC3 is often useful in smaller samples or when leverage is a concern.

Weighted Least Squares

If the error variance structure is known or can be modeled, observations can be weighted inversely to their variances:\[ w_i\propto\frac{1}{\operatorname{Var}(\varepsilon_i)}. \]

Observations with greater uncertainty receive less weight.

Modeling the Variance Directly

Some models explicitly specify how variance changes with the mean or predictors. Generalized linear models are useful when the response distribution naturally implies nonconstant variance.

Formal Heteroscedasticity Tests

Diagnostic tests include:

• Breusch–Pagan test
• White test
• Goldfeld–Quandt test

These tests can supplement a residual plot, but they should not replace graphical examination.

With large samples, formal tests may detect small and practically unimportant deviations. With small samples, they may have limited power. The magnitude and consequences of heteroscedasticity matter more than a test result alone.

Nonlinearity and Heteroscedasticity Can Occur Together

A residual plot can display both:

• A curved center
• Changing vertical spread

Transforming \(y\) may stabilize variance but introduce or reveal nonlinearity. Transforming \(x\) may improve linearity without correcting unequal variance.

The model may therefore require transformations of both variables:\[ \log(Y) = \beta_0+\beta_1\log(X)+\varepsilon. \]

After every transformation or model revision, residual diagnostics should be repeated.

A transformation is not the end of model checking. It creates a new model with assumptions that must also be evaluated.

Florida Election Example

County-level election data from Florida’s 2000 presidential election provide a useful example.

Suppose the explanatory variable is the number of votes received by George W. Bush in each county, and the response is the number received by Pat Buchanan. Excluding Palm Beach County initially, the original-scale scatterplot may appear approximately linear.

However, the residual plot can reveal increasing variability as county vote totals grow. Larger counties naturally have more opportunity for large absolute deviations, creating a fan-shaped pattern.

A model on the original count scale might be\[ \widehat{\text{Buchanan votes}} = b_0+b_1(\text{Bush votes}). \]

A log–log model instead uses\[ \log(\text{Buchanan votes}) = \beta_0+ \beta_1\log(\text{Bush votes}) +\varepsilon. \]

This transformation can:

• Compress the scale of large counties
• Reduce right skew
• Stabilize variance
• Convert a multiplicative relationship into an additive one
• Produce a more uniform residual band

A satisfactory residual plot after transformation provides evidence that the transformed specification fits better, although other assumptions must still be checked.

If any county has zero votes for a candidate, the ordinary logarithm cannot be applied directly. That issue must be addressed explicitly rather than ignored.

Additional Patterns in Residual Plots

Residual plots can reveal more than curvature and heteroscedasticity.

Clusters

Separate clusters may indicate unmodeled groups, such as:

• Regions
• Schools
• Demographic categories
• Experimental conditions

A group indicator or hierarchical model may be needed.

Horizontal Bands

Horizontal bands can arise when the response is discrete or rounded. This is not automatically a problem, but it may suggest that a continuous normal-error model is inappropriate.

Isolated Points

A point far above or below the zero line may have a large residual and should be investigated as a possible response outlier.

Time Patterns

When observations are ordered in time, runs or cycles in the residuals may indicate autocorrelation, trends, seasonality, or structural change.

Missing Predictors

A pattern in residuals plotted against a variable not included in the model suggests that the variable may contain additional predictive information.

Residual Normality

A residual-versus-fitted plot is not the best tool for judging normality. Normality is commonly evaluated using:

• A normal quantile–quantile plot
• A histogram of residuals
• Subject-matter knowledge
• Sensitivity analysis

Normal residuals are not required for ordinary least squares to define or estimate a line. Normality is mainly relevant for exact small-sample confidence intervals and tests under the classical model.

For large samples, approximate inference may remain available through asymptotic methods, especially when robust standard errors are used.

Residual Dependence

A residual plot against fitted values may appear satisfactory even if residuals are correlated.

For observations collected over time, residuals should also be plotted against:

• Time
• Observation order
• Lagged residuals

For spatial or clustered data, dependence should be assessed using the relevant grouping or geographic structure.

Ignoring dependence often causes the conventional standard errors to be too small.

Residuals and Leverage

Residual plots are effective at finding points with large vertical deviations. They are less effective at identifying high-leverage or influential observations.

A high-leverage point may pull the regression line toward itself, producing a small residual despite having a major effect on the fitted slope.

A complete diagnostic analysis should therefore consider:

• Studentized residuals
• Leverage values
• Cook’s distance
• DFBETAS
• Case-deletion results

A Practical Diagnostic Workflow

1. Plot the Original Data

Examine the scatterplot before fitting the model. Look for curvature, changing spread, clusters, gaps, and unusual observations.

2. Fit a Theory-Guided Model

Begin with a model supported by the scientific or practical context.

3. Plot Residuals Against Fitted Values

Look for:

• Curvature
• Fan shapes
• Clusters
• Extreme residuals

4. Plot Residuals Against Each Predictor

A residual-versus-fitted plot can conceal predictor-specific patterns in multiple regression.

5. Check Time, Group, or Spatial Structure

Assess whether the independence assumption is reasonable.

6. Revise the Model if Necessary

Possible revisions include transformations, polynomial terms, splines, additional predictors, robust inference, or a different model family.

7. Repeat the Diagnostics

Every revised model must be checked again.

8. Validate Predictive Performance

If prediction is the goal, assess performance using new data, a validation set, or cross-validation.

Key Takeaway

A residual is the difference between an observed response and its fitted value:\[ e_i=y_i-\hat{y}_i. \]

A suitable linear model generally produces a residual plot that resembles an unstructured horizontal band around zero.

• Curvature suggests that the conditional mean has not been modeled adequately.
• A fan shape suggests heteroscedasticity.
• Isolated residuals may indicate response outliers.
• Patterns over time or groups may indicate dependence or omitted structure.

Transformations of \(x\) or \(y\) can sometimes improve linearity and stabilize variance, but they change the model’s interpretation and may require careful back-transformation. When the mean relationship is appropriate but variance is unequal, robust standard errors or weighted least squares may be preferable to transforming the data.

Residual plots are indispensable diagnostic tools, but they should be combined with leverage, influence, dependence, and predictive-validation checks before a regression model is considered reliable.