Regression Models

A regression model predicts a numeric outcome from one or more inputs. The family is enormous — linear models, regularized variants, trees and forests, neural networks — and two things are true of all of them and decide most of what goes wrong in practice.

• The loss function decides which summary of the outcome you are estimating.
• Fitting well is not the same as identifying an effect.

The loss chooses the target

This is stated compactly in the forecasting literature and generalizes to regression: “a forecast method that minimizes the MAE will lead to forecasts of the median, while minimizing the RMSE will lead to forecasts of the mean.”

So squared error — the default almost everywhere — estimates a conditional mean, and absolute error estimates a conditional median. For a skewed outcome those differ substantially, and the choice is usually made by whatever the library defaults to rather than by anyone deciding. Where a specific position in the distribution is what the decision needs, model that instead: see quantile regression.

The same logic governs evaluation. A model trained on squared error and scored by MAE is being judged against a target it was not aiming at, and the comparison will favor a different model for reasons that have nothing to do with quality. Keep the training loss and the evaluation metric aligned with the quantity the decision uses — see forecast error.

Prediction and intervention are different questions

A regression describes association in the data it was fitted to. Reading a coefficient as “what would happen if we changed this input” is a causal claim, and it needs something the fit does not supply: “causal inference from observational data requires two elements: data and identifiability conditions.”

Adding covariates is the move people make in the belief that it converts the first row into the second. It does not: exchangeability has to hold conditional on “known measured covariates,” so a confounder you never measured leaves the estimate biased — and the output, including the coefficient’s standard error, looks identical either way. See causal effect and causal inference. Randomization is what removes the need to assume: “in ideal randomized experiments the identifiability conditions hold by design” — see A/B testing.

Reading the output

Three habits, each addressing a common failure.

• Judge on held-out data, not on fit. For nested, unpenalized OLS fits with an intercept on the same sample, training R² cannot decrease as predictors are added, including irrelevant ones. That property does not extend to test R² or general regularized fits, and training improvement alone does not establish a better predictive model. Use error on data the model has not seen.
• Report uncertainty, not just a prediction. A range for an individual future outcome is a prediction interval and is wider than an interval for the mean — see confidence intervals.
• Check the residuals rather than the summary statistic. A single fit number is consistent with both a well-behaved model and one missing an obvious pattern, and only the plot tells them apart.

One boundary worth naming: a coefficient’s standard error describes sampling variability under the model, so it narrows with more data whether or not the model is right. A very precise estimate of a mis-specified or confounded relationship is precisely wrong, and it looks more convincing than a wide interval around a sound one. See standard error.

References

• Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), §2.8 “Autocorrelation”
• §5.8 “Evaluating point forecast accuracy”
• Hernán & Robins, Causal Inference: What If