Regression models often involve many explanatory variables, some of which may be irrelevant or only weakly associated with the outcome. When the number of predictors is large relative to the sample size, classical regression becomes unstable: estimates become noisy, variables may appear important by chance, and collinearity can lead to large variances in coefficient estimates. This section discusses how regularization and Bayesian approaches help stabilize estimation, and how methods like lasso fit into the Bayesian framework.

1. Classical variable selection: stepwise regression and subset selection

Traditional non-Bayesian procedures—such as forward selection, backward elimination, or choosing the “best” subset of predictors—select a model by including or excluding variables.

Mathematically:

  • Excluding a variable = setting its coefficient exactly equal to zero.
  • This corresponds to a highly irregular prior that assigns positive probability mass to sets of coefficient vectors lying entirely on low-dimensional hyperplanes.

Geometric intuition

Imagine β-space as a 3D room:

  • The prior used in classical stepwise selection assigns some probability that the “true β” lies:
    • in the room (all coefficients nonzero),
    • on the floor,
    • on walls,
    • on edges,
    • or even in a corner (multiple coefficients exactly zero).

Thus, model selection assumes structural sparsity with sharp boundaries, which is unrealistic.

Problem with classical selection

The model chosen is heavily affected by sampling noise, meaning:

  • Real, important predictors might be dropped because their estimated effects look insignificant due to random variation.
  • Irrelevant variables may be selected by chance.
  • Estimates for included coefficients often look more certain than they truly are.

2. Bayesian approach: continuous shrinkage instead of exact zeros

Rather than assuming coefficients are exactly zero or nonzero, the Bayesian approach favors continuous prior distributions with a peak at zero and heavy tails.

A common default choice:

  • Zero-centered distributions such as the t-distribution or Cauchy distribution.
  • They say:
    “Most predictors probably have small effects, but large effects are possible if supported by the data.”

This avoids the unnatural assumption that a coefficient is exactly zero.

3. Importance of scaling predictors

In classical regression with a uniform prior:

  • Linear transformations of predictors (e.g., changing units from dollars to thousands of dollars) do not affect inferences.

In Bayesian regression with informative priors:

  • Scaling does matter, because the prior distribution is applied directly to the β coefficients.

For example:

  • If economic predictors are measured in many different units, their β coefficients will have inherently different scales.
  • Applying the same prior distribution across them is inappropriate unless the predictors are standardized.

Thus:

  • Before applying a prior for regularization, predictors should be transformed to comparable scales (e.g., standardized to range −1 to 1).

4. What is regularization?

Regularization refers to statistical methods that stabilize parameter estimates, particularly when:

  • There are many predictors
  • Predictors are noisy or weakly informative
  • Sample size is small or moderate

Bayesian regularization is governed by:

  1. Location and scale of the prior
    • Narrower priors shrink estimates more.
  2. Distributional shape of the prior
    • Normal → shrinkage proportional to estimate
    • Laplace (double exponential) → constant shrinkage → enables sparsity
    • Cauchy → strong shrinkage near zero, weak shrinkage for large coefficients
  3. Summary measure of posterior
    • Posterior mean or median → soft shrinkage
    • Posterior mode → more aggressive shrinkage, may resemble classical penalized likelihood

5. Lasso as a Bayesian method

Lasso is widely used in classical statistics but has a Bayesian interpretation.

Bayesian form

Lasso corresponds to:

  • A double-exponential (Laplace) prior on each coefficient:

$p(\beta_j) \propto \exp(-\lambda |\beta_j|)$

With the normal likelihood of linear regression:

  • The posterior is sharper at zero than with a normal prior.
  • If the posterior mode is used as the estimate, some coefficients become exactly zero.

Thus:

  • Lasso = Laplace prior + posterior mode estimation.

Why lasso works

  • The sharp peak at zero pushes small coefficients to exactly zero.
  • Good for variable selection in problems with many predictors and moderate noise.
  • More transparent and algorithmic than ad-hoc stepwise regression.
  • Allows including many predictors without unstable estimates.

6. Limitations of lasso from a Bayesian viewpoint

Although useful, lasso is not fully consistent with Bayesian modeling philosophy:

  1. The posterior mode (lasso’s estimate) discards posterior uncertainty.
  2. The Laplace prior implies coefficients can be exactly zero:
    • Unrealistic for most scientific applications.
  3. A single prior (same λ) for all predictors is rarely appropriate.
  4. If full Bayesian inference is used instead of mode:
    • Coefficients are not set to zero; they are only partially shrunk.
    • Heavy-tailed priors (e.g., t-distribution) may work better.

Conclusion

Lasso is a helpful regularizing tool, not a principled Bayesian model of sparsity.

7. Why regularization is necessary

Regularization is unavoidable:

  • With many predictors, classical least squares becomes unstable.
  • Bayesian priors impose structure that improves stability.
  • Even if coefficients are believed to be non-zero, regularization helps prevent overfitting and improves predictive performance.

Methods like lasso offer:

  • Clear algorithms
  • Transparent selection
  • Ability to keep more predictors without causing computational instability

Even though full Bayesian shrinkage methods may be conceptually preferable, lasso remains a strong practical step forward.

Summary

Regularization and dimension reduction help stabilize regression when dealing with many explanatory variables. Classical subset selection corresponds to unrealistic priors that place probability on exact zero coefficients. Bayesian methods prefer continuous shrinkage priors, which acknowledge that while most predictors likely have small effects, large effects are still possible. Scaling predictors becomes important when using informative priors. Regularization depends on the prior’s scale, shape, and how posterior results are summarized. Lasso is a special case of Bayesian regularization that uses a Laplace prior combined with posterior mode estimation, enabling coefficients to be exactly zero. While useful and more transparent than classical model selection, lasso has conceptual limitations from a Bayesian standpoint, but still represents an important practical improvement for high-dimensional regression settings.