Forward selection is a stepwise regression technique used in regression analysis to build a model incrementally. The procedure starts with the simplest possible model—typically containing only an intercept—and then adds predictor variables one at a time. At each step, the variable that provides the best improvement to the model is selected and included.

The key idea behind forward selection is controlled model growth: instead of starting with all possible predictors, the model is allowed to grow only when there is clear evidence that adding a variable improves performance.

Forward Selection in the Context of Stepwise Regression

Forward selection is one of the two most commonly used stepwise regression methods. The other is backward elimination, which operates in the opposite direction. Backward elimination begins with a full model that includes all candidate predictors and removes variables one by one if they do not contribute meaningfully.

In contrast, forward selection begins with no predictors (other than the intercept) and builds complexity gradually. This makes forward selection particularly useful when the number of potential predictors is large or when interpretability and parsimony are important.

General Procedure of Forward Selection

The forward selection process typically unfolds as follows.

The procedure starts with a model that includes only the intercept term. All candidate predictor variables are then evaluated individually by fitting separate models, each adding exactly one predictor to the intercept-only model. Among these candidates, the variable that leads to the greatest improvement—according to a predefined criterion—is selected.

Once this variable is added, the process repeats. At each subsequent step, the remaining candidate variables are tested by adding them one at a time to the current model. Again, the variable that produces the strongest improvement is chosen.

This iterative process continues until adding any remaining variable no longer improves the model according to the chosen criterion. At that point, the algorithm stops, and the current model is returned as the final model.

Criteria for Adding Variables

The definition of “best improvement” depends on the criterion chosen in advance. Common choices include:

  • Statistical significance measures, such as the smallest p-value
  • Likelihood-based measures, such as deviance reduction
  • Information criteria, such as AIC or BIC
  • Predictive performance metrics, such as cross-validated error

The same criterion must be applied consistently at every step to ensure a coherent selection process.

Overfitting Considerations

A known drawback of stepwise regression methods, including forward selection, is their tendency toward overfitting. Because variables are selected based on performance on the observed data, the resulting model may fit the training data extremely well while performing poorly on new or unseen data.

Overfitting often manifests as a model that appears highly accurate within the sample but generalizes poorly. For this reason, forward selection should be used with strict entry criteria and, whenever possible, validated using out-of-sample testing or cross-validation.

Summary

Forward selection is a systematic approach to model building that:

  • Starts with a minimal model
  • Adds variables one at a time
  • Retains a variable only if it meaningfully improves the model
  • Stops automatically when no further improvement is achieved

When applied carefully and evaluated with appropriate validation techniques, forward selection can produce interpretable and parsimonious regression models while avoiding unnecessary complexity.