Feature Importance in Linear Regression

1) Coefficient of Determination: What $R^2$ Measures

To discuss feature importance, it is necessary to first understand the Coefficient of Determination, denoted by $R^2$.

The coefficient of determination is defined as

$R^2 = 1 – \dfrac{SSE}{SST}$,

where:

• $SST = \sum_{i=1}^{n} (y_i – \bar y)^2$ is the total sum of squares, and
• $SSE = \sum_{i=1}^{n} (y_i – \hat y_i)^2$ is the sum of squared errors, also called the residual sum of squares.

The interpretation of these quantities is straightforward:

• $SST$ measures the overall variability of the response variable around its mean.
• $SSE$ measures the remaining variability after fitting the regression model.

Because $SSE \le SST$, the coefficient of determination always satisfies
$0 \le R^2 \le 1$.

---

2) Interpreting Extreme Values of $R^2$

The value of $R^2$ has clear boundary interpretations:

• If a regression model fits the data perfectly, then all residuals satisfy $y_i – \hat y_i = 0$. In this case, $SSE = 0$ and therefore $R^2 = 1$.
• If a regression model includes only an intercept and no explanatory variables, then every predicted value equals the sample mean $\bar y$. In this case, $SSE = SST$ and therefore $R^2 = 0$.

Because of this, $R^2$ is commonly used as a measure of how effective a model is at explaining the variability of the response variable.

---

3) Example: Coefficient of Determination in the Chicago Taxi Model

Consider the regression model

Trip_Payment ~ Intercept + Trip_Miles + Payment_Method.

For this model:

• $SST = 1139.232375$
• $SSE = 28.460827$

The resulting coefficient of determination is $R^2 = 0.9750$.

This means that 97.50% of the variability in Trip_Payment is explained by the model, using only trip distance and payment method. The unexplained variability is relatively small.

If Trip_Minutes is added to the model, the coefficient of determination increases to $R^2 = 0.9794$. This illustrates an important property of $R^2$:

Adding predictors to a linear regression model will never decrease $R^2$.

However, this does not automatically mean the new predictor is necessary or important.

---

4) The Principle of Parsimony and the Limits of $R^2$

Model building always involves a trade-off between two competing goals:

1. Explaining as much variability of the response as possible, which pushes $R^2$ upward.
2. Keeping the model simple, interpretable, and easy to deploy.

Because $R^2$ always increases when predictors are added, it cannot by itself determine which features should be included. This is why statistical tests (such as the F test in forward selection) and interpretability considerations are essential.

---

5) Are All Features Equally Predictive?

Forward selection includes only predictors that are statistically significant contributors to the response. Together, these predictors maximize $R^2$ under the selection criteria.

However, this raises deeper questions:

• Are the selected features truly the most important ones?
• Are excluded features completely uninformative, or are they redundant due to correlation with other features?
• How much does each feature contribute to the overall explanatory power of the model?

To answer these questions, a framework is needed that can fairly assign credit to each feature, even when predictors are correlated. This leads naturally to game theory.

---

6) Coalition Games and Utility

Game theory studies how self-interested players make decisions and how their interactions determine outcomes.

In a coalitional (or cooperative) game:

• Players form groups called coalitions.
• Each coalition receives a payoff.
• The payoff of the empty coalition is zero.
• Larger coalitions generally receive larger payoffs.
• The central question is how to fairly allocate the payoff of the grand coalition (the coalition containing all players).

Each player’s benefit is measured using a utility value, which is a numeric representation of preference or gain. A utility function maps coalitions to numeric payoffs.

---

7) Shapley Value: A Fair Measure of Importance

The Shapley value provides a principled way to distribute the payoff of the grand coalition among individual players.

Its key idea is fairness across all possible coalition formation orders:

• Each player is assumed to be equally likely to join a coalition at any position.
• All possible orders of players are considered.
• For each order, the marginal contribution of a player is computed as the increase in payoff caused by that player joining.
• The Shapley value is the average marginal contribution across all possible orders.

This approach has a major advantage over sequential methods such as forward selection:
importance does not depend on the order in which features enter the model.

Even when features are correlated or aliased, the Shapley value provides an absolute measure of importance.

---

8) Intuition Through the Taxi Ride Example

Consider three students—Amy (A), Betty (B), and Cathy (C)—sharing a taxi ride, with a total fare of $30. Each student has a different solo fare and different overlaps with others.

By examining all possible orders in which the students could join the taxi and computing each student’s marginal contribution, the average contributions turn out to be:

• Amy: $3
• Betty: $7.5
• Cathy: $19.5

These values sum to $30, exactly matching the total fare. This illustrates the core idea of the Shapley value: the total payoff is fully and fairly allocated among participants.

---

9) Linear Regression as a Coalitional Game

A linear regression model can be interpreted in exactly the same way:

• The players are the features.
• A coalition is any subset of features.
• The payoff of a coalition is the coefficient of determination $R^2$ achieved by a model using those features.
• The grand coalition includes all candidate features.
• A feature’s importance is its fair share of the grand coalition’s $R^2$.

By computing $R^2$ for all possible subsets of features and applying the Shapley value formula, the contribution of each feature to explanatory power can be quantified.

---

10) Shapley Values in the Chicago Taxi Regression

Using Payment_Method, Trip_Minutes, and Trip_Miles as predictors, the full model has
$R^2 = 0.9794$.

The Shapley value decomposition yields:

• Trip_Miles: 0.6415 (65.50%)
• Payment_Method: 0.2902 (29.63%)
• Trip_Minutes: 0.0477 (4.87%)

These values have several important interpretations:

• Trip_Miles is by far the most important predictor, explaining the majority of variability.
• Payment_Method contributes substantially, consistent with differences in tipping behavior.
• Trip_Minutes contributes relatively little once distance and payment method are accounted for.

Notably, the Shapley values sum exactly to the model’s $R^2$, reinforcing their interpretation as a fair allocation of explained variability.

---

11) Feature Importance Revisited

When Trip_Payment is modeled using Payment_Method, Trip_Minutes, and Trip_Miles, the regression explains 97.94% of the variability in the response.

That explained variability can be meaningfully decomposed by feature:

1. Trip_Miles explains approximately 64.15% of the variability.
2. Payment_Method explains approximately 29.02%.
3. Trip_Minutes explains the remaining 4.77%.

This decomposition provides a clear, order-independent understanding of feature importance, complementing traditional model selection methods such as forward selection.