Multiple Linear Regression

1) What a Multiple Linear Regression Model Is

The goal is simple: we want to explain or predict a numeric outcome using several numeric factors at the same time.

Suppose we observe a response value $y_i$ for each observation $i$. At the same time, we also measure $p$ numeric features for that observation. We collect $n$ such observations in total. The question is how these features jointly relate to the outcome.

The multiple linear regression model assumes that the expected value of $y_i$ can be written as a weighted sum of the predictors, plus a random error term:

$y_i = \sum_{j=0}^{p} x_{ij} b_j + \varepsilon_i$, for $i = 1, \dots, n$.

Here is how to interpret each part:

• $x_{ij}$ is the value of feature $j$ for observation $i$.
• $b_j$ is the regression coefficient for feature $j$. This is an unknown quantity that we want to estimate from the data.
• We include an intercept by setting $x_{i0} = 1$. This allows the model to have a baseline level even when all predictors are zero.
• $\varepsilon_i$ represents everything that affects $y_i$ but is not explained by the predictors in the model.

A key assumption in the classical linear regression framework is that the error terms satisfy $\varepsilon_i \sim \text{iid } N(0, \sigma^2)$.

This assumption has an important conceptual meaning. It says that the unexplained part of the outcome:

• has no systematic bias (it is centered at zero),
• behaves independently across observations,
• and has the same variability everywhere.

What this model is really saying is not that every data point lies exactly on a line or plane. Instead, it says that the average behavior of $y$ changes linearly with the predictors, while individual observations deviate from that average because of noise or unobserved factors.

---

2) Ordinary Least Squares (OLS): What We Are Minimizing

Once the model is specified, the next question is how to estimate the unknown coefficients $b_0, \dots, b_p$.

Ordinary Least Squares answers this question by comparing what the model predicts to what we actually observe. For each observation $i$, the model produces a predicted value $\hat y_i$. The difference between the observed value and the predicted value is called the residual:

$e_i = y_i – \hat y_i$.

OLS chooses the coefficients that make these residuals as small as possible overall, by minimizing the sum of their squares:

$S = \sum_{i=1}^{n} \left(y_i – \sum_{j=0}^{p} x_{ij} b_j\right)^2$.

The use of squared residuals is not arbitrary. Squaring ensures that positive and negative errors do not cancel each other out, and it places more weight on large errors than small ones. This reflects the idea that a few large mistakes are more problematic than many small ones.

Another important reason is mathematical: the squared objective function is smooth and convex, which guarantees a unique minimum when the model is well specified. This is what allows OLS to have a clean, closed-form solution.

---

3) Matrix Form: Why We Rewrite Everything

To handle regression systematically, especially when the number of predictors grows, we rewrite the model using matrices and vectors.

We collect all predictor values into a design matrix $X$ of size $n \times (p+1)$. Each row corresponds to an observation, and each column corresponds to a feature, including the intercept column of ones.

We collect the coefficients into a vector $b$ of size $(p+1) \times 1$, and the response values into a vector $y$ of size $n \times 1$. The error terms form a vector $\varepsilon$ of the same size.

With this notation, the entire regression model becomes:

$y = Xb + \varepsilon$.

The error assumption can now be written compactly as $\varepsilon \sim N_n(0, \sigma^2 I_n)$.

This tells us three things at once:

• the errors follow a multivariate normal distribution,
• each observation has variance $\sigma^2$,
• and the identity matrix $I_n$ encodes independence between observations.

The matrix formulation is not just shorter; it makes the geometry of regression explicit. The columns of $X$ span a space, and regression is about projecting the response vector onto that space.

---

4) Derivatives: Gradient and Hessian (Why the Normal Equations Appear)

Using matrix notation, the OLS objective can be written as:

$S(b) = (y – Xb)^T (y – Xb)$.

To find the coefficients that minimize this expression, we take derivatives with respect to $b$.

The gradient is $\partial S / \partial b = -2 X^T (y – Xb)$.

The Hessian is $\partial^2 S / (\partial b^T \partial b) = 2 X^T X$.

Conceptually, the gradient tells us the direction in which the objective function increases most rapidly, and the Hessian describes the curvature of the surface.

Setting the gradient equal to zero identifies the point where the objective stops decreasing. Because the Hessian matrix $2 X^T X$ is positive semidefinite—and positive definite when $X^T X$ is invertible—this point is a minimum rather than a maximum or saddle point.

---

5) Normal Equations and the Closed-Form OLS Estimator

Setting the gradient to zero leads to the normal equations:

$X^T X b = X^T y$.

If the matrix $X^T X$ is invertible, we can solve for $b$ directly:

$\hat b = (X^T X)^{-1} X^T y$.

This expression shows that OLS has a direct algebraic solution. However, this solution exists if and only if the columns of $X$ are linearly independent. If any column can be written as a linear combination of others, the inverse does not exist, and the coefficients cannot be uniquely determined.

---

6) What “Inner Product” Means in $X^T X$

Each element of the matrix $X^T X$ has a very concrete interpretation. The $(r, s)$ entry is:

$(X^T X){rs} = \sum{i=1}^{n} x_{ir} x_{is}$.

This is the inner product, or dot product, of column $r$ and column $s$ of the design matrix.

In plain English, we multiply the two columns element by element and then add the results. This tells us how strongly the two predictors move together across observations.

Why this matters is that linear dependence shows up directly in these inner products. If one column can be constructed from others, then $X^T X$ loses invertibility, and the regression coefficients stop being identifiable.

---

7) BLUE: Why OLS Is Special

Under the standard regression assumptions, the OLS estimator is known as the Best Linear Unbiased Estimator, or BLUE.

This means three things:

• Linear: $\hat b$ is a linear function of the observed response vector $y$.
• Unbiased: on average, the estimator equals the true parameter value, so $E[\hat b] = b$.
• Best: among all estimators that are linear and unbiased, OLS has the smallest variance.

It is important to emphasize that “best” here has a specific meaning. It does not mean OLS is the best possible estimator in all situations. It is best within the class of linear unbiased estimators, given the assumptions.

---

8) Predicted Values, Residuals, and Residual Variance

After estimating the coefficients, we can compute predicted values:

$\hat y_i = \sum_{j=0}^{p} x_{ij} \hat b_j$.

The residuals are then:

$e_i = y_i – \hat y_i$.

These residuals summarize how well the model fits each observation. The overall lack of fit is measured by the residual sum of squares:

$RSS = \sum_{i=1}^{n} e_i^2$.

From this, we estimate the error variance as:

$\hat \sigma^2 = RSS / (n – p – 1)$.

The denominator reflects the fact that estimating $p + 1$ parameters consumes degrees of freedom. This adjustment prevents us from underestimating the variability of the errors.

---

9) Aliasing (Perfect Multicollinearity): Why It Breaks the Inverse

Aliasing occurs when some predictors are exact linear combinations of others. For example, if $x_3 = 2x_1 – x_2$, then the columns of $X$ are linearly dependent.

In this situation, $X^T X$ is not invertible, and the formula $(X^T X)^{-1}$ does not exist. As a result, the regression coefficients are not uniquely defined.

Aliasing can be:

• Intrinsic, when it is introduced during feature construction, such as dummy-variable traps.
• Extrinsic, when it arises naturally from the data.

The practical consequence is that individual coefficients lose a clear interpretation, even though predictions may remain stable.

---

10) What the SWEEP Operator Does

The SWEEP operator is a computational tool related to Gauss–Jordan elimination. Instead of explicitly inverting $X^T X$, it systematically examines pivot elements to detect linear dependence.

If a pivot is effectively zero, this signals that the corresponding predictor is aliased. That predictor can then be excluded, and the model can be fit using only the non-aliased features.

When predictors satisfy constraints such as $x_1 + x_2 + x_3 = 1$, there is no unique way to decide which variable should be removed. Different choices lead to different coefficient estimates, but all choices produce the same fitted values.

This highlights an important principle: coefficient estimates may not be unique under aliasing, but predicted responses are.

---

11) The Chicago Taxi Example: Interpreting the Model

Consider a model that predicts total taxi payment using trip duration and trip distance:

$\widehat{Trip_Payment} = 6.6456 + 0.0834 \cdot Trip_Minutes + 2.1768 \cdot Trip_Miles$.

The intercept represents the estimated base fare charged at the beginning of a trip. Even though a trip with zero miles and zero minutes is not realistic, the intercept captures the baseline pricing component.

The coefficient for Trip_Miles indicates that each additional mile is associated with about $2.18 more in payment, holding time constant. This effect is statistically significant and clearly meaningful.

The coefficient for Trip_Minutes suggests an increase of about 8.3 cents per additional minute, holding distance constant. However, this effect is not statistically significant, as its confidence interval includes zero.

---

12) Why Trip Minutes May Not Be Necessary

When a confidence interval includes zero, it means the data are consistent with the possibility that the true effect is negligible. In this case, once distance is accounted for, elapsed time does not appear to add much explanatory power.

This result is intuitive. Taxi fares are often primarily distance-based. Time matters mainly when traffic, idling, or detours play a major role. With a small dataset of similar airport trips, the effect of time is difficult to detect.

This does not prove that time never matters, but it suggests that removing Trip_Minutes could simplify the model without substantially harming predictive performance.

---

13) Diagnostic Reasoning: Why More Features May Be Needed

When comparing predicted values to observed values, a perfect model would place all points on a 45-degree line. Deviations above or below that line indicate overestimation or underestimation.

Residual plots provide deeper insight. In a well-specified linear model, residuals should appear random, centered around zero, with roughly constant spread. Patterns or structure in residuals indicate missing information.

In a taxi setting, additional predictors such as airport destination, time of day, traffic conditions, surcharges, or payment method could capture variation that the current model misses.