Regression Coefficient

A regression coefficient says how much the outcome changes per unit of a predictor, with the other predictors held fixed. In a simple regression

\[ Y = \beta_0 + \beta_1 X + \varepsilon \]

\(\beta_0\) is the intercept — the fitted value of \(Y\) when \(X\) is zero — and \(\beta_1\) is the slope. With several predictors each \(\beta_i\) is a partial coefficient, describing that variable’s contribution with the others accounted for.

The sign is straightforward: positive means the outcome rises with the predictor. The magnitude is where the trouble starts.

A big coefficient is not a big effect

import numpy as np
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(0)
n = 20_000
income = rng.normal(52_000, 15_000, size=n)
rooms = rng.normal(3.2, 1.1, size=n)
price = 4.0 * income + 30_000 * rooms + rng.normal(scale=40_000, size=n)
"""The two predictors are recorded in different units. Nothing about the data
   changes below - only what the numbers are counted in."""

for label, columns in (("income in dollars, rooms in rooms", np.column_stack([income, rooms])),
                       ("income in thousands", np.column_stack([income / 1000, rooms])),
                       ("rooms counted in pairs", np.column_stack([income, rooms / 2]))):
    model = LinearRegression().fit(columns, price)
    print(f"  {label:<34} income {model.coef_[0]:>12,.2f}   rooms {model.coef_[1]:>12,.2f}"
          f"   R2 {model.score(columns, price):.4f}")

X = np.column_stack([income, rooms])
standardized = LinearRegression().fit((X - X.mean(axis=0)) / X.std(axis=0), price)
print(f"\n  standardized, so a coefficient is 'per standard deviation of that column':")
print(f"    income {standardized.coef_[0]:,.2f}   rooms {standardized.coef_[1]:,.2f}")
print(f"    one standard deviation is {X.std(axis=0)[0]:,.0f} dollars and {X.std(axis=0)[1]:.2f} rooms")
#   income in dollars, rooms in rooms  income         3.99   rooms    30,214.38   R2 0.7470
#   income in thousands                income     3,994.80   rooms    30,214.38   R2 0.7470
#   rooms counted in pairs             income         3.99   rooms    60,428.76   R2 0.7470
#
#   standardized, so a coefficient is 'per standard deviation of that column':
#     income 59,684.66   rooms 33,489.84
#     one standard deviation is 14,941 dollars and 1.11 rooms

Read the first three rows. Income’s coefficient is 3.99 when income is in dollars and 3,994.80 when it is in thousands. The rooms coefficient doubles when rooms are counted in pairs. The \(R^2\) is 0.7470 in every row — it is the same model, fitted to the same data, making the same predictions.

So “large \(|\beta|\) means a stronger effect” is not a rule. A coefficient is a rate with units attached — dollars of price per dollar of income, dollars of price per room — and comparing 3.99 against 30,214 compares two quantities measured in different things. The first looks negligible only because a dollar is a small amount of income.

The last block does the comparison properly. Standardizing each predictor first makes every coefficient “change in outcome per standard deviation of this column,” which is a common scale. On that scale income is 59,684.66 and rooms is 33,489.84 — income matters nearly twice as much, the opposite of the impression the dollar-denominated coefficients give.

Two cautions on standardized coefficients. They express importance relative to how much each variable happens to vary in this sample, so a predictor held nearly constant by the sampling design will look unimportant however much it matters. And they are still not causal effects.

“Holding the others constant” is a condition, not a guarantee

The partial coefficient describes what happens when one predictor moves and the rest stay put. Whether that combination occurs in the data is a separate question.

Where two predictors are strongly correlated, the fit has little evidence about moving one without the other, and the coefficients become unstable — splitting between the columns, taking large opposite signs, or swinging between refits. The logistic regression article measures the simplest version: adding a near-duplicate column splits a coefficient of +0.8022 into +0.4022 and +0.4000. The predictions are unaffected; the interpretation is not.

Ridge regularization spreads weight smoothly across correlated columns and makes the failure predictable; lasso picks one of them arbitrarily. Neither recovers which column deserved the credit, because nothing in the data answers that.

Reporting one

• Give the units. “+3.99 dollars of price per dollar of income” is a statement; “+3.99” is not.
• Give an interval. A coefficient is an estimate, and its standard error decides whether the sign can be relied on at all.
• Standardize before ranking predictors, and say that you did.
• Report the correlations among predictors when the coefficients are the deliverable, because that is what determines whether they can be read individually.

And keep the largest caveat in view: a coefficient describes an association in the data the model was fitted to. Whether changing the input would change the outcome is a different question with different requirements, set out in regression models.

References

• scikit-learn, Common pitfalls in the interpretation of coefficients of linear models
• scikit-learn User Guide, Linear Models