Linear Models

A linear model predicts by taking a weighted sum:

\[ \hat{y} = w_1 x_1 + w_2 x_2 + \dots + w_d x_d + b \]

The family covers logistic regression, which puts a sigmoid around that sum, linear SVMs, which threshold it, and the regularized variants — ridge penalizing squared weights, lasso penalizing absolute weights and driving some to zero, elastic net doing both.

“Linear” refers to the parameters, not the features. That distinction does more work than it looks.

A linear model fitting a circle

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(0)
n = 4000
X = rng.uniform(-2, 2, size=(n, 2))
y = (X[:, 0] ** 2 + X[:, 1] ** 2 < 1.6).astype(int)
"""The positive class is a disc. No straight line separates it from the rest."""

plain = cross_val_score(LogisticRegression(max_iter=2000), X, y, cv=5).mean()
squared = cross_val_score(make_pipeline(PolynomialFeatures(2), LogisticRegression(max_iter=5000)),
                          X, y, cv=5).mean()
print(f"  logistic regression on x1, x2                       accuracy {plain:.4f}")
print(f"  the same model on degree-2 polynomial features      accuracy {squared:.4f}")

model = make_pipeline(PolynomialFeatures(2), LogisticRegression(max_iter=5000)).fit(X, y)
names = model[0].get_feature_names_out(["x1", "x2"])
coefs = model[1].coef_[0]
print("\n  the fitted model is still a weighted sum - of transformed inputs")
for name, c in zip(names, coefs):
    print(f"    {name:<8} {c:+.4f}")
#   logistic regression on x1, x2                       accuracy 0.6973
#   the same model on degree-2 polynomial features      accuracy 0.9963
#
#   the fitted model is still a weighted sum - of transformed inputs
#     1        -0.0027
#     x1       +0.0391
#     x2       +0.0429
#     x1^2     -8.6138
#     x1 x2    +0.0269
#     x2^2     -8.6043

The positive class is a disc, so no straight line separates it. Logistic regression on the two raw coordinates manages 0.6973 — barely better than guessing the majority class.

Hand the same model the degree-2 polynomial expansion of those two columns and it reaches 0.9963. The coefficients show why: \(x_1^2\) and \(x_2^2\) both get about \(-8.6\), the cross term gets essentially nothing, and the model has written down \(x_1^2 + x_2^2 < c\) — the equation of a disc.

Nothing about the model changed. It is still a weighted sum, still fitted by the same optimizer, still linear in its parameters. What changed is what it was given to weight.

So “linear models can only capture linear decision boundaries” is not right as stated, and it is worth replacing with the accurate version: a linear model draws a linear boundary in the space of the features you supply. Supply squares and it draws conics; supply interactions and it captures them; supply a kernel’s implicit features and it draws whatever that kernel allows.

Where the real limitation sits

The constraint is that you have to know which transformation. The model will not discover that squaring the inputs is the right move; it fits weights for whatever columns arrive. With two features, a degree-2 expansion is six columns and you can just try it. With fifty features, a degree-2 expansion is over a thousand, degree-3 is over twenty thousand, and most of them are noise.

That is the trade the other model families make differently. Decision trees discover interactions by splitting recursively. Kernel SVMs work in a high-dimensional feature space without constructing it. Neural networks learn the transformation and the weights together. Each buys that at a cost in interpretability, data, or compute.

The loss and design determine the limits. Squared-error linear regression is sensitive to large residuals; robust linear models change that loss, and logistic regression uses a different one again. Correlated columns can make individual coefficients unstable even when predictions in the observed region are relatively stable. Predictions outside that region need a separate assessment.

Why they remain the default first move

• The coefficient is a statement. “One more unit of this feature adds \(w\) to the prediction, holding the others fixed” is a claim a domain expert can dispute, which is more than most models offer.
• Many linear-model objectives are convex and computationally tractable. Convexity alone does not give a unique finite optimum. Rank-deficient least squares has multiple coefficient solutions, and unregularized logistic regression on separable data may have no finite minimizer. The penalty, data rank and solver stopping criteria still matter.
• They set the baseline that matters. A gradient-boosting model that beats a regularized linear model by two points has told you something; one that has never been compared to it has not.
• Regularization makes them hard to overfit. With ridge or lasso and cross-validated strength, they degrade gracefully as features are added.

The generalized linear model family extends the same machinery to other outcome types — Poisson regression for counts, gamma for positive skewed quantities — by changing the link function rather than the structure. It is the same weighted sum, read through a different lens.

References

• scikit-learn User Guide, Linear Models
• scikit-learn API reference, PolynomialFeatures