Logistic Regression

Logistic regression models the log-odds of the positive class as a linear function of the features:

\[ \ln\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 x_1 + \dots + \beta_n x_n \]

Solving for \(p\) gives the sigmoid, which keeps the output inside [0, 1] where a plain linear fit would not:

\[ p = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \dots + \beta_n x_n)}} \]

It is often introduced as “a classification model despite the name.” The name is accurate: this is a regression, of the log-odds. It becomes a classifier only when a threshold is applied — and as the article on binary classification shows, that threshold is a separate decision that moves precision and recall a long way without touching the model.

The coefficient is an odds ratio

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
n = 40_000
hours = rng.normal(6, 2.5, size=n)
passed = (rng.random(n) < 1 / (1 + np.exp(-(0.8 * hours - 4.0)))).astype(int)
model = LogisticRegression(max_iter=2000).fit(hours.reshape(-1, 1), passed)
beta = model.coef_[0][0]
print(f"  coefficient on hours {beta:+.4f}   odds ratio exp(beta) {np.exp(beta):.4f}")
print(f"  so one more hour multiplies the odds of passing by {np.exp(beta):.4f}, not the probability")
for h in (2, 5, 8):
    p1 = model.predict_proba([[h]])[0, 1]
    p2 = model.predict_proba([[h + 1]])[0, 1]
    print(f"    at {h} hours p={p1:.4f}, at {h + 1} hours p={p2:.4f}   probability change {p2 - p1:+.4f}")

print("\n  training-bin mean predictions and observed rates are close in this experiment")
p = model.predict_proba(hours.reshape(-1, 1))[:, 1]
for lo, hi in ((0.2, 0.4), (0.4, 0.6), (0.6, 0.8)):
    band = (p >= lo) & (p < hi)
    print(f"    predicted in [{lo:.1f}, {hi:.1f})  mean prediction {p[band].mean():.4f}"
          f"   observed rate {passed[band].mean():.4f}")

twin = hours + rng.normal(scale=0.01, size=n)
both = LogisticRegression(max_iter=2000).fit(np.column_stack([hours, twin]), passed)
print(f"\n  adding a near-duplicate column splits the coefficient: "
      f"{both.coef_[0][0]:+.4f} and {both.coef_[0][1]:+.4f}, summing to {both.coef_[0].sum():+.4f}")
#   coefficient on hours +0.8022   odds ratio exp(beta) 2.2305
#   so one more hour multiplies the odds of passing by 2.2305, not the probability
#     at 2 hours p=0.0829, at 3 hours p=0.1678   probability change +0.0849
#     at 5 hours p=0.5007, at 6 hours p=0.6911   probability change +0.1903
#     at 8 hours p=0.9176, at 9 hours p=0.9613   probability change +0.0437
#
#   training-bin mean predictions and observed rates are close in this experiment
#     predicted in [0.2, 0.4)  mean prediction 0.2995   observed rate 0.3010
#     predicted in [0.4, 0.6)  mean prediction 0.5023   observed rate 0.5025
#     predicted in [0.6, 0.8)  mean prediction 0.7062   observed rate 0.7075
#
#   adding a near-duplicate column splits the coefficient: +0.4022 and +0.4000, summing to +0.8022

The coefficient on hours is +0.8022. Exponentiating gives 2.2305, and that number is exact and constant: one more hour of study multiplies the odds of passing by 2.2305, wherever you start.

The probability is another matter. The same extra hour raises the chance of passing by 0.0849 at two hours, 0.1903 at five hours, and 0.0437 at eight. That is the sigmoid’s shape — steep in the middle, flat at both ends — and it is why “each hour adds about eight percentage points” is a sentence with no fixed truth value.

When reporting a logistic model to a non-specialist, the honest options are to give the odds ratio and say it is an odds ratio, or to give the probability change at a stated reference point. Giving an average marginal effect without its reference is where the confusion starts.

Why calibration is plausible, and must still be checked

Log loss is a proper scoring rule: at the population optimum, with conditional probabilities free to vary, reporting the true conditional probability minimizes expected loss. A fitted logistic model is restricted to its specified logit function and is estimated from finite data, often with regularization. Those restrictions mean that training with log loss does not automatically calibrate its predictions.

The three training bins in this synthetic experiment have close mean predictions and observed rates. That observation is consistent with the specified logistic data-generating mechanism, but it is not a general calibration guarantee. Finite samples, model restrictions and regularization remain relevant even when the training objective is log loss.

A well-specified logistic model with suitable regularization can produce well-calibrated risk estimates, which makes it a useful baseline. Check calibration on representative held-out data rather than treating it as free. Class weighting and resampling can change the probability target, but unweighted training is not a calibration guarantee either.

Correlated features wreck the coefficients, not the predictions

The last line adds a column that is the study-hours figure plus a whisker of noise. The coefficient splits: +0.4022 and +0.4000, summing to +0.8022 — the original value, distributed across two columns that say the same thing.

The model’s predictions are unaffected, because the sum is what matters. What breaks is the interpretation. Each coefficient is now half the size, and a reader taking either one at face value would conclude study hours matter half as much as they do. With three correlated columns it splits three ways; with columns correlated in opposite directions the individual coefficients can take large values of opposite sign.

The same arithmetic appears in SHAP, where two columns that the model and the explanation game treat symmetrically — an exact duplicate, for example — split their attribution equally. If the model weights the two columns differently, or the method does not treat them as interchangeable, the split need not be even. The underlying difficulty is a property of additive credit assignment rather than of any one method: when two columns carry one signal, no procedure can say which of them owns it.

Ridge regularization is the usual remedy — it distributes weight smoothly across correlated columns rather than letting them fight — and lasso is not, since it picks one arbitrarily. Neither recovers the truth; they make the failure predictable.

Where it fits

Logistic regression assumes linearity in the log-odds, which is a real constraint and a checkable one — plot the outcome rate against each feature in bins and see whether the log-odds trend is straight. Where it is not, the same fixes apply as for any linear model: transform the feature, add a spline, or move to trees.

For three or more classes it extends through a multinomial fit, which is what scikit-learn does by default, or through One-vs-Rest if you ask for it. And where the classes are ordered, ordinal logistic regression uses that order instead of discarding it.

References

• scikit-learn API reference, LogisticRegression
• scikit-learn User Guide, Linear Models