Logistic regression is a statistical modeling framework designed for situations where the response variable is binary, meaning it takes only two possible values representing whether an event occurs or does not occur. Typical examples include whether an individual finds love, responds to a marketing campaign, churns from a service, commits fraud, survives a medical event, or drops out of school.
Unlike linear regression, which models continuous outcomes, logistic regression focuses on probability modeling while respecting the inherent constraints of probabilities.
Understanding Odds Through a Real-World Example
The concept of odds can initially seem unintuitive, but it is commonly used in everyday contexts. For example, according to a report published on July 17, 2017 by www.eharmony.co.uk, the odds that an adult in the United Kingdom will find love were reported as 1 in 562.
Let $p$ denote the probability of finding love. The statement “odds are 1 in 562” means that
$p/(1-p)=1/562$. Solving this equation yields $p=1/(1+562)\approx0.001776$.
To place this number in context, the probability of having twins is approximately 1 in 67, and the probability of becoming a millionaire (in UK pounds) is approximately 1 in 55. This comparison highlights how odds provide an alternative and often more expressive way of describing rare events.
Odds for a Binary Outcome
For a general binary event, where the probability of occurrence satisfies $0\le p\le1$, the odds of the event are defined as $p/(1-p)$.
Odds take values in the interval $[0,\infty)$. Several special cases clarify their interpretation. If $p=0$, the odds are 0, meaning the event never occurs. If $p=1/2$, the odds are 1, indicating equal chances of occurrence and non-occurrence. If $p=1$, the odds become infinite, reflecting certainty.
This unbounded range distinguishes odds from probabilities and plays a key role in logistic regression.
Odds Versus Probability
While probability is bounded between 0 and 1, odds can grow arbitrarily large. Empirically, it has been observed that the logarithm of the odds behaves approximately linearly with probability over a central range, typically between probabilities of about 0.2 and 0.8.
This near-linear behavior motivates the use of log-odds rather than probability itself in predictive modeling.
Motivation for Logistic Regression
Before the widespread availability of modern computing, practitioners attempted to predict probabilities using linear regression models. These attempts were flawed because linear regression can produce negative predicted probabilities or values greater than 1.
Recognizing that $\log(p/(1-p))$ is approximately linear over much of the probability scale leads to a new idea: instead of modeling probability directly, model the logarithm of the odds as a linear function of predictors.
This idea forms the foundation of logistic regression.
Logistic Regression for a Binary Response
Assume a dataset with $n$ observations. For each observation, $p$ predictors are observed along with a binary response indicating whether an event occurred.
The event represents a category of interest, such as “Found Love,” “Responded to Campaign,” “Retained Customer,” “Identified Fraud,” “Customer Churned,” or “Dropped Out of School.”
Binary Logistic Regression Model
Let $x_{ij}$ denote the $j$th feature of the $i$th observation. Let $\beta_0$ denote the intercept and $\beta_j$ the regression coefficient associated with feature $j$. Let $p_i$ denote the probability that the event occurs for observation $i$.
The logistic regression model is defined by
$\log(p_i/(1-p_i))=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}$.
Solving for $p_i$ yields
$p_i=e^{u_i}/(1+e^{u_i})$, where $u_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}$.
The function $e^u/(1+e^u)$ is known as the logistic function, which maps any real-valued input to a valid probability between 0 and 1. This mapping gives logistic regression its name.
For simplicity, $p_i$ is commonly referred to as the event probability.
Application Domains
Binary logistic regression is widely applied across domains. In marketing, it is used to identify factors influencing purchase behavior and to quantify purchase propensity. In insurance, it helps identify policy attributes predictive of churn and to define customer profiles that maximize retention. In political science, it is used to study demographic drivers of election outcomes and to evaluate how features may favor one candidate over another.
Other examples include predicting patient survival after a heart attack or employment retention several years after hiring.
Parameter Estimation via Maximum Likelihood
Logistic regression parameters are estimated using the maximum likelihood method, which is the industry standard.
This method determines parameter values that maximize the likelihood of observing the data given the model. In doing so, it produces estimated event probabilities that best match the observed outcomes.
Because the likelihood equations do not have a closed-form solution, estimation proceeds through an iterative numerical procedure.
Convergence of the Iterative Procedure
Except in problematic data situations, the iterative estimation procedure converges reliably. Models with more predictors typically require more iterations to converge than simpler models.
The estimation procedure usually implements the Newton–Raphson method or a closely related adaptation. In theory, Newton–Raphson converges at a quadratic rate, meaning that up to two additional significant digits of accuracy may be gained per iteration near convergence.
Logistic Regression in StatsModels
In the StatsModels library, binary logistic regression is implemented via the Logit() class, while multi-class logistic regression is implemented via MNLogit().
Both classifiers estimate model parameters using maximum likelihood. The MNLogit() implementation is generally preferred because it is more flexible.
For a binary response, both classifiers treat the last category as the event by default. The last category corresponds to the lexicographically higher label; for example, if the response categories are “No” and “Yes,” the event category is “Yes.”
Practical Usage Notes in StatsModels
Neither Logit() nor MNLogit() automatically includes an intercept term. A column of ones must be explicitly added to the model matrix.
Additionally, neither classifier checks for linearly dependent columns. The user must ensure that the design matrix contains no aliased or redundant predictors.
Checking for Linear Dependence
To detect linear dependence, the SWEEP operator can be used. After identifying aliased rows and columns, only the non-aliased columns of the design matrix should be retained for model training.
In practice, these steps are often wrapped into a custom logistic modeling function to ensure robustness.
The Separation Phenomenon
A distinctive issue in logistic regression is the separation phenomenon, which occurs when predictors deterministically explain the response.
In complete separation, predictors perfectly determine the response for every observation. In quasi-complete separation, predictors do so for most, but not all, observations.
For example, if $x_1=0.4$ or $0.6$ always corresponds to $y=1$, the data exhibit separation.
Common Causes of Separation
Separation commonly arises when categorical variables have as many unique values as observations, such as employee IDs. It can also occur when continuous variables are mistakenly treated as categorical or when too many predictors are included relative to the number of observations.
Technical Signs of Separation
Several warning signs indicate separation. The log-likelihood becomes nearly zero, predicted event probabilities approach 1 for event observations and 0 otherwise, and parameter estimates grow extremely large in magnitude, with even larger standard errors.
Although such models may appear to fit the training data perfectly, they are unreliable and lack generalizability.
Why Separation Is Undesirable
Perfect prediction in training data is not guaranteed to hold for new data. Moreover, extremely large parameter estimates are not meaningful and indicate instability rather than strong evidence.
Workarounds for Separation
Several practical remedies exist. Categories with low counts can be merged. Continuous variables can be rounded to reduce precision. Alternatively, continuous predictors can be pre-binned and used as categorical variables instead.
These approaches reduce deterministic patterns and enable stable estimation.
Final Perspective
Logistic regression models binary outcomes by transforming probabilities into log-odds, fitting a linear structure in that space, and mapping predictions back to probabilities. When combined with maximum likelihood estimation and careful data preprocessing, it provides a powerful, interpretable, and widely applicable framework for binary classification problems.