1 Introduction
1.1 Definition and purpose
Logistic regression is a statistical modeling technique used to estimate the probability of a binary outcome (e.g., success/failure, yes/no) based on one or more predictor variables. Its purpose is to model the relationship between predictors and a dichotomous dependent variable, providing interpretable measures of association and allowing classification into one of two categories.
1.2 Historical development
Logistic regression originated in the mid‑20th century, with early work by statisticians such as David Cox (1958) and developments in biostatistics for analyzing dose‑response relationships. The method was later popularized in epidemiology and social sciences through the work of researchers like Joseph Berkson and John Nelder. In the 1990s and 2000s, logistic regression became a cornerstone of machine learning for binary classification tasks.
1.3 Relationship to linear regression
Unlike linear regression, which predicts a continuous response, logistic regression predicts the probability of an event using the logistic (sigmoid) function. This ensures predicted probabilities are bounded between 0 and 1. Linear regression assumes a linear relationship between predictors and the outcome; logistic regression assumes a linear relationship between predictors and the log‑odds (logit) of the outcome.
2 Mathematical formulation
2.1 Logistic function (sigmoid)
The logistic function maps any real‑valued number to a value in (0, 1): \[ \pi(x) = \frac{e^{x}}{1+e^{x}} = \frac{1}{1+e^{-x}}. \] In logistic regression, the argument is the linear combination of predictors.
2.2 Logit transformation
The logit (log‑odds) is the inverse of the logistic function. For a probability \(p\), the logit is given by \[ \operatorname{logit}(p) = \ln\left(\frac{p}{1-p}\right). \]
2.2.1 Odds and log‑odds
Odds are defined as \(p/(1-p)\), ranging from 0 to ∞. The log‑odds is the natural logarithm of the odds and ranges from −∞ to +∞. Logistic regression models the log‑odds as a linear function of predictors.
2.3 Model equation
2.3.1 Single predictor case
For a single predictor \(x\): \[ p(Y=1\mid x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}. \]
2.3.2 Multiple predictors
For \(k\) predictors \(x_1, x_2, \dots, x_k\): \[ p(Y=1\mid \mathbf{x}) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k)}}. \]
2.4 Estimation of coefficients
2.4.1 Maximum likelihood estimation (MLE)
Coefficients are estimated by maximizing the likelihood function, which expresses the probability of observing the data given the parameters. The log‑likelihood for a binary response is: \[ \ell(\boldsymbol{\beta}) = \sum_{i=1}^{n} \left[ y_i \ln(p_i) + (1-y_i)\ln(1-p_i) \right]. \]
2.4.2 Iterative algorithms (Newton‑Raphson, IRLS)
Because the likelihood equations are nonlinear, closed‑form solutions are not available. Iterative algorithms such as the Newton‑Raphson method or iteratively reweighted least squares (IRLS) are used to find the MLE numerically.
3 Assumptions and diagnostics
3.1 Binary outcome and independence
Logistic regression assumes the response variable is binary and that observations are independent of one another (no clustering or repeated measures unless accounted for in extensions).
3.2 Linearity of logit
The log‑odds of the outcome must be linearly related to continuous predictors. This can be checked using component‑plus‑residual plots or by adding polynomial/spline terms.
3.3 Absence of multicollinearity
High correlations among predictor variables can inflate standard errors and destabilize coefficient estimates. Variance inflation factors (VIF) are commonly examined.
3.4 Goodness‑of‑fit tests
3.4.1 Hosmer–Lemeshow test
The Hosmer–Lemeshow test groups observations by predicted probabilities and compares observed to expected frequencies using a chi‑squared statistic. A significant result suggests poor fit.
3.4.2 Deviance and Pearson residuals
Deviance residuals measure each observation’s contribution to model deviance; Pearson residuals compare observed to fitted values standardized by variance. Both can identify outliers or systematic misfit.
3.5 Influence and leverage (Cook’s distance)
Cook’s distance quantifies the change in fitted coefficients when an observation is removed. High leverage points (unusual predictor combinations) can disproportionately affect estimates.
4 Model interpretation
4.1 Odds ratios
Odds ratios (OR) are exponentiated logistic regression coefficients: \(\text{OR} = e^{\beta}\). They indicate the multiplicative change in odds of the outcome for a one‑unit increase in the predictor, holding other variables constant.
4.1.1 Interpretation for continuous predictors
For a continuous predictor, OR represents the factor by which the odds change when the predictor increases by one unit. Values greater than 1 indicate increased odds, less than 1 decreased odds.
4.1.2 Interpretation for categorical predictors
For a categorical predictor (e.g., treatment vs. control), the OR compares the odds for one level against a reference category. The reference category must be clearly stated.
4.2 Predicted probabilities
Probabilities are obtained by plugging the estimated coefficients back into the logistic function. For a given set of predictors, the model outputs a number between 0 and 1 that can be used for classification (e.g., using a cutoff of 0.5).
4.3 Marginal effects
Marginal effects measure the average change in predicted probability for a one‑unit increase in a predictor. Unlike odds ratios, they are expressed on the probability scale and can differ depending on the level of other predictors (e.g., at mean values vs. averaged over the sample).
5 Extensions and variants
5.1 Multinomial logistic regression
5.1.1 Nominal outcomes
When the outcome has more than two unordered categories (e.g., choice of transportation: car, bus, bike), multinomial logistic regression models the log‑odds relative to a baseline category.
5.1.2 Baseline category logit
One category is designated as the reference. For each remaining category, a separate logistic model is estimated. The probabilities for all categories sum to 1.
5.2 Ordinal logistic regression
5.2.1 Proportional odds assumption
For ordered outcomes (e.g., low/medium/high), ordinal logistic regression assumes that the effect of predictors is constant across cumulative logits (proportional odds). This assumption can be tested with a likelihood‑ratio or Brant test.
5.3 Regularized logistic regression
5.3.1 Lasso (L1) and ridge (L2)
Regularization adds a penalty to the log‑likelihood to shrink coefficients, reducing overfitting. Lasso performs variable selection by shrinking some coefficients exactly to zero; ridge shrinks coefficients but keeps all variables. Elastic net combines both.
5.4 Bayesian logistic regression
In a Bayesian framework, prior distributions are placed on coefficients. Posterior distributions are estimated via Markov chain Monte Carlo (MCMC) or variational inference. This approach naturally quantifies uncertainty and can incorporate prior knowledge.
6 Applications
6.1 Medical diagnosis (disease presence)
Logistic regression is used to estimate the probability of a disease based on symptoms, biomarkers, or imaging results. For example, predicting the likelihood of diabetes from age, BMI, and glucose level.
6.2 Credit risk modeling
Banks use logistic regression to assess the probability of loan default based on income, credit history, and debt‑to‑income ratio. The predicted probability determines whether to approve or deny credit.
6.3 Marketing (customer churn prediction)
Companies model the probability that a customer will discontinue service (churn) using usage patterns, tenure, and support interactions. This allows targeted retention efforts.
6.4 Biology (species distribution)
Ecologists use logistic regression to model the presence or absence of a species as a function of environmental variables (temperature, precipitation, elevation). This helps map habitat suitability.
7 Software implementation
7.1 R (glm function)
In R, logistic regression is performed using glm() with family = binomial(link = "logit"). The function returns coefficient estimates, standard errors, z‑statistics, and p‑values. Summary and diagnostic functions are available.
7.2 Python (scikit‑learn, statsmodels)
Scikit‑learn provides LogisticRegression for regularized models, while statsmodels offers Logit with comprehensive statistical output (coefficients, confidence intervals, fit statistics). Both support multinomial and ordinal extensions.
7.3 SAS and SPSS
SAS uses PROC LOGISTIC for binary, multinomial, and ordinal logistic regression. SPSS offers logistic regression through the LOGISTIC REGRESSION and NOMREG procedures. Both packages include built‑in diagnostics and variable selection options.
8 Limitations and alternatives
8.1 Nonlinear relationships
Logistic regression assumes linearity in the logit; fitting truly nonlinear patterns may require polynomial terms, splines, or interaction terms. Alternatively, non‑parametric methods like kernel logistic regression can be used.
8.2 Separation and quasi‑complete separation
When a predictor perfectly or nearly perfectly separates the outcome categories, maximum likelihood estimates become infinite. Solutions include bias‑corrected methods (Firth’s logistic regression) or regularization (lasso/ridge).
8.3 Alternatives (decision trees, neural networks)
Decision trees handle nonlinearities automatically and are more interpretable for many categorical predictors. Neural networks (including simple feed‑forward networks) can model complex decision boundaries but offer less direct interpretability. Support vector machines and random forests are other common alternatives.