Regression analysis is a fundamental statistical method used to model the relationship between a dependent (outcome) variable and one or more independent (predictor) variables. Its primary purposes include prediction, inference, and understanding the strength and form of associations. Originating in the 19th century with the work of Francis Galton on heredity, regression has evolved into a broad family of techniques—ranging from simple linear regression to complex nonparametric and regularization methods—that form the backbone of applied statistics, machine learning, and data science.
1 Introduction to Regression
1.1 Definition and Core Purpose
Regression analysis is a set of statistical processes for estimating the relationships among variables. It focuses on the dependence of a response variable on one or more explanatory variables, with the goal of modeling the conditional expectation of the response given the predictors. Core purposes include: (a) prediction of the response for new observations, (b) inference about the strength and significance of associations, and (c) quantification of the effect size of predictors.
1.2 Historical Background
The term "regression" was coined by Francis Galton in the 1880s while studying heredity. He observed that the heights of offspring tended to "regress" toward the average height of the population, rather than being as extreme as their parents. Karl Pearson and George Udny Yule later formalized the mathematical framework. The method was extended to multiple predictors by Yule and others in the early 20th century. Modern developments include robust estimation, nonparametric smoothing, and regularization techniques.
2 Simple Linear Regression
2.1 Model Specification and Equation
Simple linear regression models the relationship between a single predictor variable \(X\) and a continuous response \(Y\) as:
\[ Y = \beta_0 + \beta_1 X + \varepsilon \]
where \(\beta_0\) is the intercept, \(\beta_1\) is the slope coefficient, and \(\varepsilon\) is the random error term, typically assumed to have mean zero and constant variance.
2.2 Ordinary Least Squares Estimation
The coefficients \(\beta_0\) and \(\beta_1\) are estimated by minimizing the sum of squared residuals (SSE). The ordinary least squares (OLS) estimators are:
\[ \hat{\beta}_1 = \frac{\sum (X_i - \bar{X})(Y_i - \bar{Y})}{\sum (X_i - \bar{X})^2}, \quad \hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}. \]
OLS yields the best linear unbiased estimators under the Gauss–Markov assumptions.
2.3 Goodness-of-Fit Measures
2.3.1 R‑squared
R-squared (\(R^2\)) measures the proportion of variance in \(Y\) explained by the model:
\[ R^2 = 1 - \frac{SSE}{SST}, \]
where SST is total sum of squares. Values range from 0 to 1, with higher values indicating better fit.
2.3.1.1 Adjusted R‑squared
Adjusted R‑squared penalizes for the number of predictors \(k\):
\[ R^2_{\text{adj}} = 1 - \frac{SSE/(n - k - 1)}{SST/(n - 1)}. \]
It is useful when comparing models with different numbers of predictors, as it does not automatically increase with added variables.
2.4 Inference: Confidence Intervals and Hypothesis Tests
For the slope \(\beta_1\), a confidence interval is \(\hat{\beta}_1 \pm t_{\alpha/2, n-2} \cdot SE(\hat{\beta}_1)\). The common test \(H_0: \beta_1 = 0\) uses a t‑statistic \(t = \hat{\beta}_1 / SE(\hat{\beta}_1)\). Similar inference applies to the intercept. The overall model significance can be tested with an F‑test.
3 Multiple Linear Regression
3.1 Model with Multiple Predictors
Multiple linear regression extends simple regression to \(p\) predictors:
\[ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p + \varepsilon. \]
Estimation is performed via OLS using matrix algebra: \(\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{Y}\).
3.2 Interpretation of Coefficients
Each coefficient \(\beta_j\) represents the expected change in \(Y\) for a one‑unit increase in \(X_j\), holding all other predictors constant. Interpretation depends on the scale and coding of predictors; interactions and transformations require careful exposition.
3.3 Variable Selection Methods
3.3.1 Stepwise Selection
Stepwise selection iteratively adds or removes predictors based on statistical criteria (e.g., AIC, BIC, p‑values). Common variants include forward selection, backward elimination, and bidirectional stepwise.
3.3.1.1 Forward Selection Algorithm
Forward selection starts with an empty model and adds the predictor that most significantly improves fit (lowest p‑value or largest increase in criterion). The process continues until no additional predictor meets a predefined threshold.
3.3.2 Penalized Methods (Lasso and Ridge)
| Penalized regression methods add a shrinkage penalty to the OLS objective. Ridge regression uses an L2 penalty (\(\lambda \sum \beta_j^2\)), shrinking coefficients toward zero but never exactly to zero. Lasso uses an L1 penalty (\(\lambda \sum | \beta_j | \)), which can set some coefficients to zero, performing automatic variable selection. |
|---|
3.4 Multicollinearity Issues
Multicollinearity occurs when predictors are highly correlated, inflating the variance of coefficient estimates and making them unstable and difficult to interpret.
3.4.1 Variance Inflation Factor
The variance inflation factor (VIF) quantifies the severity of multicollinearity for each predictor:
\[ \text{VIF}_j = \frac{1}{1 - R_j^2}, \]
where \(R_j^2\) is the R‑squared from regressing \(X_j\) on all other predictors. A VIF exceeding 10 (or 5 in some contexts) indicates problematic collinearity.
4 Key Assumptions and Diagnostics
4.1 Linearity Assumption
The relationship between each predictor and the response (conditional on other predictors) is assumed linear. Diagnostics include scatterplots of residuals versus fitted values or partial‑residual plots.
4.2 Independence of Errors
The error terms are assumed to be independent of each other. Violation (e.g., in time‑series data) can be detected with the Durbin‑Watson test or by examining residual autocorrelation plots.
4.3 Homoscedasticity (Constant Variance)
The variance of errors should be constant across all levels of the predictors. Heteroscedasticity can be assessed via residual plots or formal tests.
4.3.1 Tests for Heteroscedasticity
4.3.1.1 Breusch–Pagan Test
The Breusch–Pagan test regresses squared residuals on the predictors and tests whether the resulting model explains a significant amount of variance. A large test statistic suggests heteroscedasticity.
4.4 Normality of Residuals
For valid inference in small samples, residuals should follow a normal distribution. Departures can be assessed with Q‑Q plots or the Shapiro–Wilk test. In large samples, normality is less critical due to the central limit theorem.
4.5 Outliers and Influential Observations
Outliers are data points with extreme predictor or response values. Influential observations have a disproportionate impact on estimated coefficients.
4.5.1 Cook’s Distance
Cook’s distance measures the influence of each observation on the fitted values. It combines the leverage and residual size. A common cutoff is \(D_i > 4/n\) or \(D_i > 1\).
5 Extensions and Variants
5.1 Logistic Regression (Binary Outcomes)
Logistic regression models the probability of a binary outcome using the logistic function:
\[ \log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 X_1 + \cdots + \beta_p X_p. \]
Coefficients are estimated via maximum likelihood. It is widely used in classification and risk assessment.
5.2 Poisson and Negative Binomial Regression (Count Data)
Poisson regression models count outcomes assuming a Poisson distribution. When overdispersion occurs (variance > mean), negative binomial regression provides a more flexible alternative.
5.3 Nonparametric Regression (Kernel and Splines)
Nonparametric methods relax the linearity assumption. Kernel regression estimates the conditional mean using weighted local averages. Spline regression uses piecewise polynomials with smoothness constraints, often chosen via cross‑validation.
5.4 Regularized Regression (Ridge, Lasso, Elastic Net)
| Elastic Net combines L1 and L2 penalties: \(\lambda_1 \sum | \beta_j | + \lambda_2 \sum \beta_j^2\). It handles groups of correlated predictors better than Lasso alone. Regularized methods are essential for high‑dimensional data where \(p > n\). |
|---|
6 Applications in Practice
6.1 Economics and Finance
Regression is used to estimate demand elasticities, forecast asset returns, and model risk factors. Common examples include the capital asset pricing model (CAPM) and hedonic pricing models.
6.2 Biological and Medical Sciences
Dose‑response relationships, survival analysis extensions (Cox regression), and genome‑wide association studies (GWAS) rely on regression. Logistic regression is frequently applied in clinical risk prediction.
6.3 Social Sciences and Marketing
Sociologists use regression to study social mobility, while marketers model customer lifetime value and advertising response. Causal inference techniques, such as regression with instrumental variables, address endogeneity.
6.4 Common Misinterpretations and Pitfalls
Common errors include: (a) assuming causation from correlation, (b) interpreting coefficients without considering multicollinearity, (c) overfitting by including too many predictors, (d) ignoring nonlinearity, and (e) extrapolating beyond the range of observed data.
7 Software Implementation
7.1 R (lm, glm, caret packages)
The lm function fits linear models, glm handles generalized linear models (e.g., logistic, Poisson). The caret package provides a unified interface for training, tuning, and evaluating regression models, including regularization methods via glmnet.
7.2 Python (statsmodels, scikit‑learn)
statsmodels offers comprehensive regression functions with detailed inference (e.g., OLS, Logit, Poisson). scikit‑learn provides LinearRegression, Ridge, Lasso, and ElasticNet for prediction‑focused tasks, with cross‑validation tools.
7.3 Practical Considerations for Data Preparation
Key steps include: handling missing values (imputation or deletion), encoding categorical variables (dummy or effect coding), scaling predictors for regularized methods, checking for outliers, and splitting data into training/testing sets. Interaction terms and polynomial features should be carefully justified.