Linear regression is a statistical modeling technique that describes the relationship between a dependent variable (outcome) and one or more independent variables (predictors) by fitting a linear equation to observed data. The model assumes that the conditional mean of the dependent variable is a linear function of the predictors, with errors typically assumed to be independent, homoscedastic, and normally distributed. Estimation is most commonly performed using ordinary least squares (OLS), which minimizes the sum of squared residuals. Linear regression serves as a foundation for many advanced methods and is widely applied in fields such as economics, social sciences, and engineering.

1 Introduction

1.1 History

The method of least squares was first published by Adrien-Marie Legendre in 1805 and independently by Carl Friedrich Gauss in 1809, who claimed to have used it since 1795. Gauss provided a rigorous justification by linking least squares to the principle of maximum likelihood under the assumption of normally distributed errors. Francis Galton later introduced the term "regression" in the 1880s while studying heredity, observing that extreme traits in parents tended to "regress" toward the average in offspring. The development of multiple regression and matrix formulations occurred in the early 20th century, with contributions from Karl Pearson, George Udny Yule, and R. A. Fisher, who integrated regression into the framework of analysis of variance.

1.2 Basic Concepts

Linear regression models the expected value of the dependent variable \(Y\) as a linear combination of the predictors \(X_1, X_2, \dots, X_p\). The relationship is expressed as \(Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \varepsilon\), where \(\beta_0\) is the intercept, \(\beta_j\) are coefficients, and \(\varepsilon\) is the error term. The goal is to estimate the unknown parameters from observed data. The simplest case involves a single predictor (simple linear regression), while multiple predictors are handled by multiple linear regression.

2 Model Specification

2.1 Simple Linear Regression

In simple linear regression, there is one predictor \(X\) and one dependent variable \(Y\). The model is \(Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i\) for \(i = 1, \dots, n\). The slope \(\beta_1\) represents the expected change in \(Y\) for a one-unit increase in \(X\). The intercept \(\beta_0\) is the expected value of \(Y\) when \(X = 0\). Estimation typically uses OLS to find the line that minimizes the sum of squared vertical distances from the data points.

2.2 Multiple Linear Regression

Multiple linear regression extends the model to include two or more predictors: \(Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \dots + \beta_p X_{ip} + \varepsilon_i\). Each coefficient \(\beta_j\) measures the partial effect of \(X_j\) on \(Y\) while holding other predictors constant. The interpretation assumes no perfect multicollinearity among predictors. The model can also accommodate categorical predictors through dummy coding.

2.3 Matrix Notation

The multiple linear regression model is expressed compactly in matrix form as \(\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}\), where \(\mathbf{Y}\) is an \(n \times 1\) vector of responses, \(\mathbf{X}\) is an \(n \times (p+1)\) design matrix (including a column of ones for the intercept), \(\boldsymbol{\beta}\) is a \((p+1) \times 1\) vector of coefficients, and \(\boldsymbol{\varepsilon}\) is an \(n \times 1\) vector of errors. This notation facilitates derivation of estimators and properties using linear algebra.

3 Estimation Methods

3.1 Ordinary Least Squares (OLS)

OLS minimizes the sum of squared residuals: \(\hat{\boldsymbol{\beta}} = \arg \min_{\boldsymbol{\beta}} (\mathbf{Y} - \mathbf{X}\boldsymbol{\beta})^T (\mathbf{Y} - \mathbf{X}\boldsymbol{\beta})\).

3.1.1 Derivation

3.1.1.1 Normal Equations

Taking the derivative of the sum of squared residuals with respect to \(\boldsymbol{\beta}\) and setting to zero yields the normal equations: \(\mathbf{X}^T\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}^T\mathbf{Y}\). If \(\mathbf{X}^T\mathbf{X}\) is invertible, the OLS estimator is \(\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}\).

3.1.2 Properties

Under the Gauss–Markov assumptions (linearity, zero conditional mean, homoscedasticity, and uncorrelated errors), the OLS estimator is the best linear unbiased estimator (BLUE). It is consistent and asymptotically normal. If errors are also normally distributed, OLS coincides with the maximum likelihood estimator.

3.2 Maximum Likelihood Estimation

Assuming normally distributed errors \(\varepsilon_i \sim N(0, \sigma^2)\), the likelihood function for the regression model is \(L(\boldsymbol{\beta}, \sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(Y_i - \mathbf{X}_i\boldsymbol{\beta})^2}{2\sigma^2}\right)\). Maximizing the log-likelihood yields the same estimator for \(\boldsymbol{\beta}\) as OLS, and the MLE for \(\sigma^2\) is \(\hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^n \hat{\varepsilon}_i^2\), which is biased downward.

4 Assumptions and Diagnostics

4.1 Key Assumptions

4.1.1 Linearity

The relationship between the predictors and the conditional mean of the dependent variable is linear. Violations can be detected by residual plots showing systematic patterns.

4.1.2 Independence

Observations are independent of each other. This is often ensured by random sampling. Time-series data may require autocorrelation checks.

4.1.3 Homoscedasticity

The variance of errors is constant across all levels of the predictors. Heteroscedasticity can be detected by a funnel-shaped pattern in residual versus fitted plots.

4.1.4 Normality

Errors are normally distributed, especially important for small-sample inference. Moderate departures are less critical with large samples due to the central limit theorem.

4.2 Diagnostic Tools

4.2.1 Residual Plots

A plot of residuals versus fitted values is used to assess linearity and homoscedasticity. Residuals should scatter randomly around zero with no obvious pattern.

4.2.2 Q–Q Plots

A quantile-quantile plot compares the distribution of residuals to a normal distribution. Deviations from the diagonal line indicate non-normality.

4.2.3 Formal Tests (e.g., Durbin-Watson, Breusch-Pagan)

The Durbin-Watson test detects autocorrelation in residuals. The Breusch-Pagan test checks for heteroscedasticity by regressing squared residuals on predictors. Other common tests include the Shapiro-Wilk test for normality.

5 Inference and Hypothesis Testing

5.1 Coefficient Significance (t-tests)

For each coefficient \(\beta_j\), the null hypothesis \(H_0: \beta_j = 0\) is tested using a t-statistic: \(t = \frac{\hat{\beta}_j}{\text{SE}(\hat{\beta}_j)}\), which follows a t-distribution with \(n-p-1\) degrees of freedom under the null. A small p-value suggests the predictor has a statistically significant linear relationship with the response.

5.2 Overall Model Significance (F-test)

The F-test evaluates whether all slope coefficients (excluding the intercept) are jointly zero: \(H_0: \beta_1 = \beta_2 = \dots = \beta_p = 0\). The test statistic is \(F = \frac{( \text{TSS} - \text{RSS} ) / p}{\text{RSS} / (n-p-1)}\), where TSS is total sum of squares and RSS is residual sum of squares. A significant F-statistic indicates that the model explains a significant portion of the variance in \(Y\).

5.3 Confidence Intervals

A \(100(1-\alpha)\%\) confidence interval for a coefficient \(\beta_j\) is given by \(\hat{\beta}_j \pm t_{\alpha/2, n-p-1} \cdot \text{SE}(\hat{\beta}_j)\). Confidence intervals provide a range of plausible values for the unknown parameter.

6 Model Evaluation

6.1 R-squared and Adjusted R-squared

R-squared (\(R^2\)) is the proportion of variance in \(Y\) explained by the model: \(R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}\). Adding predictors always increases \(R^2\), so adjusted \(R^2\) penalizes for the number of predictors: \(\bar{R}^2 = 1 - \frac{\text{RSS}/(n-p-1)}{\text{TSS}/(n-1)}\).

6.2 AIC and BIC

The Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) are model selection metrics that balance fit and complexity. Lower values indicate better models. For linear regression, AIC = \(n \ln(\text{RSS}/n) + 2(p+1)\), and BIC = \(n \ln(\text{RSS}/n) + \ln(n)(p+1)\).

6.3 Cross-Validation

Cross-validation splits the data into training and validation sets to assess predictive performance. Common approaches include k-fold cross-validation and leave-one-out cross-validation. The mean squared error (MSE) on validation folds is used to compare models.

7.1 Polynomial Regression

Polynomial regression adds powers of predictors (e.g., \(X^2, X^3\)) to model nonlinear relationships while remaining linear in the parameters. It is fitted using OLS but requires careful choice of degree to avoid overfitting.

7.2 Regularization (Ridge, Lasso)

Ridge regression adds an L2 penalty (\(\lambda \sum \beta_j^2\)) to the OLS objective, shrinking coefficients but keeping all predictors. Lasso regression uses an L1 penalty (\(\lambda \sum\beta_j\)), which can shrink some coefficients to exactly zero, performing automatic variable selection. Both methods are useful when predictors are numerous or highly correlated.

7.3 Generalized Linear Models

Generalized linear models (GLMs) extend linear regression to response variables with non-normal error distributions (e.g., binary, count, or skewed data). They use a link function to relate the mean of the response to the linear predictor. Common examples include logistic regression (binary outcomes) and Poisson regression (count data).

8 Applications

8.1 Economics

Linear regression is used to estimate demand elasticities, analyze wage determinants, forecast GDP growth, and evaluate policy effects. For example, economists may regress consumption on income to estimate the marginal propensity to consume.

8.2 Social Sciences

Social scientists apply regression to study relationships such as education–income correlations, voting behavior, and health outcomes. It allows controlling for confounding variables through multiple regression.

8.3 Engineering

Engineers use linear regression for calibration, quality control, and performance modeling. Examples include predicting material strength from composition, modeling energy consumption based on temperature, and fitting sensor data to linear models.