In statistics, a linear model is an approach that models the relationship between a dependent variable and one or more independent variables using a linear predictor function. The model assumes that the conditional expectation of the response variable is a linear combination of the predictors. Linear models are foundational to regression analysis, analysis of variance (ANOVA), and many machine learning techniques. Their simplicity, interpretability, and well-understood statistical properties make them a cornerstone of applied statistics.

1 Introduction

1.1 Definition and basic form

A linear model expresses the dependent variable \(Y\) as a linear function of the independent variables \(X_1, X_2, \dots, X_k\) plus a random error term \(\varepsilon\):

\[ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_k X_k + \varepsilon, \]

where \(\beta_0, \beta_1, \dots, \beta_k\) are unknown parameters (coefficients). The model is "linear" in the parameters; the predictors themselves can include nonlinear transformations (e.g., \(X^2\) or \(\log X\)) as long as the coefficients enter linearly.

1.2 Notation and assumptions

Common notation uses \(n\) observations, with the \(i\)-th observation written as \(Y_i = \beta_0 + \beta_1 X_{i1} + \dots + \beta_k X_{ik} + \varepsilon_i\). Classical assumptions for inference include:

  • Linearity: The expected value of \(Y\) is a linear combination of the \(X\)s.
  • Independence: The errors \(\varepsilon_i\) are independent.
  • Homoscedasticity: The errors have constant variance \(\sigma^2\).
  • Normality: The errors are normally distributed (required for exact finite-sample inference).
  • No perfect multicollinearity: The predictors are not perfectly linearly dependent.

2 Types of linear models

2.1 Simple linear regression

2.1.1 One predictor case

Simple linear regression has a single independent variable: \(Y = \beta_0 + \beta_1 X + \varepsilon\). The slope \(\beta_1\) represents the expected change in \(Y\) per unit change in \(X\). The least-squares estimates are given by \(\hat{\beta}_1 = \frac{\sum (X_i - \bar{X})(Y_i - \bar{Y})}{\sum (X_i - \bar{X})^2}\) and \(\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}\).

2.2 Multiple linear regression

2.2.1 Matrix formulation

Multiple linear regression extends to \(k\) predictors. In matrix notation, \(\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}\), where \(\mathbf{X}\) is an \(n \times (k+1)\) design matrix (first column of ones for the intercept), \(\boldsymbol{\beta}\) is a \((k+1) \times 1\) coefficient vector, and \(\boldsymbol{\varepsilon}\) is an \(n \times 1\) error vector. The ordinary least squares (OLS) estimator is \(\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{Y}\).

2.2.2 Interpretation of coefficients

Each coefficient \(\beta_j\) (for \(j \ge 1\)) measures the expected change in \(Y\) when \(X_j\) increases by one unit, holding all other predictors constant. This "ceteris paribus" interpretation is central to multiple regression. The intercept \(\beta_0\) gives the expected value of \(Y\) when all \(X_j = 0\), provided that zero is meaningful.

2.3 Analysis of variance (ANOVA)

2.3.1 One-way ANOVA

One-way ANOVA tests whether the means of \(Y\) differ across \(g\) groups. It can be expressed as a linear model: \(Y_{ij} = \mu + \alpha_i + \varepsilon_{ij}\), where \(\mu\) is the overall mean and \(\alpha_i\) is the effect of group \(i\) (with a constraint such as \(\sum \alpha_i = 0\)). The total variability is partitioned into between-group and within-group sums of squares.

2.3.2 Two-way ANOVA and interactions

Two-way ANOVA includes two categorical factors (A and B) and possibly their interaction. The model is \(Y_{ijk} = \mu + \alpha_i + \beta_j + (\alpha\beta)_{ij} + \varepsilon_{ijk}\). Interaction terms allow the effect of one factor to depend on the level of the other. The linear model framework easily handles such designs by creating dummy variables.

3 Estimation methods

3.1 Ordinary least squares (OLS)

3.1.1 Derivation and normal equations

OLS minimizes the sum of squared residuals: \(\text{RSS}(\boldsymbol{\beta}) = \sum_{i=1}^n (Y_i - \mathbf{X}_i \boldsymbol{\beta})^2\). Setting the gradient to zero yields the normal equations: \(\mathbf{X}^\top \mathbf{X} \hat{\boldsymbol{\beta}} = \mathbf{X}^\top \mathbf{Y}\). When \(\mathbf{X}^\top \mathbf{X}\) is invertible, the unique solution is \(\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{Y}\).

3.1.2 Properties of OLS estimators

Under the classical assumptions, the OLS estimators are unbiased (\(\mathbb{E}[\hat{\boldsymbol{\beta}}] = \boldsymbol{\beta}\)), have minimum variance among all linear unbiased estimators (Gauss–Markov theorem), and are consistent. Under normality, \(\hat{\boldsymbol{\beta}}\) is also normally distributed.

3.2 Maximum likelihood estimation

3.2.1 Under normality assumption

If \(\varepsilon_i\) are independent and normally distributed with mean 0 and variance \(\sigma^2\), the likelihood function is \(L(\boldsymbol{\beta}, \sigma^2) = (2\pi\sigma^2)^{-n/2} \exp\left(-\frac{1}{2\sigma^2} (\mathbf{Y} - \mathbf{X}\boldsymbol{\beta})^\top (\mathbf{Y} - \mathbf{X}\boldsymbol{\beta})\right)\). Maximizing the log-likelihood yields the same \(\hat{\boldsymbol{\beta}}\) as OLS. The ML estimate of \(\sigma^2\) is \(\hat{\sigma}^2_{\text{ML}} = \frac{1}{n}\sum \hat{\varepsilon}_i^2\), which is biased; the unbiased OLS estimate uses divisor \(n - k - 1\).

3.3 Regularized variants

3.3.1 Ridge regression (L2)

Ridge regression adds a penalty proportional to the sum of squared coefficients: \(\hat{\boldsymbol{\beta}} = \arg\min \left\{ \sum_{i=1}^n (Y_i - \mathbf{X}_i \boldsymbol{\beta})^2 + \lambda \sum_{j=1}^k \beta_j^2 \right\}\). The ridge solution shrinks coefficients toward zero and handles multicollinearity. It is not invariant to scaling, so predictors are typically standardized.

3.3.2 Lasso regression (L1)

Lasso uses an L1 penalty: \(\hat{\boldsymbol{\beta}} = \arg\min \left\{ \sum_{i=1}^n (Y_i - \mathbf{X}_i \boldsymbol{\beta})^2 + \lambda \sum_{j=1}^k\beta_j\right\}\). The L1 penalty forces some coefficients to exactly zero, performing variable selection. The tuning parameter \(\lambda\) controls the amount of shrinkage.

4 Diagnostic and inference

4.1 Goodness-of-fit measures

4.1.1 R-squared and adjusted R-squared

\(R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}\) measures the proportion of variance in \(Y\) explained by the model, where TSS is total sum of squares. Adding more predictors always increases \(R^2\). The adjusted \(R^2\) penalizes for the number of predictors: \(\bar{R}^2 = 1 - \frac{\text{RSS}/(n-k-1)}{\text{TSS}/(n-1)}\).

4.2 Hypothesis testing

4.2.1 F-test for overall significance

The F-test evaluates the null hypothesis that all slope coefficients are zero: \(H_0: \beta_1 = \dots = \beta_k = 0\). The test statistic is \(F = \frac{(\text{TSS} - \text{RSS})/k}{\text{RSS}/(n-k-1)}\), which under \(H_0\) follows an \(F\)-distribution with \((k, n-k-1)\) degrees of freedom.

4.2.2 t-test for individual coefficients

For a single coefficient \(\beta_j\), the t-test statistic is \(t = \frac{\hat{\beta}_j - \beta_{j0}}{\text{SE}(\hat{\beta}_j)}\). Under the null \(H_0: \beta_j = 0\) and standard assumptions, \(t\) follows a t-distribution with \(n-k-1\) degrees of freedom. Confidence intervals for \(\beta_j\) are constructed as \(\hat{\beta}_j \pm t_{\alpha/2, n-k-1} \cdot \text{SE}(\hat{\beta}_j)\).

4.3 Residual analysis

4.3.1 Normality of residuals

Deviation from normality can be assessed via Q–Q plots, histograms, or formal tests (e.g., Shapiro–Wilk). Violations affect the validity of t-tests and F-tests in small samples but are less critical in large samples due to the central limit theorem.

4.3.2 Homoscedasticity

Constant error variance is checked by plotting residuals versus fitted values or using tests like Breusch–Pagan. Heteroscedasticity leads to inefficient OLS estimates and biased standard errors; robust (sandwich) standard errors can be used.

4.3.3 Independence and autocorrelation

Independence is often assumed from sampling design. For time-series data, autocorrelation in residuals (e.g., Durbin–Watson test) violates assumptions and may require autoregressive error models or generalized least squares.

5.1 Generalized linear models (GLMs)

GLMs extend linear models to non-normal response distributions (e.g., binary, count). They consist of three components: a random component from an exponential family, a linear predictor \(\eta = \mathbf{X}\boldsymbol{\beta}\), and a link function \(g(\cdot)\) such that \(\mathbb{E}[Y] = g^{-1}(\eta)\). Common examples include logistic regression (logit link) and Poisson regression (log link).

5.2 Mixed-effects models

Mixed-effects (or multilevel) models include both fixed effects (population-level parameters) and random effects (subject-specific deviations). For example, a model with random intercepts: \(Y_{ij} = \beta_0 + b_i + \beta_1 X_{ij} + \varepsilon_{ij}\), where \(b_i \sim N(0, \sigma_b^2)\). These account for clustering and repeated measures.

5.3 Nonparametric and semiparametric alternatives

When the linearity assumption is too restrictive, nonparametric methods (e.g., kernel regression, splines) relax the functional form. Semiparametric models like generalized additive models (GAMs) replace linear terms with smooth functions while retaining additive structure.

6 Computational aspects

6.1 Algorithms for fitting

OLS is typically solved via QR decomposition or singular value decomposition (SVD) for numerical stability. For high-dimensional data, coordinate descent is efficient for lasso and ridge. Regularized methods often use cross-validation to select the tuning parameter \(\lambda\).

6.2 Software implementations

6.2.1 R and Python packages

In R, the lm() function fits OLS models; glm() handles GLMs; lme4 for mixed models; glmnet for elastic net. In Python, statsmodels provides comprehensive linear model tools, while scikit-learn includes LinearRegression, Ridge, and Lasso.

6.2.2 Large-scale and online learning

For datasets exceeding memory or streaming scenarios, online algorithms (e.g., stochastic gradient descent) update parameter estimates incrementally. Libraries like scikit-learn's SGDRegressor and Apache Spark's LinearRegression support distributed computing for large-scale linear models.

7 Applications

7.1 Economics and social sciences

Linear models are widely used for estimating demand elasticities, analyzing wage determinants, and evaluating policy interventions. For example, multiple regression can control for confounding variables in observational studies. Econometric variants include instrumental variables and panel data models.

7.2 Biological and medical statistics

In biostatistics, linear models examine dose–response relationships, clinical trial outcomes (ANOVA), and gene expression studies (after appropriate transformations). Mixed models handle longitudinal data and hierarchical designs common in biomedical research.

7.3 Engineering and physical sciences

Engineers use linear regression for calibration, system identification, and quality control. In physics, linear models fit experimental data (e.g., Hooke’s law, Ohm’s law). The simplicity of linear relationships allows straightforward parameter interpretation and uncertainty quantification.