1 Foundations of least squares
1.1 Residuals and the sum of squared errors
Given observed data points and a chosen model, the *residual* is the difference between an observed value and the corresponding value predicted by the model. Least squares selects the model parameters to make these residuals as small as possible, using the criterion that sums the squared residuals. Squaring serves two purposes: it penalizes larger deviations more heavily and removes sign cancellation.
1.2 The least squares objective function
For a model with predicted outputs depending on parameters, the basic objective function is the *sum of squared errors* (often abbreviated SSE). Minimizing SSE yields parameter values that best match the data in a least-squares sense. In practice, the model class may be linear in parameters, nonlinear, or restricted by constraints, which changes how the objective is optimized.
1.3 Geometric interpretation (projection)
When the model is linear in parameters, least squares has a clean geometric meaning. The predicted values form a vector in a space spanned by the model’s basis functions. The least-squares solution corresponds to the orthogonal projection of the observed data vector onto the model’s column space, producing the closest attainable approximation under the usual Euclidean norm.
1.4 Assumptions and typical use cases
The classical formulation often assumes measurement errors have roughly constant variance and are not systematically biased with respect to the model. Under these assumptions, least squares provides a natural estimator and a computationally convenient optimization problem. Typical uses include fitting trends to noisy measurements, estimating regression coefficients, and finding parameters in numerical simulation and scientific data analysis.
2 Linear least squares
2.1 Model in matrix form
| In linear least squares, the model predictions are written as \( \hat{y} = X\beta \), where \(X\) is a design matrix whose columns represent basis functions (or features), \(\beta\) is the parameter vector, and \(y\) is the observed data vector. The residual is \(r = y - X\beta\), and the least-squares problem seeks \(\beta\) minimizing \(\|y - X\beta\|_2^2\). |
|---|
2.2 Derivation of the normal equations
Minimizing the quadratic objective leads to linear conditions on \(\beta\). Differentiating the SSE with respect to \(\beta\) and setting the result to zero yields the *normal equations*, typically expressed as \(X^T X \beta = X^T y\).
2.2.1 Full-rank case and closed-form solution
If \(X\) has full column rank, then \(X^T X\) is invertible. The normal equations then produce a unique closed-form solution: \[ \hat{\beta} = (X^T X)^{-1} X^T y. \] This solution is algebraically direct but may be numerically sensitive when \(X^T X\) is ill-conditioned.
2.2.2 Rank-deficient case and non-uniqueness
When columns of \(X\) are linearly dependent, \(X^T X\) becomes singular, and the least-squares objective may have infinitely many minimizers. In that setting, one often selects a particular solution such as the minimum-norm solution, commonly obtained via the singular value decomposition or related techniques.
2.3 Normal equations vs. direct minimization
Although normal equations are derived from optimization, solving them directly by forming \(X^T X\) can magnify numerical errors. Direct minimization using more stable matrix decompositions avoids squaring the condition number and typically yields better reliability, especially for large or ill-conditioned problems.
2.4 Connection to projection onto column space
The normal-equation solution ensures that the residual vector is orthogonal to every column of \(X\). Equivalently, the predicted vector \(X\hat{\beta}\) is the orthogonal projection of \(y\) onto the column space of \(X\). This characterization also provides a route to understanding how goodness-of-fit relates to geometry.
3 Basis functions and polynomial approximation
3.1 Polynomial least squares fitting
A common approximation uses polynomials, with basis terms \(1, x, x^2, \dots\). Least squares then chooses coefficients so the polynomial values best match measured data points in the SSE sense. Polynomial fitting is widely used because it is flexible and easy to implement, but its practical performance depends strongly on scaling and choice of basis.
3.2 Orthogonal polynomials and stability
Using orthogonal (or near-orthogonal) basis functions reduces numerical interference between terms. When basis functions are orthogonal under an appropriate inner product, coefficient estimation becomes more stable, and the resulting computation often suffers less from ill-conditioning than naive monomial bases.
3.3 Scaling and conditioning considerations
Even with a correct least-squares formulation, the numerical behavior can degrade if the input variable magnitudes are large or unevenly scaled. Centering and scaling the independent variables, as well as using numerically stable solvers, often improves conditioning and reduces sensitivity to floating-point rounding.
3.4 Choosing a function space (bias–variance intuition)
Selecting the function space involves trade-offs. A richer space (higher-degree polynomials or more basis functions) can reduce approximation error but increases sensitivity to noise and can worsen generalization. Less expressive spaces may miss structure but remain more robust. This bias–variance tension motivates careful selection of basis size and complexity.
4 Computation and numerical methods
4.1 Solving normal equations directly
One approach is to form \(X^T X\) and solve the resulting linear system. While conceptually simple, this method may be less stable because it can amplify conditioning issues. It can still be useful for small, well-conditioned problems or as a baseline method.
4.2 QR decomposition approach
A widely used alternative is the QR decomposition of \(X\). By rewriting the least-squares problem in terms of an orthogonal-triangular factorization, one can compute the minimizer without forming \(X^T X\). QR methods are typically more numerically stable and are common in implementations of linear least squares.
4.3 Singular value decomposition (SVD)
The SVD decomposes \(X\) into orthogonal factors and a diagonal matrix of singular values. It provides a robust framework for handling rank deficiency and for computing minimum-norm solutions. In addition, singular values offer insight into effective model rank and numerical stability.
4.4 Regularized solvers (ridge-type)
| Regularization modifies the objective to discourage overly large parameter norms. A typical ridge-type objective adds a penalty proportional to \(\|\beta\|_2^2\), improving conditioning and reducing variance at the cost of introducing bias. Regularization is especially helpful when features are highly correlated or when data are limited relative to model complexity. |
|---|
5 Weighted and generalized least squares
5.1 Weighted least squares formulation
When different observations have different error magnitudes, ordinary least squares treats all residuals equally. Weighted least squares assigns weights inversely related to error variance, effectively scaling residuals so that noisier measurements influence the fit less.
5.2 Estimating with heteroscedastic errors
In heteroscedastic settings, residual variance changes across observations. If the variance structure is known or well-estimated, weighting improves efficiency and yields more appropriate parameter estimates. If the variance model is uncertain, iterative reweighting or robust estimation can be used as practical compromises.
5.3 Generalized least squares with correlated errors
If errors are correlated across observations, a simple weighting by individual variances is insufficient. Generalized least squares accounts for the full covariance structure of the noise, leading to an objective that measures residuals in a transformed metric aligned with the covariance.
5.4 Whitening transformation and covariance modeling
A common computational strategy is to apply a whitening transformation that converts correlated errors into approximately uncorrelated ones with equal variance. This reduces generalized least squares to an ordinary least-squares problem in the transformed variables, making implementation more straightforward when covariance information is available.
6 Statistical interpretation
6.1 Maximum likelihood under Gaussian noise
Least squares has a direct connection to statistical estimation. Under the assumption that errors are independent and Gaussian with constant variance, maximizing the likelihood of the observed data is equivalent to minimizing the sum of squared residuals. This equivalence motivates least squares as both an optimization method and a statistical estimator.
6.2 Estimation of parameters and uncertainty
Beyond point estimates, statistical interpretation seeks uncertainty quantification. The covariance of the parameter estimates can be derived from the design matrix (and an estimate of error variance). This informs standard errors and supports approximate confidence intervals for parameters.
6.3 Residual diagnostics and fit quality measures
Residuals can be analyzed to assess whether the model captures systematic patterns. Common diagnostic ideas include checking whether residuals are randomly scattered, comparing residual magnitudes across regions of the predictor space, and using goodness-of-fit metrics derived from SSE and related quantities.
6.4 Degrees of freedom and model complexity
Model complexity influences how much residual reduction can be achieved by chance. The effective degrees of freedom relate to the number of fitted parameters and the rank of the design matrix. This concept is important when comparing models with different sizes so that improvements are not mistaken for meaningful structure.
7 Model evaluation and selection
7.1 Overfitting and underfitting in approximation
Overfitting occurs when a model captures noise rather than underlying signal, often yielding good training fit but poorer performance on new data. Underfitting arises when the model class is too restrictive to represent the data’s structure. Least squares solutions can be accurate on training data yet still fail to generalize if complexity is not controlled.
7.2 Cross-validation for choosing approximation space
Cross-validation assesses how a model trained on one subset performs on held-out data. By repeating the process across folds, one can estimate the expected prediction error for different approximation spaces (e.g., different polynomial degrees or different basis sizes). This guides selection toward a balance between fit and generalization.
7.3 Information criteria (AIC/BIC style overview)
Information criteria provide an approximate trade-off between goodness-of-fit and model complexity. They typically combine a fit term derived from the likelihood or SSE with a penalty related to the number of parameters. While assumptions vary, these criteria offer a convenient alternative to repeated resampling.
7.4 Train–test performance concepts
A practical workflow often uses a train–test split: the model is fit on training data and then evaluated on a separate test set. Comparing error measures across the two sets helps detect whether the model is overly tuned to the training observations. Consistent performance indicates that the approximation captures stable structure.
8 Variants and extensions
8.1 Constrained least squares
Constrained least squares restricts the solution to satisfy additional conditions such as bounds on parameters, linear equality constraints, or monotonicity requirements. These constraints change the geometry of the feasible set and typically require specialized solvers, but they can encode domain knowledge and improve interpretability.
8.2 Total least squares (errors in variables)
In many measurement problems, both the dependent and independent variables contain noise. Total least squares addresses this by finding a perturbation to the entire data matrix that minimizes a global error measure. Unlike ordinary least squares, which assumes predictors are exact, total least squares adjusts for uncertainties in all variables.
8.3 Nonlinear least squares and iterative methods
When the model is nonlinear in parameters, the residuals depend on parameters through nonlinear functions. The objective remains the sum of squared residuals, but solving it typically requires iterative optimization methods such as Gauss–Newton or Levenberg–Marquardt. These algorithms linearize the model locally and update parameters until convergence.
8.4 Robust alternatives (overview of Huber-style ideas)
Least squares is sensitive to outliers because squared errors grow rapidly with residual magnitude. Robust methods modify the loss function to reduce the influence of extreme residuals. Huber-type approaches interpolate between quadratic behavior for small residuals and more linear or bounded penalties for large residuals.
9 Practical applications
9.1 Curve fitting in measurement data
In instrumentation and experimental science, measured outputs often contain noise and systematic deviations. Least squares provides a standard mechanism for fitting a theoretical curve (polynomial trends, exponentials, or other parameterized models) to noisy observations, enabling estimation of quantities of interest.
9.2 Regression as approximation
Regression can be viewed as approximation: a model family is used to represent an unknown relationship between inputs and outputs. Ordinary least squares regression is the linear-in-parameters case, while generalized forms handle transformed outcomes, weighted data, or regularization for high-dimensional settings.
9.3 Data smoothing and denoising intuition
Smoothing can be interpreted as fitting a structured approximation that balances fidelity to data with suppression of noise. Regularization and basis selection are common tools for this goal, encouraging smoother fitted functions or smaller parameter magnitudes to prevent noise overreaction.
9.4 Least squares in scientific computing
Beyond direct data analysis, least squares appears in numerical simulation tasks such as parameter estimation, inverse problems, and solving overdetermined systems. Its ubiquity stems from converting many practical fitting or equilibrium conditions into tractable optimization problems, frequently leveraging matrix decompositions.
10 Common pitfalls and best practices
10.1 Multicollinearity and unstable solutions
When predictor columns are highly correlated, the design matrix can be close to rank deficient. This can lead to large parameter swings that still produce similar fitted values. Regularization, careful feature design, and stable solvers (e.g., SVD or QR) help mitigate these issues.
10.2 Extrapolation vs. interpolation risks
Least squares provides a best fit within the span supported by the data. Extrapolation beyond the observed range is often unreliable, especially for high-degree polynomial fits or when the model class does not reflect underlying physics. Interpolation can also be risky if the basis is poorly conditioned, but it is generally less problematic than extrapolation.
10.3 Outliers and sensitivity
Because squared residuals emphasize large errors, a few outliers can dominate the solution. It is advisable to inspect residual plots, use robust alternatives when appropriate, and consider data cleaning or measurement reassessment before relying on the fitted parameters.
10.4 Choosing units, scaling, and numerical tolerances
Using appropriate units and scaling improves numerical stability and interpretability. Additionally, solver tolerances affect convergence and rank decisions, particularly in floating-point computations. Practical implementations should use stable algorithms, consistent data preprocessing, and diagnostics to verify that the solution behaves as expected.