1 Concept

Least squares is a method for choosing values of unknown parameters so that a model fits observed data as closely as possible. The usual criterion is to minimize the sum of the squared differences, or residuals, between measured values and the corresponding values predicted by the model. Because squaring makes large deviations count more heavily than small ones, the method tends to favor solutions that reduce prominent errors.

The approach is used whenever a problem can be expressed in terms of measurements, approximations, and adjustable parameters. It is especially effective when exact agreement is impossible because of noise, limited data, or modeling simplifications.

1.1 Definition

In its basic form, least squares seeks the parameter values that minimize an objective function built from squared residuals. If the observed data are denoted by values \(y_i\) and the model predictions by \(f_i\), then the residual for each observation is \(y_i - f_i\). Least squares chooses the parameters that make the total of these squared differences as small as possible.

The method may be applied to linear models, where predictions depend linearly on the unknown parameters, or to nonlinear models, where the dependence is more complicated. In either case, the central idea is the same: reduce the overall discrepancy between data and model.

1.2 Historical background

Least squares was developed in the early nineteenth century in connection with astronomy, geodesy, and orbit calculation, where scientists needed a systematic way to combine imperfect measurements. It became associated with the work of mathematicians such as Adrien-Marie Legendre and Carl Friedrich Gauss, who used related ideas for estimating unknown quantities from observational data.

Over time, the method spread into statistics and applied mathematics. Its appeal came from both practical success and strong mathematical properties, including connections to geometry, probability, and linear algebra. Today it is a standard tool across scientific and engineering disciplines.

1.3 Geometric interpretation

Geometrically, least squares can be understood as finding the point in a model space that lies closest to the observed data point in an ordinary Euclidean sense. For linear problems, the fitted values are often the orthogonal projection of the data vector onto a subspace formed by the model variables.

This viewpoint helps explain why the method is so widely used. The best fit is not merely the one that looks closest in a casual sense; it is the one that minimizes the squared length of the error vector. The geometry also clarifies why linear algebra methods are so effective for solving least-squares problems.

1.4 Error minimization

Least squares does not eliminate error; instead, it distributes it in a way that produces the best overall fit according to the chosen criterion. By squaring residuals, the method penalizes larger deviations more strongly than smaller ones, which often leads to stable and smooth estimates.

This feature is useful when errors are expected to be random and centered near zero. However, it can also make the method sensitive to unusual observations, since a few large deviations may have a disproportionate influence on the result.

2 Types of least squares

Several forms of least squares have been developed to address different data structures and error patterns. The main distinctions depend on how the observations are weighted, how the errors are correlated, and whether the model is linear or nonlinear in its parameters.

2.1 Ordinary least squares

Ordinary least squares is the simplest and most widely used version. It assumes that each residual is treated equally and that the best solution is the one minimizing the unweighted sum of squared errors.

This form is common in basic regression analysis and linear modeling. Its popularity comes from its simplicity, closed-form solutions in many cases, and clear statistical interpretation under standard assumptions.

2.2 Weighted least squares

Weighted least squares assigns different importance to different observations. Data points with greater reliability or smaller measurement variance receive larger weights, while less trustworthy observations contribute less to the final objective.

This approach is useful when the error variance changes from one observation to another. By adapting the influence of each point, weighted least squares can produce estimates that better reflect the quality of the available data.

2.3 Generalized least squares

Generalized least squares extends the method to situations where errors are correlated or have nonuniform covariance. Instead of assuming independent and identically distributed residuals, it accounts for the full error structure through a covariance matrix.

This form is especially valuable in time series, spatial data, and other settings where observations are not independent. By incorporating dependence among errors, the method can yield more accurate parameter estimates than ordinary least squares.

2.4 Nonlinear least squares

Nonlinear least squares applies when the model depends nonlinearly on the unknown parameters. Because the objective function is then generally more difficult to minimize, exact closed-form solutions are rarely available.

Instead, iterative algorithms are often used to improve an initial guess step by step. Nonlinear least squares is widely used in scientific modeling, calibration, and fitting of curves that do not follow a straight-line relationship.

3 Mathematical formulation

The mathematical description of least squares is usually written in terms of a model, a set of observations, and an objective function. The aim is to find parameter values that minimize the discrepancy between the observed data and the model output.

3.1 Linear least squares

In linear least squares, the predicted values are linear combinations of unknown parameters. This structure allows the problem to be expressed compactly and, in many cases, solved efficiently with matrix methods.

3.1.1 Matrix notation

Using matrix notation, the model is written as \(Ax \approx b\), where \(A\) is the design matrix, \(x\) is the vector of unknown parameters, and \(b\) is the observation vector. The residual vector is \(b - Ax\), and the least-squares problem is to minimize its squared Euclidean norm.

This notation is widely used because it captures many applications in a single framework. It also makes the connection to linear algebra explicit and enables the use of powerful computational techniques.

3.1.2 Normal equations

The normal equations are obtained by setting the gradient of the least-squares objective to zero. For a linear system, this leads to \(A^T A x = A^T b\), which characterizes the minimizing solution under appropriate conditions.

Although the normal equations provide a simple theoretical description, they are not always the most numerically stable way to compute the solution. In practice, alternative factorizations are often preferred for difficult or ill-conditioned problems.

3.2 Constrained least squares

Constrained least squares incorporates additional conditions that the solution must satisfy. These may be equality constraints, inequality constraints, or structural restrictions on the unknown parameters.

Such constraints arise in calibration, optimization under physical laws, and problems where certain parameters must remain nonnegative or satisfy conservation rules. The method then balances fit quality with adherence to the imposed requirements.

3.3 Least-squares objective function

The objective function in least squares is typically a sum of squared residuals, possibly with weights or other modifications. Its shape determines how the optimization proceeds and what type of solution is obtained.

Because the function is quadratic in linear problems, it often has a clear minimum and favorable mathematical properties. In nonlinear settings, the objective may have several local minima, making the optimization process more challenging.

4 Solution methods

Least-squares problems can be solved by direct algebraic techniques or by iterative numerical algorithms. The best method depends on the size of the system, the structure of the model, and the numerical properties of the data.

4.1 Direct methods

Direct methods aim to compute the solution in a finite sequence of algebraic steps. They are often efficient for moderate-sized problems and are commonly used when high accuracy is needed.

4.1.1 Gaussian elimination

Gaussian elimination transforms the system into an upper-triangular form that can be solved by back substitution. For least squares, the method may be applied to the normal equations or adapted to related formulations.

Although conceptually straightforward, this approach may suffer from numerical instability when the problem is poorly conditioned. For that reason, it is often replaced by more robust factorization methods in practical computing.

4.1.2 QR decomposition

QR decomposition factors the design matrix into an orthogonal matrix and an upper-triangular matrix. This makes it possible to solve least-squares problems without explicitly forming the normal equations.

The method is valued for numerical stability and is a standard technique in scientific computing. Its use of orthogonal transformations helps preserve accuracy when the data matrix is nearly singular or difficult to handle.

4.1.3 Singular value decomposition

Singular value decomposition, or SVD, expresses a matrix as a product of orthogonal factors and a diagonal matrix of singular values. It provides a very detailed picture of the problem’s numerical structure.

SVD is especially useful for ill-conditioned or rank-deficient systems. Because it reveals small singular values directly, it can help identify directions in which the data provide little information.

4.2 Iterative methods

Iterative methods improve an initial guess gradually until the solution converges to an acceptable level of accuracy. They are often used for large-scale or nonlinear problems where direct methods are too costly.

4.2.1 Gradient-based methods

Gradient-based methods move in directions that reduce the objective function. They include steepest descent and related schemes that use local derivative information to update the parameters.

These methods are simple and flexible, but they may converge slowly if the problem is poorly scaled or the surface of the objective function is shallow in some directions. Their main advantage is applicability to a broad range of optimization tasks.

4.2.2 Gauss-Newton method

The Gauss-Newton method is designed for nonlinear least squares. It approximates the objective function near the current estimate by linearizing the model and then solving a sequence of linear least-squares subproblems.

This method is often effective when the initial guess is reasonably close to the solution. Its efficiency comes from using the local structure of the problem rather than full second-derivative information.

4.2.3 Levenberg-Marquardt method

The Levenberg-Marquardt method combines features of Gauss-Newton and gradient descent. It adjusts the step size and direction to improve convergence, especially when the problem is difficult or the current estimate is far from the minimum.

Because it is robust and adaptable, this method is widely used for nonlinear curve fitting and parameter estimation. It often performs well in practice even when simpler methods struggle.

5 Statistical interpretation

Least squares has a strong statistical meaning when the observed data are viewed as noisy measurements of an underlying relationship. Under common assumptions, the method provides estimates with desirable properties and a clear probabilistic interpretation.

5.1 Relationship to maximum likelihood

When measurement errors are independent and normally distributed with constant variance, least squares coincides with maximum likelihood estimation. In that setting, minimizing the sum of squared residuals is equivalent to maximizing the probability of observing the given data.

This relationship explains much of the method’s popularity in statistics. It connects a simple computational rule to a principled inferential framework.

5.2 Estimation of variance

After fitting a least-squares model, the remaining residuals can be used to estimate the variance of the error term. This estimate helps quantify how much unexplained variation remains after the model has been applied.

Variance estimation is important for assessing uncertainty and for comparing fitted models. It also plays a role in standard error calculations and hypothesis testing.

5.3 Confidence intervals

Least-squares estimates can be accompanied by confidence intervals that describe the uncertainty in the fitted parameters. These intervals are derived from the estimated variance and the sampling distribution of the estimators.

In linear models with suitable assumptions, confidence intervals provide a practical way to summarize the range of plausible parameter values. They are commonly used in regression reporting and scientific analysis.

5.4 Assumptions and limitations

The standard statistical theory relies on assumptions such as independent errors, constant variance, and correct model specification. When these conditions fail, least-squares estimates may still be useful, but their interpretation can change.

A further limitation is sensitivity to outliers and model mismatch. If the data contain unusual observations or the chosen model is too simplistic, the resulting fit may be misleading.

6 Applications

Least squares appears in many fields because it offers a general strategy for fitting models to data. Its usefulness spans both theoretical analysis and practical computation.

6.1 Regression analysis

In regression analysis, least squares is used to estimate relationships between dependent and independent variables. It is one of the standard tools for building predictive models and describing trends in data.

The method is especially important in linear regression, where the fitted line or hyperplane summarizes the data in a compact form. It also serves as a foundation for many more advanced statistical techniques.

6.2 Curve fitting

Curve fitting uses least squares to match a chosen function to observed points. The fitted curve may represent a physical law, an empirical approximation, or a smoothed trend through noisy measurements.

This application is common in experimental science and engineering. The method helps extract meaningful parameters from imperfect data, such as growth rates, decay constants, or calibration coefficients.

6.3 Signal processing

In signal processing, least squares is used to estimate signal parameters, remove noise, and design filters. It can help separate desired structure from random fluctuations in recorded data.

The technique is useful in applications such as spectral estimation, channel modeling, and data reconstruction. Its algebraic form makes it suitable for both offline analysis and real-time computation.

6.4 Control and system identification

Control theory and system identification use least squares to infer dynamic models from input-output measurements. The method can estimate system parameters, transfer functions, or state-space representations.

By fitting a model to observed behavior, engineers can analyze stability, tune controllers, and predict future responses. Least squares is particularly valuable when the underlying system is only partially known.

6.5 Machine learning

In machine learning, least squares appears in linear models, feature selection pipelines, and approximation tasks. It is often used as a baseline method because it is simple, fast, and easy to interpret.

The method also underlies more elaborate learning procedures, including regularized regression and certain training objectives for predictive models. Its connection to optimization makes it a building block in modern data analysis.

7 Numerical considerations

Practical computation with least squares requires attention to accuracy, stability, and problem structure. Even when the mathematics is simple, numerical issues can strongly affect the quality of the result.

7.1 Stability and conditioning

A least-squares problem is said to be well conditioned when small changes in the data lead to small changes in the solution. Poor conditioning can amplify measurement noise and rounding error, making the fitted parameters unreliable.

Stable algorithms, such as QR decomposition and SVD, are often preferred because they reduce the risk of numerical distortion. Good scaling of the variables can also improve performance.

7.2 Overdetermined systems

An overdetermined system has more equations than unknowns, so an exact solution may not exist. Least squares provides a natural compromise by selecting the parameters that minimize the total mismatch.

This case is the most common in data fitting, where many observations are used to estimate a relatively small number of unknowns. The method finds the best approximate solution in a precise algebraic sense.

7.3 Underdetermined systems

An underdetermined system has fewer equations than unknowns, so infinitely many exact solutions may exist. In such cases, least squares alone may not identify a unique answer unless additional criteria are introduced.

To resolve this ambiguity, practitioners often impose constraints or regularization. These extra conditions help select one solution among many possible fits.

7.4 Regularization

Regularization modifies the least-squares problem by adding penalties or constraints that discourage unstable or overly complex solutions. This can improve robustness when the data are noisy or the model is ill posed.

Common regularization strategies reduce sensitivity to overfitting and help control the size or structure of the fitted parameters. They are widely used in statistics, machine learning, and inverse problems.

8 Variants and extensions

Many extensions of least squares address specific modeling goals or data challenges. These variants often adjust the objective function to better reflect prior knowledge or practical constraints.

8.1 Ridge regression

Ridge regression adds a penalty on the size of the coefficients to the least-squares objective. This tends to shrink parameter estimates and reduce variance.

It is useful when predictors are highly correlated or when the model risks instability. The added penalty often improves predictive performance by trading a small amount of bias for greater robustness.

8.2 Lasso

Lasso is a least-squares variant that uses an absolute-value penalty rather than a squared one. A notable effect is that some coefficients may be driven exactly to zero.

Because of this feature, lasso is often used for variable selection as well as estimation. It is especially helpful when a sparse model is desired.

8.3 Robust regression

Robust regression changes the loss function so that extreme observations have less influence than they do in ordinary least squares. This makes the fit less sensitive to outliers and heavy-tailed noise.

Such methods are preferred when unusual data points may distort the result. They can provide a more reliable summary of the dominant pattern in the data.

8.4 Total least squares

Total least squares allows errors in both the dependent and independent variables. This differs from ordinary least squares, which typically assumes that only the observations on one side of the equation are noisy.

The method is useful when all measured quantities contain uncertainty. It is especially relevant in calibration and experimental settings where input and output variables are both approximate.

Least squares is closely connected to several broader mathematical ideas. These relationships help explain its behavior and its role in applied analysis.

9.1 Projection methods

Projection methods seek the best approximation to a target by projecting it onto a simpler space. Least squares is a particularly important example because it often corresponds to orthogonal projection onto the span of the model columns.

This connection gives a geometric explanation for the fitted solution and its residual structure. It also links least squares to approximation theory and functional analysis.

9.2 Pseudoinverse

The pseudoinverse generalizes the idea of matrix inversion to cases where the matrix is not square or not invertible. It provides a compact way to express least-squares solutions, especially in linear problems.

Using the pseudoinverse can clarify the structure of solutions in rank-deficient or underdetermined systems. It is a central concept in modern linear algebra and numerical computation.

9.3 Optimization theory

Least squares is a fundamental example in optimization theory. It illustrates how objective functions are minimized, how constraints alter solutions, and how local and global behavior can differ.

Because of its simplicity and broad usefulness, least squares serves as a standard test case for algorithms and theory alike. Many methods in numerical optimization were developed or refined with least-squares problems in mind.

</INTERNAL_LINK_CANDIDATES> Least squares objective function (the summed-square error criterion minimized by the method) Residual (the difference between observed and predicted values) Ordinary least squares (the unweighted basic form of least squares) Weighted least squares (a form that assigns different importance to observations) Generalized least squares (a form that accounts for correlated errors) Nonlinear least squares (least squares applied to nonlinear parameter models) Design matrix (the matrix of predictors in a linear model) Normal equations (the equations obtained by setting the least-squares gradient to zero) QR decomposition (a stable matrix factorization used to solve least-squares problems) Singular value decomposition (a factorization that reveals rank and conditioning) Gauss-Newton method (an iterative method for nonlinear least squares) Levenberg-Marquardt method (a hybrid iterative method for nonlinear least squares) Maximum likelihood estimation (a statistical framework linked to least squares under normal errors) Confidence interval (an interval estimate for a parameter) Regularization (a modification that stabilizes or constrains the solution) Ridge regression (least squares with an \(L^2\) penalty) Lasso (least squares with an \(L^1\) penalty) Robust regression (a fitting approach less sensitive to outliers) Pseudoinverse (a generalized matrix inverse used in least-squares solutions) Projection method (an approximation approach based on projecting onto a subspace)