1. Definition and basic formulation
Singular value decomposition (SVD) is a factorization that represents a real or complex matrix as a product of orthogonal/unitary matrices and a diagonal matrix of nonnegative numbers. It generalizes eigen-decomposition by applying to arbitrary (possibly non-square or rank-deficient) matrices.
1.1 SVD statement for real matrices
Let \(A\in\mathbb{R}^{m\times n}\). There exist orthogonal matrices \(U\in\mathbb{R}^{m\times m}\) and \(V\in\mathbb{R}^{n\times n}\), and a diagonal matrix \(\Sigma\in\mathbb{R}^{m\times n}\) with nonnegative entries such that \[ A = U\Sigma V^{T}. \] The diagonal entries of \(\Sigma\) are the singular values of \(A\).
1.2 SVD statement for complex matrices
For \(A\in\mathbb{C}^{m\times n}\), there exist unitary matrices \(U\in\mathbb{C}^{m\times m}\) and \(V\in\mathbb{C}^{n\times n}\), and a diagonal matrix \(\Sigma\in\mathbb{C}^{m\times n}\) with nonnegative real diagonal entries satisfying \[ A = U\Sigma V^{*}, \] where \(V^{*}\) denotes the conjugate transpose. The diagonal entries of \(\Sigma\) are again the singular values.
1.3 Interpretation of the factors: U, Σ, and V*
In the decomposition \(A=U\Sigma V^{*}\), the columns of \(U\) and \(V\) provide orthonormal bases for the domain and codomain subspaces involved in the action of \(A\). The diagonal matrix \(\Sigma\) scales each corresponding pair of singular directions by a nonnegative amount. The operator \(A\) can be viewed as: rotate/transform input coordinates into a singular-vector basis (via \(V^{*}\)), scale along those directions (via \(\Sigma\)), then map into the output basis (via \(U\)).
1.4 Properties of singular values (nonnegativity and ordering)
Singular values are always nonnegative real numbers. When listed in decreasing order, \[ \sigma_{1}\ge \sigma_{2}\ge \cdots \ge 0, \] they quantify the strength of the linear map along the corresponding singular directions. The number of strictly positive singular values equals the rank of \(A\). The largest singular value is closely tied to induced operator norms, while the smallest positive singular value influences stability and conditioning.
2. Geometric and algebraic interpretation
SVD provides both geometric insight and algebraic structure. It decomposes the action of a linear transformation into independent, orthogonally aligned scaling behaviors.
2.1 Action of a linear map on the singular vector bases
Let \(\{u_i\}\) and \(\{v_i\}\) be the left and right singular vectors corresponding to singular values \(\sigma_i\). Then \[ A v_i = \sigma_i u_i,\qquad A^{*}u_i=\sigma_i v_i. \] Thus, \(A\) maps each right singular vector to a left singular vector scaled by \(\sigma_i\). Components orthogonal to the span of the singular vectors associated with positive singular values lie in the null spaces.
2.2 Orthogonality and invariance relationships
Because \(U\) and \(V\) are orthogonal/unitary, singular vectors form orthonormal bases. The subspaces spanned by the left singular vectors associated with positive singular values are invariant under \(AA^{*}\), while those associated with positive singular values are also invariant under \(A^{*}A\) on the domain side. This produces a clean correspondence between SVD structure and spectral properties of the two derived Hermitian matrices.
2.3 Rank, null spaces, and image spaces via SVD
If \(A=U\Sigma V^{*}\) with singular values \(\sigma_1,\dots,\sigma_r>0\) and the rest zero, then:
- \(\operatorname{rank}(A)=r\).
- The null space of \(A\) is spanned by right singular vectors \(v_{r+1},\dots,v_n\).
- The image of \(A\) is spanned by left singular vectors \(u_1,\dots,u_r\).
- Similarly, the left null space is spanned by left singular vectors corresponding to zero singular values.
These relationships make SVD a convenient tool for analyzing solvability and projecting solutions.
3. Computation and algorithms
Practical use of SVD depends on robust numerical algorithms. Most implementations rely on transforming the original matrix to a simpler structured form and then extracting the diagonal singular values and singular vectors.
3.1 Reduction to bidiagonal form
A common approach first reduces \(A\) to a bidiagonal matrix \(B\) using orthogonal/unitary transformations: \[ A = U_{1} B V_{1}^{*}, \] where \(B\) has nonzero entries only on the main diagonal and one off-diagonal. This step preserves singular values and allows subsequent computations to focus on a smaller, highly structured matrix.
3.2 Diagonalization of the bidiagonal matrix
Once bidiagonal, specialized eigensolvers and iterative schemes compute the singular values and associated vectors of \(B\). Many algorithms use repeated orthogonal transformations (often variants of QR-like or divide-and-conquer strategies) tailored for bidiagonal inputs, yielding the singular values and the singular vectors of \(B\), and therefore of the original matrix.
3.3 Numerical stability considerations
SVD is widely regarded as numerically stable because it avoids subtractive cancellation inherent in some other factorizations and uses orthogonal/unitary transformations, which preserve norms. Still, finite precision affects convergence of iterative methods and the accuracy of singular vectors when singular values are clustered. Careful scaling and stopping criteria help maintain reliability.
3.4 Complexity and practical runtime considerations
The computational cost depends on matrix shape. For dense matrices, standard SVD routines scale roughly as \(O(mn\min(m,n))\). In practice, performance is influenced by implementations, hardware, and whether a “thin” SVD is sufficient. For very large problems, one often uses randomized or truncated methods (see also sparse and randomized SVD variants) to reduce runtime and memory.
4. Connections to other decompositions
SVD relates to several classical matrix factorizations and spectral results. These links clarify why SVD is broadly useful and how it fits into the landscape of matrix analysis.
4.1 Relationship to eigen-decomposition
Eigen-decomposition applies cleanly to square matrices and diagonalizable cases, but it does not directly handle non-square or rank-deficient operators. SVD can be interpreted as an eigen-decomposition of the symmetric/Hermitian matrices \(A^{*}A\) and \(AA^{*}\): \[ A^{*}A = V\Sigma^{T}\Sigma V^{*},\qquad AA^{*} = U\Sigma\Sigma^{T}U^{*}. \] Thus, singular values are square roots of the eigenvalues of these positive semidefinite matrices.
4.2 SVD and the spectral theorem for symmetric matrices
| When \(A\) is real symmetric, eigen-decomposition exists under standard conditions: \(A=Q\Lambda Q^{T}\). In that setting, singular values are \( | \lambda_i | \), and singular vectors can be chosen from the eigenvectors. SVD therefore extends the concept of diagonalization to a broader class of matrices, including those that are not diagonalizable or not even square. |
|---|
4.3 Relationship to QR and LU decompositions
QR and LU factorization produce triangular structures suited for solving linear systems and least squares. SVD, by contrast, exposes geometric scaling and orthogonal bases. In least-squares workflows, QR is often faster, but SVD can be preferable when matrices are ill-conditioned, when rank deficiency must be handled reliably, or when one needs explicit information about null spaces and effective rank.
4.4 Polar decomposition viewpoint
Polar decomposition expresses a matrix \(A\) as \[ A = QH, \] where \(Q\) is orthogonal/unitary and \(H\) is positive semidefinite Hermitian. SVD connects directly: one can take \(H = U\Sigma U^{*}\) and \(Q = UV^{*}\) (in a compatible dimension convention). This viewpoint emphasizes that SVD separates “rotation/reflection” from “stretching” in a structured way.
5. Least squares and pseudoinverses
SVD is a canonical tool for solving least-squares problems and for defining a stable generalized inverse that works even when the matrix is not invertible.
5.1 Moore–Penrose pseudoinverse from SVD
For \(A=U\Sigma V^{*}\), the Moore–Penrose pseudoinverse is \[ A^{\dagger} = V\Sigma^{\dagger}U^{*}, \] where \(\Sigma^{\dagger}\) is formed by taking reciprocals of nonzero singular values and transposing positions in the rectangular diagonal structure. This definition yields a generalized inverse that satisfies the Moore–Penrose conditions and behaves well under singularity and rank deficiency.
5.2 Minimum-norm least-squares solutions
Given \(Ax\approx b\), the SVD-based pseudoinverse produces the least-squares minimizer \(x^{\ast}=A^{\dagger}b\) with minimum Euclidean norm among all minimizers. Decomposing \(b\) in the left singular vector basis shows how components aligned with small singular values strongly influence \(x^{\ast}\), which is central to understanding regularization and truncation.
5.3 Handling overdetermined and underdetermined systems
| - Overdetermined case (\(m>n\)): least squares seeks \(x\) minimizing \(\|Ax-b\|\). SVD delivers minimizers even when \(A\) is rank deficient. |
|---|
- Underdetermined case (\(m<n\)): solutions may not be unique. The pseudoinverse selects the minimum-norm solution among all exact or least-squares solutions.
In both regimes, SVD provides a unified formula and clear geometric meaning.
5.4 Regularization and truncated SVD
Regularization mitigates instability caused by very small singular values. Two common strategies are:
- Tikhonov regularization, which modifies singular-value reciprocals with a damping factor.
- Truncated SVD, which keeps only singular values above a threshold and discards the rest. This reduces variance amplification at the cost of bias, often improving performance when data are noisy.
6. Low-rank approximation and optimality
One of SVD’s most powerful applications is constructing the best low-rank representation of a matrix under common error measures.
6.1 Best rank-k approximation (Eckart–Young–Mirsky)
Let \(A\) have singular values \(\sigma_1\ge \cdots \ge \sigma_r>0\). The truncated SVD \[ A_k = \sum_{i=1}^{k} \sigma_i u_i v_i^{*} \]
| has rank at most \(k\). The Eckart–Young–Mirsky theorem states that \(A_k\) is optimal: it minimizes \(\|A - X\|\) among all matrices \(X\) of rank \(\le k\) for both the spectral norm and the Frobenius norm. The optimal error equals: |
|---|
\[
| \|A-A_k\|_2 = \sigma_{k+1},\qquad \|A-A_k\|_F = \sqrt{\sum_{i>k}\sigma_i^2}. |
|---|
\]
6.2 Error bounds under common norms
The theorem provides explicit error formulas for the two most used norms:
- Spectral norm (largest singular value of the residual): governed by the next discarded singular value.
- Frobenius norm (energy of the residual): governed by the sum of squares of discarded singular values.
These results enable direct interpretation of how many singular components are required for a desired accuracy.
6.3 Truncation strategies and choice of k
Choosing \(k\) depends on the application and data characteristics. Strategies include:
- Selecting \(k\) to capture a target fraction of Frobenius energy.
- Using cross-validation to optimize predictive performance in modeling tasks.
- Employing noise-aware thresholds, such as retaining components significantly above estimated noise levels.
The practical choice often balances approximation quality against computational cost and generalization.
6.4 Applications to denoising and compression
If noise contributes mainly to small singular values, truncating the SVD can separate structure from perturbations, yielding denoised signals or images. In compression, storing only the leading singular triplets \((u_i, \sigma_i, v_i)\) reduces memory and bandwidth while preserving the dominant features of the original matrix.
7. Dimensionality reduction and data analysis
SVD underpins many modern data-analysis techniques by identifying directions that capture variability or structure.
7.1 Principal component analysis (PCA) via SVD
For a centered data matrix \(X\), PCA can be obtained through SVD. If \(X = U\Sigma V^{*}\), then the columns of \(V\) provide principal directions in feature space, while the squared singular values relate to the variance explained by each component. Truncating the SVD yields a low-dimensional projection that preserves maximal variance within the chosen subspace.
7.2 Latent structure and factor loading interpretation
In many applications, the entries of \(U\Sigma\) and \(V\) (or related scalings) are interpreted as latent factors and loadings. Each singular component corresponds to a pattern shared across rows and columns of the data. While these factors are not unique when singular values repeat, the subspaces they span capture the underlying dominant structure.
7.3 Whitening and scaling variants
Beyond standard PCA, SVD supports whitening: transforming data so that components have equal variance. Using singular values, one can rescale projected coordinates by functions of \(\Sigma\) (often inverse square roots) to achieve the desired covariance structure. Variants may also incorporate robust scaling or alternative weighting to emphasize particular features.
7.4 Applications in collaborative filtering
User-item rating matrices can be approximated as low rank. SVD-based collaborative filtering decomposes the matrix into latent user and item factors and predicts missing entries using the truncated reconstruction. This approach reduces sparsity challenges by relying on shared latent patterns rather than individual observations alone.
8. Applications in applied science and engineering
SVD appears across scientific computing because it simultaneously offers stable numerical behavior and interpretable geometric decomposition.
8.1 Solving inverse problems
Inverse problems often involve retrieving underlying parameters from indirect measurements, frequently modeled as \(Ax=b\). When \(A\) is ill-conditioned or not invertible, SVD supports regularized least-squares solutions. By controlling contributions from small singular values, one can stabilize recovery against measurement noise.
8.2 Image processing and SVD-based features
Images can be represented as matrices (e.g., grayscale intensity) or tensors reduced to matrix forms. SVD enables tasks such as:
- rank-based compression,
- denoising by truncation,
- feature extraction where singular vectors capture global patterns.
Beyond pixel-level processing, the singular components can serve as compact descriptors for classification or retrieval.
8.3 System identification and model reduction
In system identification, data-driven models aim to approximate the behavior of dynamical systems. For large-scale models, model reduction seeks a smaller system that preserves key input-output characteristics. SVD can be used to identify dominant modes and construct reduced-order models, especially when data are gathered in snapshot matrices amenable to low-rank approximation.
8.4 Signal processing perspectives
Signal processing often seeks representations that separate signal from noise or express signals in energy-efficient bases. SVD provides a structured approach to decomposing signals into orthogonal components, and truncated versions correspond to retaining dominant spectral-energy modes. This aligns naturally with denoising, subspace tracking, and related methods.
9. Theoretical considerations
Beyond computation and applications, SVD has subtle theoretical aspects related to uniqueness, conditioning, and perturbation behavior.
9.1 Uniqueness and ambiguity in singular vectors
Singular values are unique, but the corresponding singular vectors may not be. If a singular value is simple (isolated from neighbors), the associated singular vectors are determined up to sign (or phase in the complex case). If singular values are repeated, the singular vectors spanning that singular subspace can rotate without changing the product \(U\Sigma V^{*}\).
9.2 Degeneracy and repeated singular values
When \(\sigma_i=\sigma_{i+1}=\cdots=\sigma_{i+t}\), the individual vectors within that block are not uniquely identifiable from the factorization alone. Instead, only the subspace spanned by \(\{u_i,\dots,u_{i+t}\}\) and \(\{v_i,\dots,v_{i+t}\}\) is determined. This has implications for interpreting components as stable features.
9.3 Conditioning and sensitivity to perturbations
The sensitivity of singular values and vectors to perturbations depends on spectral gaps between singular values. Small gaps imply that nearby singular subspaces can change significantly under noise. Consequently, algorithms and downstream tasks that rely on singular vectors may be more stable when leading singular values are well separated.
9.4 Bounds using Weyl-type inequalities
| Weyl-type inequalities bound how much singular values can change when the matrix is perturbed. Such results typically relate \( | \sigma_i(A+E)-\sigma_i(A) | \) to the norm of the perturbation \(E\). These inequalities provide theoretical guarantees for robustness and motivate error control in applications. |
|---|
10. Variants and special cases
Numerous practical variants tailor SVD to computational constraints, sparsity, or problem-specific structure.
10.1 Thin (economy) SVD
The thin SVD computes only the nonzero or relevant portions of the factors. If \(A\in\mathbb{R}^{m\times n}\) with \(r=\operatorname{rank}(A)\), one can write \[ A = U_r \Sigma_r V_r^{T}, \] where \(U_r\in\mathbb{R}^{m\times r}\), \(\Sigma_r\in\mathbb{R}^{r\times r}\), and \(V_r\in\mathbb{R}^{n\times r}\). This reduces storage and can accelerate computations when full factors are unnecessary.
10.2 Full SVD and dimensional conventions
The full SVD uses square orthogonal/unitary matrices \(U\) and \(V\) of sizes \(m\times m\) and \(n\times n\). Different software may adopt different conventions for \(\Sigma\) dimensions and whether the decomposition is “economy” by default. Mathematically, all conventions describe the same underlying singular values and subspaces, differing only in how unused components are represented.
10.3 Sparse and randomized SVD methods
For large matrices, full SVD may be infeasible. Randomized SVD uses probabilistic sampling to approximate the leading singular subspaces with high probability, often dramatically reducing cost. Sparse-aware approaches exploit the sparsity pattern to avoid dense intermediate computations. Truncated methods focus on computing only the leading components needed for approximation or learning.
10.4 Weighted and generalized SVD formulations
In some problems, the geometry of the domain and codomain is nonstandard, leading to weighted least squares or generalized formulations. Weighted or generalized SVD incorporates positive definite weight matrices, effectively modifying the inner products under which optimality is measured. This yields decompositions compatible with the metric relevant to the application, improving interpretability and accuracy.