1 Definition and intuition of a PCA subspace
A PCA subspace is the linear subspace spanned by the leading principal components of a dataset. It is commonly used as a low-dimensional approximation space: data are projected onto this subspace to retain the dominant variation while discarding less informative directions.
1.1 Centering, covariance, and principal directions
In standard PCA, each feature dimension is centered by subtracting the sample mean. This centering makes the covariance matrix capture how variables vary together around the origin. Principal directions are the axes along which the centered data exhibit the greatest spread, ordered from most to least variance.
1.2 From eigenvectors to a spanned subspace
Let \( \{v_1,\dots,v_k\} \) denote the top \(k\) principal directions (eigenvectors corresponding to the largest eigenvalues of the covariance matrix). The PCA subspace of dimension \(k\) is \[ \mathcal{S}_k = \text{span}\{v_1,\dots,v_k\}. \] Geometrically, \(\mathcal{S}_k\) is the \(k\)-dimensional “best-fit” plane through the origin (in the centered coordinate system) for approximating the dataset.
1.3 Dimensionality choice and interpretation (k components)
The parameter \(k\) determines how much structure is retained. Small \(k\) yields a compact representation that emphasizes the dominant modes of variation, while large \(k\) approaches the original dimensionality. In practice, \(k\) is selected to balance approximation quality, noise suppression, computational cost, and interpretability.
2 Mathematical formulation
PCA subspaces are defined and computed through eigen-decomposition or singular value decomposition (SVD) of centered data. Once the top directions are identified, projections and reconstruction follow directly.
2.1 Eigen-decomposition of the covariance matrix
Given centered data matrix \(X \in \mathbb{R}^{n\times d}\), the sample covariance (up to a scaling convention) is \[ C = \frac{1}{n}X^\top X. \] Eigen-decomposition yields eigenvalues and eigenvectors that rank directions by variance.
2.1.1 Constructing the subspace basis from top eigenvectors
If \(C v_i = \lambda_i v_i\) with \(\lambda_1 \ge \lambda_2 \ge \cdots \ge 0\), then the PCA basis for the \(k\)-dimensional subspace is \[ V_k = [v_1\ \cdots\ v_k] \in \mathbb{R}^{d\times k}. \] The PCA subspace is \(\mathcal{S}_k = \{V_k z : z \in \mathbb{R}^k\}\).
2.1.2 Orthogonal projector onto the PCA subspace
Projection onto an orthonormal basis is linear and corresponds to an orthogonal projector. With \(V_k\) having orthonormal columns, \[ P_k = V_k V_k^\top, \] and for a centered point \(x\), the projected representation is \[ \hat{x} = P_k x. \] The residual component orthogonal to the subspace is \(r = x - \hat{x}\).
2.2 SVD formulation for computation
Because eigendecomposition of a \(d\times d\) covariance matrix can be expensive when \(d\) is large, PCA is often computed via SVD of the centered data matrix itself.
2.2.1 Relationship between SVD and principal components
Let the thin SVD of \(X\) be \[ X = U \Sigma W^\top, \] where \(\Sigma\) contains singular values \(\sigma_1 \ge \cdots \ge 0\), and columns of \(W\) are right singular vectors. Then \[ C = \frac{1}{n}X^\top X = W \left(\frac{\Sigma^2}{n}\right) W^\top. \] Thus, the principal directions are the columns of \(W\), and the eigenvalues are proportional to \(\sigma_i^2\). The top-\(k\) right singular vectors define the same PCA subspace as the top-\(k\) eigenvectors of \(C\).
2.3 Projection, reconstruction, and residuals
PCA uses the subspace to approximate points and to quantify how much variability is omitted.
2.3.1 Reconstruction error as discarded variance
Given centered \(x\), the reconstruction from the PCA subspace is \(\hat{x}=P_k x\). The residual \(r=x-\hat{x}\) lies in the orthogonal complement of \(\mathcal{S}_k\). Averaged over the dataset, the expected squared reconstruction error corresponds to the sum of discarded eigenvalues (up to the same covariance scaling convention). Consequently, removing low-variance directions reduces noise-like components but introduces approximation error tied to those eigenvalues.
3 Subspace selection and explained variance
Selecting \(k\) is central: it determines the balance between retained signal and compressed representation.
3.1 Variance explained by each component
Because principal components diagonalize the covariance in the principal basis, the variance along component \(i\) is proportional to \(\lambda_i\). The total variance of the centered data is proportional to \(\sum_{i=1}^d \lambda_i\), and the contribution of component \(i\) is quantified relative to this sum.
3.2 Choosing k using cumulative variance thresholds
A common approach chooses the smallest \(k\) such that cumulative explained variance reaches a target level, for example \(90\%\) or \(95\%\): \[ \frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^d \lambda_i} \ge \tau. \] This heuristic ties subspace size to an explicit retention criterion: larger \(\tau\) yields less information loss but a higher-dimensional embedding.
3.3 Scree plots and elbow-style heuristics
A scree plot displays eigenvalues \(\lambda_i\) (often in decreasing order) or explained variance by component. An “elbow” region—where the slope changes—suggests diminishing returns from adding further components. While not a formal guarantee, this visual method is frequently used when no fixed threshold is preferred.
4 Properties and guarantees
PCA subspaces come with mathematical structure: they are optimal under squared-error objectives and behave predictably under transformations.
4.1 Optimality for least-squares reconstruction
For a chosen dimension \(k\), the PCA subspace minimizes the total squared reconstruction error among all \(k\)-dimensional linear subspaces of the centered space. Equivalently, it provides the best rank-\(k\) approximation to the data in the Frobenius norm sense. This optimality is the main justification for using PCA rather than arbitrary projection bases.
4.2 Orthogonality and basis invariance within the subspace
The columns of \(V_k\) are orthonormal, so directions inside the subspace are mutually perpendicular in the principal basis. Moreover, while the basis vectors themselves may not be unique when eigenvalues are repeated (or nearly equal), the subspace spanned by the corresponding eigenvectors is invariant: any orthonormal basis of that span yields the same projector \(P_k\).
4.3 Effects of scaling and whitening on the subspace
PCA is sensitive to how features are measured. Scaling features changes the covariance structure and therefore can alter the principal directions and the resulting subspace. In contrast, whitening transforms the data to equalize variances across components; when applied before PCA (or conceptually through related preprocessing), it changes the relative weighting of directions and can produce different subspaces than those obtained from raw covariance. These effects underline that a PCA subspace is partly a property of both the dataset and the chosen representation.
5 Practical considerations
Real datasets introduce numerical and statistical issues that affect computation and interpretation of PCA subspaces.
5.1 Numerical stability and rank-deficient data
If the centered data lie near a lower-dimensional manifold, the covariance matrix may be ill-conditioned or rank-deficient. In such cases, small eigenvalues can be dominated by numerical noise. Robust computation via SVD helps because it directly handles rank and returns singular values in a stable manner. Truncating at an appropriate rank can also prevent unstable directions from being included.
5.2 Handling missing data (overview-level approaches)
Standard PCA requires complete data for straightforward covariance computation. For missing values, common strategies include iterative imputation (filling missing entries based on current low-rank estimates), expectation-maximization style PCA, matrix completion formulations, or algorithms that operate on observed entries without explicitly imputing all missing values. These methods typically aim to approximate the same underlying low-rank structure implied by PCA subspaces, but exact equivalence to classic PCA holds only under restrictive assumptions.
5.3 Outliers and robust PCA variants (high-level mention)
Outliers can strongly influence the covariance and hence the PCA subspace, because squared-error objectives overweight extreme deviations. Robust PCA approaches attempt to separate sparse outlier effects from a low-rank component, yielding a subspace less sensitive to anomalous points. While the details vary across methods, the goal is usually to preserve the dominant structure while preventing a few aberrant observations from steering the principal directions.
6 Applications of PCA subspaces
PCA subspaces are widely used wherever dimensionality reduction, denoising, or compact representation is helpful.
6.1 Dimensionality reduction for visualization and modeling
By projecting data into a lower-dimensional subspace, PCA enables compact embeddings for visualization (e.g., 2D or 3D plots) and as a preprocessing step before modeling. Reduced dimensions can simplify subsequent learning tasks, mitigate multicollinearity, and speed up algorithms sensitive to feature count.
6.2 Feature extraction and compression
The coordinates of projected points in the PCA basis provide new features: component scores. These features often capture correlated structure more effectively than the original variables, especially when the covariance exhibits strong eigenvalue decay. Keeping only the top components also reduces storage and computational load.
6.3 Denoising via projection onto dominant components
When noise is approximately isotropic or contributes mostly to low-variance directions, projecting onto the dominant PCA subspace can suppress that noise. The result is a denoised estimate \(\hat{x}\) that retains large-scale structure while trimming fluctuations associated with smaller eigenvalues.
6.4 Subspace tracking (brief overview of dynamic settings)
In streaming or time-evolving contexts, recalculating PCA from scratch may be expensive. Subspace tracking methods update the PCA subspace incrementally as new data arrive, attempting to track changes while maintaining computational efficiency. These approaches are useful in applications where the underlying patterns drift gradually over time.