Principal Component Analysis (PCA) is a statistical technique used for dimensionality reduction while preserving as much variance in the data as possible. It transforms a set of possibly correlated variables into a smaller number of uncorrelated variables called principal components, ordered by the amount of variance they capture. PCA is widely applied in exploratory data analysis, feature extraction, noise filtering, and data visualization.
1 Mathematical foundations
1.1 Covariance matrix and eigenvalues
The covariance matrix \(\Sigma\) of a dataset with \(p\) variables is a \(p \times p\) symmetric matrix whose entry \(\sigma_{ij}\) measures the covariance between variable \(i\) and variable \(j\). The diagonal entries are the variances of each variable. PCA seeks linear combinations of the original variables that maximize variance, which leads to the eigenproblem for \(\Sigma\). The eigenvalues of \(\Sigma\) represent the variance explained by each principal component, and the eigenvectors define the directions (principal component axes).
1.2 Eigendecomposition
Eigendecomposition decomposes a square matrix into its eigenvectors and eigenvalues. For the covariance matrix \(\Sigma\), the eigendecomposition is \(\Sigma = Q \Lambda Q^{-1}\), where \(Q\) is an orthogonal matrix whose columns are the eigenvectors, and \(\Lambda\) is a diagonal matrix with the eigenvalues \(\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_p \ge 0\). The eigenvectors are the principal component loadings; the eigenvalues indicate the variance along each component.
1.2.1 Spectral theorem
The spectral theorem states that any real symmetric matrix (such as the covariance matrix) can be diagonalized by an orthogonal matrix. This ensures that the eigenvectors of \(\Sigma\) are mutually orthogonal and that \(\Sigma\) can be expressed as a sum of rank‑1 matrices: \(\Sigma = \sum_{i=1}^p \lambda_i \mathbf{v}_i \mathbf{v}_i^\top\). This property guarantees that the principal components are uncorrelated and provides a direct link between eigenvalues and variance decomposition.
1.2.2 Singular value decomposition (SVD)
The singular value decomposition of the centered data matrix \(X\) (size \(n \times p\)) is \(X = U S V^\top\). The columns of \(V\) are the right singular vectors, which are identical to the eigenvectors of \(X^\top X\) (the covariance matrix up to scaling). The singular values \(s_i\) relate to eigenvalues via \(\lambda_i = s_i^2 / (n-1)\). SVD is numerically more stable for computing PCA and is often used in practice, especially when \(n < p\).
1.3 Proportion of variance explained
The proportion of variance explained by the \(k\)-th principal component is \(\lambda_k / \sum_{i=1}^p \lambda_i\). The cumulative proportion up to the first \(m\) components is \(\sum_{k=1}^m \lambda_k / \sum_{i=1}^p \lambda_i\). This metric guides the number of components to retain, as it quantifies how much of the total variability is captured by a reduced set of dimensions.
2 Performing PCA
2.1 Data preprocessing
2.1.1 Centering and scaling
Centering subtracts the mean of each variable so that the data has zero mean in every dimension. Scaling divides each variable by its standard deviation (or another measure of spread) to give all variables equal weight. Centering is essential for PCA to reflect variance around the origin; scaling is recommended when variables are measured in different units or have widely different variances.
2.1.2 Handling missing values
PCA cannot be applied directly to incomplete data. Common approaches include omitting rows or columns with missing values, imputing missing entries (e.g., using the mean or a regression model), or using algorithms such as iterative PCA that estimate missing values during the decomposition.
2.2 Computing principal components
2.2.1 Using covariance matrix
After centering (and optionally scaling) the data matrix \(X\), compute the covariance matrix \(S = \frac{1}{n-1} X^\top X\). Perform eigendecomposition on \(S\) to obtain eigenvalues and eigenvectors. The principal components are then the projections of the centered data onto the eigenvectors: \(T = X V\), where \(V\) is the matrix of eigenvectors. The first component corresponds to the eigenvector with the largest eigenvalue.
2.2.2 Using correlation matrix
When variables are measured on different scales, PCA on the correlation matrix is equivalent to PCA on the standardized data. The correlation matrix \(R\) is the covariance matrix of the standardized variables. Using \(R\) ensures that each variable contributes equally, regardless of its original variance. The steps are identical: compute eigenvalues and eigenvectors of \(R\), then project the standardized data.
2.3 Choosing the number of components
2.3.1 Scree plot and elbow criterion
A scree plot displays eigenvalues (or variance explained) in descending order. The "elbow" point, where the curve levels off, suggests retaining components up to that point. This heuristic is subjective but widely used.
2.3.2 Cumulative variance threshold
A predetermined threshold (e.g., 80% or 90%) of total variance is specified. The minimum number of components needed to reach or exceed that threshold is retained. This approach provides a quantitative and interpretable rule but may still retain many components in high‑dimensional settings.
2.3.3 Kaiser rule
The Kaiser rule retains components with eigenvalues greater than 1 when using the correlation matrix (or greater than the average eigenvalue when using the covariance matrix). For a correlation matrix, an eigenvalue less than 1 indicates that the component explains less variance than a single original variable, suggesting it is not worth retaining.
3 Interpretation and visualization
3.1 Loading vectors and contribution
Loadings are the coefficients of the linear combination defining a principal component. For component \(k\), the loading vector is the eigenvector \(\mathbf{v}_k\). The magnitude of a loading indicates the contribution of the corresponding original variable to that component. Squared loadings sum to 1 per component. Variables with large absolute loadings on a component dominate its interpretation.
3.2 Biplots
A biplot overlays scores (projections of observations) and loadings (variables) in the same coordinate system, typically using the first two principal components. Observations appear as points; variables appear as vectors. The angle between variable vectors indicates correlation (acute angle = positive, obtuse = negative, right angle = zero). Distances between observation points reflect similarity. Biplots are useful for detecting patterns, clusters, and influential variables.
3.3 Principal component scores
Scores are the coordinates of the original observations in the new principal component space. For observation \(i\), the score on component \(k\) is the dot product of the centered (or standardized) observation vector with the eigenvector \(\mathbf{v}_k\). Scores allow for interpretation of individual observations relative to the main axes of variation and are used in subsequent analyses (e.g., regression, clustering).
4 Applications
4.1 Dimensionality reduction
PCA reduces the number of variables from \(p\) to a smaller number \(m\) while retaining most of the variance. This simplifies subsequent modeling, reduces overfitting, and speeds up computation. Common in genomics, finance, and image processing where datasets often have hundreds or thousands of variables.
4.2 Data compression and noise reduction
By keeping only the first \(m\) components and discarding the rest, one can reconstruct an approximation of the original data. High‑variance components capture the signal; low‑variance components often contain noise. Truncated PCA therefore serves as a noise filter and a compression technique, e.g., for image compression (eigenfaces).
4.3 Feature engineering
Principal components can be used as new features for machine learning models. These features are uncorrelated and ordered by variance, which can improve performance of algorithms that assume independent inputs (e.g., linear regression, Naive Bayes). PCA also helps avoid multicollinearity.
4.4 Visualization of high‑dimensional data
Scatter plots of the first two or three principal components allow humans to visually explore high‑dimensional datasets. This is one of the most common uses of PCA, enabling pattern recognition, cluster identification, and outlier detection in a low‑dimensional space.
4.5 Multivariate outlier detection
Outliers often have extreme scores on one or more principal components, especially on later components where they appear as points far from the bulk of the data. The squared prediction error (SPE) or the Hotelling’s \(T^2\) statistic derived from PCA scores are used to flag anomalous observations.
5 Extensions and variants
5.1 Sparse PCA
Sparse PCA modifies the optimization to obtain loadings that contain many zero entries, thereby improving interpretability. It is achieved by adding an \(L_1\) penalty (lasso) to the PCA objective. Sparse PCA is useful when the number of variables is very large and one desires a parsimonious set of contributing variables for each component.
5.2 Kernel PCA
Kernel PCA applies PCA in a high‑dimensional feature space induced by a kernel function (e.g., Gaussian RBF). It captures non‑linear structures in the data without explicitly transforming the observations. The kernel matrix is eigendecomposed, and the principal components become non‑linear combinations of the original variables.
5.3 Incremental PCA
Incremental PCA allows the sequential updating of principal components as new data arrives, without needing to recompute the full decomposition from scratch. It is particularly valuable for large datasets that cannot fit into memory or for streaming data. The algorithm maintains a running estimate of the mean and covariance (or uses SVD updates).
5.4 Robust PCA
Robust PCA aims to recover a low‑rank matrix (the principal components) in the presence of outliers or gross corruption. It decomposes the data matrix into a low‑rank part plus a sparse error matrix. This is achieved through optimization techniques such as the Principal Component Pursuit. Robust PCA is used in background subtraction, video surveillance, and fault detection.