Principal Component Analysis (PCA) is a statistical technique used for dimensionality reduction, feature extraction, and data visualization. It transforms a set of possibly correlated variables into a smaller number of uncorrelated variables called principal components, which are ordered by the amount of variance they capture from the original data. PCA is widely applied in fields such as machine learning, genetics, finance, and image processing to simplify complex datasets while retaining essential patterns and structures.
1.1 Covariance Matrix and Eigendecomposition
The covariance matrix summarizes pairwise covariances between variables in a dataset. For a data matrix X with *n* observations and *p* variables, the covariance matrix Σ is a *p* × *p* symmetric matrix. Eigendecomposition of Σ yields eigenvectors (directions of maximum variance) and eigenvalues (magnitude of variance along those directions). The eigenvectors are orthogonal and form the principal component axes.
1.2 Orthogonal Projections
PCA projects original data onto a lower‑dimensional subspace defined by the top *k* eigenvectors. The projection is an orthogonal linear transformation; each projected point is the linear combination of original variables with weights given by the eigenvector entries. Orthogonality ensures that the new axes are uncorrelated.
1.3 Variance Maximization
The first principal component is the direction that maximizes the variance of the projected data. Subsequent components maximize variance under the constraint of being orthogonal to all previous components. This sequential variance maximization is equivalent to solving the eigenvalue problem of the covariance matrix.
2.1 Data Standardization
Before applying PCA, variables are often centered (subtract the mean) and scaled (divide by standard deviation). Standardization ensures that variables with larger scales do not dominate the analysis. This step is crucial when variables are measured in different units.
2.2 Computing the Covariance Matrix
After standardization, the covariance matrix (or correlation matrix) of the centered data is computed. For a dataset with *p* variables, this matrix is *p* × *p* and symmetric.
2.3 Eigenvalue and Eigenvector Decomposition
The covariance matrix is decomposed into eigenvalues and eigenvectors. Each eigenvector corresponds to a principal component direction, and its associated eigenvalue indicates the variance explained by that component. The eigenvectors are sorted in descending order of eigenvalues.
2.4 Selecting Principal Components
The number of principal components to retain can be based on a predefined threshold of explained variance, the “elbow” in a scree plot, or domain knowledge. A common rule is to keep components that cumulatively explain 70–90% of the total variance.
2.4.1 Explained Variance Ratio
The explained variance ratio for component *i* is λᵢ / ∑λⱼ, where λᵢ is the eigenvalue. It quantifies the proportion of total variance captured by that component. Cumulatively summing these ratios helps decide the number of components to retain.
2.5 Projecting Data onto New Axes
The original standardized data matrix is multiplied by the matrix of selected eigenvectors (principal component loadings). The result is a reduced‑dimensional representation, where each observation is expressed in terms of the principal component scores.
3.1 Orthogonality of Components
All principal components are orthogonal (uncorrelated) in the original variable space. This property simplifies interpretation and avoids multicollinearity issues in subsequent analyses.
3.2 Variance and Information Retention
The total variance of the original data equals the sum of all eigenvalues. By retaining the top *k* components, PCA preserves as much variance as possible for a given number of dimensions. However, variance may not always equate to “information” in a task‑specific sense.
3.3 Loadings and Feature Contributions
The entries of each eigenvector are called loadings. They represent the contribution (weight) of each original variable to a principal component. High absolute loadings indicate variables that strongly influence that component, aiding interpretation.
3.4 Limitations of PCA
PCA assumes linear relationships and is sensitive to outliers. It also requires the data to be approximately Gaussian for optimal interpretation. Additionally, PCA treats all variables equally and may discard low‑variance components that could be important for certain tasks (e.g., classification).
4.1 Dimensionality Reduction
PCA reduces the number of features in high‑dimensional datasets, making subsequent modeling faster and less prone to overfitting. It is a common preprocessing step in fields such as genomics, where the number of variables often exceeds the number of samples.
4.2 Data Visualization
By projecting data onto two or three principal components, PCA enables visual exploration of high‑dimensional structures, clusters, and outliers.
4.2.1 Scree Plot
A scree plot displays eigenvalues (or explained variance ratios) in descending order. The “elbow” point—where the curve levels off—suggests a natural cutoff for the number of components to retain.
4.2.2 Biplot
A biplot overlays the data points (as scores) and the original variable vectors (as loadings) in the same coordinate system. It helps visualize which variables contribute most to the principal components and how observations relate to them.
4.3 Noise Reduction
Retaining only the first few principal components can filter out noise that is spread across many low‑variance components. This is used in image denoising, signal processing, and other applications where variance correlates with signal.
4.4 Feature Extraction for Machine Learning
Principal components can serve as new input features for regression, classification, or clustering algorithms. By decorrelating features and reducing dimensionality, PCA often improves model performance and training speed.
5.1 Kernel PCA
Kernel PCA applies the kernel trick to perform PCA in a high‑dimensional feature space without explicitly computing coordinates. It captures nonlinear structures by using kernel functions such as the radial basis function (RBF) or polynomial kernel.
5.2 Sparse PCA
Sparse PCA modifies the optimization problem to produce loadings that have many zero entries. The resulting components are easier to interpret because they involve only a subset of the original variables.
5.3 Incremental PCA
Incremental PCA (IPCA) processes data in mini‑batches, making it suitable for very large datasets that cannot fit into memory. It approximates the eigendecomposition in an online manner.
5.4 Robust PCA
Robust PCA (RPCA) decomposes the data matrix into a low‑rank component (the “clean” data) and a sparse component (outliers or anomalies). It is resilient to gross corruptions, unlike standard PCA.
6.1 Implementation in Python (scikit‑learn)
The sklearn.decomposition.PCA class provides a convenient implementation. It supports SVD‑based computation, automatic centering, and methods such as fit_transform and explained_variance_ratio_. Key parameters include n_components (number of components to keep) and whiten (to normalize component variances).
6.2 Computational Efficiency for Large Datasets
Standard eigendecomposition of the covariance matrix scales as O(*p*³) in the worst case, where *p* is the number of variables. For datasets with many observations but few variables, SVD on the data matrix directly (O(*np*²)) is more efficient. For high‑dimensional, low‑sample data, the “economy” SVD or random‑projection methods are used.
6.3 Relationship with Singular Value Decomposition (SVD)
PCA is mathematically equivalent to SVD of the centered data matrix. The right singular vectors of X are the principal component directions, the singular values are related to eigenvalues (λ = s² / (*n*-1)), and the left singular vectors give the principal component scores. SVD‑based computation is numerically more stable than eigendecomposition of the covariance matrix.
7.1 PCA on the Iris Dataset
The Iris dataset (150 samples, 4 features) is often used to demonstrate PCA. After standardization, the first two principal components explain about 95% of the total variance. A scatter plot of the first two components clearly separates the three Iris species, showing that PCA effectively captures the distinguishing patterns.
7.2 PCA in Image Compression
PCA can compress a set of similar images (e.g., faces in a face‑recognition system) by extracting a basis of eigenimages (principal components). Projecting each image onto the first *k* components and reconstructing it yields an approximation that reduces storage space while retaining key features. A classic example is the “Eigenfaces” method.
7.3 PCA for Financial Portfolio Analysis
In finance, PCA is used to reduce the dimensionality of asset returns. The first principal component often represents a “market” factor, while subsequent components capture sector‑specific or style‑related movements. Portfolio managers use these factors to understand risk exposures and to construct hedged portfolios.