1 Concept and Motivation
1.1 Geometric Intuition
A projection is a rule that maps elements from one space to another while preserving selected geometric features. For instance, projecting a point in the plane onto a line places the point on that line in a way that reflects a chosen notion of “closest” or “direction of collapse.” The part of the point’s information that is orthogonal (or otherwise designated) to the target set is discarded or compressed, whereas the remaining component is retained.
A related viewpoint treats projection as “shadowing”: imagine a light source and a target surface. The shadow of a point on the surface is a projection whose behavior depends on how the light rays are cast (perpendicular rays yield orthogonal projection; oblique rays yield other projections).
1.2 Projection in Linear Spaces
In linear algebra, projections are often linear maps that send every vector to a component lying in a designated subspace. A key concept is that applying the map twice has the same effect as applying it once—once a vector has been “collapsed” onto the target, further projection produces no change. This stability under iteration is formalized by idempotence in operator form.
When the underlying space is equipped with an inner product, orthogonality provides a natural way to decide which component to preserve. Without an inner product, one can still define projection through algebraic decompositions, though the geometric meaning changes.
1.3 Why Projections Matter in Applications
Projections provide a unifying mechanism in many applied tasks:
- Approximation: Replace data by the nearest (or best) representation in a restricted model space.
- Dimension reduction: Represent high-dimensional objects using fewer coordinates while keeping the most informative directions.
- Constraint handling: Enforce feasibility by mapping iterates back onto sets where solutions must lie.
- Computation: Convert difficult problems into simpler ones by working in a structured subspace.
Because projections preserve specific structures—length relationships in orthogonal cases, or decomposed components in algebraic cases—they appear naturally in least-squares estimation, numerical linear algebra, and many iterative algorithms.
2 Types of Projection
2.1 Orthogonal Projection
Orthogonal projection is defined using an inner product and is characterized by minimizing distance. The projected point is the unique point in the target set that is closest to the original point.
2.1.1 Projection onto a Line
Given a nonzero direction vector \(u\), the projection of a vector \(x\) onto the line spanned by \(u\) retains the component of \(x\) parallel to \(u\) and removes the perpendicular component. In Euclidean space, the result can be written using the normalized direction. Geometrically, the difference \(x - \mathrm{proj}_{u}(x)\) is orthogonal to the line.
2.1.2 Projection onto a Subspace
| For a subspace \(W\), orthogonal projection maps any vector \(x\) to the vector in \(W\) that minimizes \(\|x-y\|\) over \(y\in W\). Equivalently, the residual \(x-\mathrm{proj}_{W}(x)\) lies in the orthogonal complement \(W^\perp\). If an orthonormal basis for \(W\) is available, the projection is obtained by summing the basis directions weighted by corresponding inner products. |
|---|
2.2 Oblique (Non-orthogonal) Projection
Oblique projection uses a decomposition principle rather than a distance-minimizing one. Instead of dropping the component orthogonal to the target, it removes a component along a chosen direction.
2.2.1 General Definition via Direct Sum Decomposition
A common algebraic definition uses a direct sum: if the ambient space can be written as \(V = W \oplus U\), then every vector \(x\in V\) has a unique decomposition \(x = w + u\) with \(w\in W\) and \(u\in U\). The oblique projection onto \(W\) along \(U\) maps \(x\) to \(w\). This operator is linear and idempotent, but generally does not correspond to a closest-point rule.
2.2.2 Properties and Geometric Interpretation
Oblique projections still “collapse” along a specified complementary direction, but the residual is not generally perpendicular to the target subspace. As a result, geometric quantities such as angles and distances behave differently than in orthogonal projection. The operator depends sensitively on the chosen complement \(U\), and changing the direction of collapse changes both the projected point and the induced error decomposition.
2.3 Perspective vs. Parallel Projection (Graphics Context)
In computer graphics, projection describes how 3D points are mapped to a 2D screen. Parallel projection uses rays with fixed direction, while perspective projection uses rays emanating from a viewpoint, producing size changes with depth. Although these are not always framed as linear maps in the simplest coordinate systems, they reflect the same underlying idea: mapping from a higher-dimensional setting to a lower-dimensional representation by “collapsing” certain degrees of freedom.
3 Projection Operators in Linear Algebra
3.1 Idempotent Matrices and Projectors
In the operator language, a projection map \(P\) satisfies \(P^2 = P\). Such maps are called projectors (or projection operators). Idempotence captures the idea that once a vector has been projected, projecting again does not alter it.
3.1.1 Characterizations of Projection Operators
A linear map is a projector exactly when the space decomposes into the image and kernel of the operator in a direct-sum fashion. Concretely, for an idempotent \(P\), vectors split into those that remain unchanged by \(P\) (lying in the image) and those that are annihilated (lying in the kernel). In orthogonal projection, additional conditions ensure compatibility with the inner product, leading to self-adjointness.
3.2 Rank, Null Space, and Image
For a linear projection \(P\), the rank equals the dimension of the image (the retained component), while the null space (kernel) contains vectors that are completely removed by projection.
3.2.1 Relationships among Image and Kernel
For an idempotent operator \(P\), the image and kernel are complementary subspaces. This implies:
- Every vector splits uniquely into a part that survives and a part that vanishes.
- Dimension counts satisfy \(\mathrm{rank}(P) + \dim(\ker P) = \dim(V)\).
These relationships guide both theoretical analysis and algorithm design, since they reveal how much information a projection preserves.
3.3 Orthonormal Bases and Coordinate Forms
3.3.1 Projection Using Basis Expansion
When an orthonormal basis \(\{e_i\}\) spans the target subspace, orthogonal projection becomes especially simple: the projected vector is the sum of the basis vectors scaled by their inner products with the original vector. This coordinate form provides efficient computation and offers direct interpretability: each retained component corresponds to a specific direction in the subspace.
For general subspaces without an orthonormal basis, one can still compute projections by converting to a basis representation, often using matrix methods that effectively orthonormalize or solve for coefficients.
4 Computing Projections
4.1 Projection Using Inner Products
4.1.1 Closed-form Formulas in Euclidean Space
| In Euclidean space with a known direction or orthonormal basis, closed-form expressions are available. For a line spanned by \(u\), the projection can be written using the dot product and normalization by \(\|u\|^2\). For a subspace spanned by an orthonormal set, the projection is a sum of weighted basis vectors, where weights are inner products. These formulas avoid iterative procedures and are efficient when the structure is favorable. |
|---|
4.2 Least-Squares Interpretation
4.2.1 Projection onto Column Spaces
Orthogonal projection onto a column space arises naturally from least squares. If a matrix \(A\) has columns spanning a subspace \( \mathcal{C}(A)\), projecting a vector \(b\) onto \( \mathcal{C}(A)\) yields the best approximation to \(b\) among all vectors of the form \(Ax\). The projected vector is the fitted value in least-squares regression.
4.2.2 Residual Vectors and Normal Equations
The residual \(r=b-\hat{b}\) (where \(\hat{b}\) is the projection of \(b\)) is orthogonal to \( \mathcal{C}(A)\). This orthogonality condition leads to normal equations \(A^\top A x = A^\top b\) in the full-rank case. The geometry of projection thus translates into algebraic conditions for optimality.
4.3 Numerical Methods
4.3.1 Stability Considerations
Directly forming normal equations can be numerically sensitive because it squares the condition number of \(A\). In practice, algorithms aim to compute projections or least-squares solutions without explicitly forming \(A^\top A\), improving robustness in floating-point arithmetic.
4.3.2 Using QR Decomposition
QR decomposition expresses \(A\) as \(A=QR\) with \(Q\) orthonormal and \(R\) upper triangular. Least-squares solutions can be obtained by transforming the system into coordinates aligned with the orthonormal columns of \(Q\). This yields stable computation of projections onto column spaces.
4.3.3 Using SVD
Singular value decomposition \(A=U\Sigma V^\top\) provides a highly reliable framework, especially when \(A\) is ill-conditioned or rank-deficient. Projections and least-squares fits can be expressed through singular vectors and singular values, allowing truncation or regularization strategies. Although more expensive than QR, SVD often delivers the most accurate and interpretable results.
5 Projections in Data Analysis
5.1 Dimensionality Reduction via Subspace Projection
5.1.1 Principal Component Analysis (PCA) as Projection
PCA identifies directions in which the data vary the most. In the projection viewpoint, PCA selects a low-dimensional subspace and projects each data point onto it, producing a compressed representation. The principal components form an orthonormal basis for this subspace in the standard formulation, making the projection orthogonal and well-defined.
5.1.2 Projection onto Principal Component Subspaces
When the first \(k\) principal components span a subspace \(W_k\), the PCA embedding of a point is its orthogonal projection onto \(W_k\). Increasing \(k\) allows more variance to be retained, trading off fidelity against compactness. The reconstruction error is tied to the discarded components’ variance, providing a quantitative measure of information loss.
5.2 Feature Extraction and Embeddings
Beyond PCA, many embedding methods can be interpreted through projection onto learned or structured subspaces. Features become coefficients in a coordinate system that emphasizes certain patterns—such as smoothness, similarity, or consistency with a model class. Even when methods are described using other language (kernels, manifold ideas, or regression), projection onto a reduced representation is often the operative mechanism.
5.3 Error Metrics under Projection
Assessing a projection typically involves measuring how far the projected object deviates from the original. In orthogonal settings, squared Euclidean distance often plays a central role because it decomposes cleanly into retained and discarded components. In non-orthogonal contexts, error depends on the choice of projection directions or weighting, and care is required when interpreting residual magnitudes.
6 Projections in Optimization and Algorithms
6.1 Constraint Sets and Feasible-Point Projection
6.1.1 Projection onto Convex Sets (Overview)
In constrained optimization, a common strategy is to take an update step in an unconstrained manner and then map the result back to a feasible region. When the feasible set is convex and the projection is orthogonal, the nearest-point projection is unique and well-behaved. This provides a practical mechanism for maintaining feasibility without solving the constrained problem from scratch at every iteration.
6.2 Projected Gradient Methods
6.2.1 Algorithmic Loop and Step Projection
Projected gradient methods alternate between:
- computing a descent direction using the objective (often a gradient),
- taking a step, and
- projecting the candidate iterate onto the constraint set.
The projection ensures iterates remain feasible, while the gradient step drives progress toward an optimum. Performance depends on step size choices, the geometry of the constraint set, and properties such as Lipschitz continuity of the gradient.
6.3 Proximal/Projection Connections (High-Level)
Projection methods relate closely to proximal methods, which generalize the idea by solving a minimization that trades off closeness to the current iterate with satisfaction of a term in the objective. Under certain conditions, a proximal update can be viewed as a projection onto a set or as a projection with respect to a weighted metric. This connection helps unify algorithms used in signal processing, machine learning, and inverse problems.
7 Applications Across Disciplines
7.1 Computer Graphics and Rendering
7.1.1 Transform Pipelines and Screen Mapping
Rendering pipelines use a sequence of linear or affine transformations followed by a mapping from 3D to 2D. Projection determines how spatial coordinates become screen coordinates and affects how objects appear (e.g., scaling with depth in perspective rendering). Combined with clipping and depth buffering, projection is central to determining visibility and pixel placement.
7.2 Signal Processing and Filtering
7.2.1 Projection and Approximation of Signals
In signal processing, projection provides a principled way to approximate a signal by a simpler structure, such as a subspace spanned by chosen basis functions. The approximation that minimizes energy in the residual corresponds to an orthogonal projection under standard inner-product settings. This underlies techniques including denoising, compressive approximations, and subspace-based modeling.
7.3 Control and System Identification (Linear Context)
For linear system identification, measured outputs can be approximated using a model whose behavior spans a subspace. Fitting the model often reduces to a least-squares problem, where projection onto a column space yields the best explanation in that model class. In controller design, projection-like steps appear when enforcing actuator constraints or restricting parameter estimates to structured sets.
8 Practical Considerations and Common Pitfalls
8.1 Choosing the Right Projection Type
Selecting orthogonal versus oblique projection depends on the meaning of “error” in the application. Orthogonal projection aligns with minimizing distance under the chosen inner product, which is natural for many approximation tasks. Oblique projection may be appropriate when the decomposition has physical or algorithmic structure, but it can change how residuals relate to preserved features.
8.2 Effects of Non-orthogonality
Non-orthogonal projections can produce unintuitive behavior: retained components may interact with discarded components, and error norms may not decompose as cleanly. Algorithms that assume orthogonality can fail or lose interpretability if the projection used is oblique. Verifying geometric and algebraic assumptions is essential when translating theory into computation.
8.3 Dimensionality and Computational Cost
Projection onto a large subspace can be expensive, especially if bases are not precomputed or if repeated projections are required. Conversely, very small subspaces may yield poor approximations. Efficient implementations often rely on factorizations (QR, SVD) or precomputed orthonormal bases to reduce per-iteration cost.
8.4 Interpretation of Projected Results
A projection’s output represents a constrained or reduced representation, not the original quantity. In data analysis, projected embeddings may obscure aspects not captured by the subspace, while in optimization, projected iterates indicate feasibility rather than optimality. Correct interpretation requires attention to what was preserved (the target subspace or constraint structure) and what was intentionally removed.