1 Definition and Core Properties

1.1 Geometric meaning: projection onto a subspace

Given a subspace \(S \subseteq \mathbb{R}^n\), the orthogonal projection maps any vector \(x\) to a point in \(S\) that is closest to \(x\) under the Euclidean distance. The projected vector \( \hat{x} \) satisfies that the “residual” \(x-\hat{x}\) is orthogonal to every vector in \(S\). In geometry, this produces the perpendicular foot from \(x\) to the subspace.

1.2 Algebraic characterization: symmetry and idempotence

When expressed as a matrix \(P\), an orthogonal projection onto \(S\) is characterized by two defining properties (with respect to the standard inner product):

Symmetry reflects that the mapping respects orthogonality, while idempotence encodes that projecting twice yields the same result as projecting once: once the vector lies in \(S\), reapplying the operator changes nothing.

1.3 Relationship to best approximation (least squares)

Orthogonal projection gives the best approximation within a subspace. If \(S\) is the set of vectors representable in a model, then \(Px\) is the unique element of \(S\) minimizing \(\|x - s\|\) over all \(s \in S\). This is the geometric core behind least-squares estimation: the residual is orthogonal to the model space, ensuring optimality.

2 Construction of the Projection Matrix

2.1 Projection onto the span of columns (full column rank)

Let \(A \in \mathbb{R}^{n \times k}\) have full column rank, and consider the subspace \(S = \text{col}(A)\). The orthogonal projector onto \(S\) is \[ P = A(A^\top A)^{-1}A^\top. \] This formula is valid because \(A^\top A\) is invertible when \(A\) has full column rank. It produces a symmetric idempotent matrix that projects any \(x\) onto the column space of \(A\).

2.2 Projection using an orthonormal basis

If \(Q \in \mathbb{R}^{n \times k}\) has orthonormal columns spanning \(S\) (so \(Q^\top Q = I_k\)), then the projector is particularly simple: \[ P = QQ^\top. \] The orthonormality eliminates the need for any inverse and improves numerical robustness. Conceptually, \(QQ^\top x\) forms the expansion coefficients in the basis and reconstructs the component within \(S\).

2.3 Projection using the Moore–Penrose pseudoinverse

If \(A\) is not full rank, one uses the Moore–Penrose pseudoinverse \(A^+\). The orthogonal projection onto \(\text{col}(A)\) is \[ P = AA^+. \] This construction remains well-defined even when \(A^\top A\) is singular. The resulting matrix is still symmetric and idempotent and projects onto the same subspace determined by the columns of \(A\).

2.4 Computing with matrix factorization (QR, SVD)

In practice, the projector is computed via stable factorizations rather than explicit inverses.

  • QR factorization: If \(A = QR\) with \(Q\) orthonormal and \(R\) upper triangular, then for full column rank \(P = QQ^\top\). When rank-deficient, one typically uses a rank-revealing QR or truncation aligned with the effective column space.
  • SVD: With \(A = U\Sigma V^\top\), the orthogonal projector onto \(\text{col}(A)\) can be built by keeping singular vectors corresponding to nonzero singular values. This approach is robust in the presence of near-dependence and clarifies rank structure.

3 Idempotent and Spectral Structure

3.1 Eigenvalues and invariance of the subspace

For an orthogonal projector \(P\), the eigenvalues are restricted to \(\{0,1\}\). Vectors in the subspace \(S\) are fixed points of the map: if \(x \in S\), then \(Px=x\), indicating eigenvalue \(1\). Vectors orthogonal to \(S\) map to zero, indicating eigenvalue \(0\). Thus the spectrum directly encodes which directions are retained.

3.2 Null space and orthogonal complement

The kernel and range relate cleanly:

  • Range: \(\text{range}(P) = S\)
  • Null space: \(\text{null}(P) = S^\perp\)

This follows from orthogonality: components orthogonal to the subspace are eliminated by the projection, while components within the subspace remain unchanged.

3.3 Trace and dimensionality interpretation

Because \(P\) has eigenvalues \(0\) and \(1\), the trace satisfies \[ \text{tr}(P) = \text{rank}(P) = \dim(S). \] The trace counts how many independent directions are preserved by the projection. This provides a quick way to verify whether the computed projector corresponds to the intended subspace dimension.

4 Projection onto Lines, Planes, and General Subspaces

4.1 Rank-one (projection onto a vector)

For a line spanned by a nonzero vector \(u\), the orthogonal projector is \[ P = \frac{uu^\top}{u^\top u}. \]

If \(u\) is normalized (\(\|u\|=1\)), then \(P = uu^\top\). This matrix maps any \(x\) to the component of \(x\) parallel to \(u\), removing the orthogonal component.

4.2 Rank-two examples (planes in Euclidean space)

In \(\mathbb{R}^3\), a plane through the origin is a two-dimensional subspace. If \(Q \in \mathbb{R}^{3 \times 2}\) has orthonormal columns spanning the plane, then \[ P = QQ^\top \] projects onto the plane, while the component along the plane’s normal direction is discarded. In such settings, it is common to construct \(Q\) from two orthogonal (or orthonormalized) direction vectors lying in the plane.

4.3 General k-dimensional subspaces

For a subspace of dimension \(k\) with orthonormal basis \(Q \in \mathbb{R}^{n \times k}\), the projector takes the universal form \[ P = QQ^\top. \] This formulation generalizes rank-one and rank-two cases, emphasizing that only the subspace (not the specific basis choice) matters. Different orthonormal bases for the same subspace lead to identical \(P\).

4.4 Numerical example workflows

A typical workflow is:

  1. Form a matrix \(A\) whose columns span the desired subspace.
  2. Compute a factorization (QR or SVD).
  3. Extract an orthonormal basis \(Q\) for the column space.
  4. Form \(P = QQ^\top\) or compute the action \(Px\) without explicitly assembling \(P\).

In numerical tasks like least squares, it is often more efficient to compute \(Px\) via solves with triangular factors or by projecting through the factorization rather than forming \(P\) as a dense matrix.

5 Applications in Applied Mathematics

5.1 Least-squares regression and normal equations

In linear regression, one often models data \(y\) as \(Ax\) plus noise, and chooses parameters by minimizing \(\|y - Ax\|^2\). The fitted values are \( \hat{y} = A(A^\top A)^{-1}A^\top y = Py \), where \(P\) is the orthogonal projector onto \(\text{col}(A)\). This connects the geometry of projection with the algebra of least-squares solutions.

5.2 Residuals and orthogonality in estimation

The residual \(r = y - \hat{y}\) satisfies \(A^\top r = 0\), meaning the residual is orthogonal to the column space of \(A\). This orthogonality condition is both a characterization of optimality and a diagnostic: it implies that no direction within the model space can reduce the squared error.

5.3 Data fitting and model reduction

Projection is used to reduce dimensionality by mapping data onto a lower-dimensional model space. In model reduction, one replaces the original variables with coordinates in a subspace capturing dominant behavior (for example, via principal components). The projected representation preserves aspects aligned with the chosen subspace while discarding orthogonal components.

5.4 Feature projection in signal processing

Many signal-processing pipelines include projecting signals onto subspaces associated with basis functions, filters, or signal components of interest. Orthogonal projection can separate a signal into components attributed to a chosen subspace and its orthogonal complement, enabling denoising, reconstruction, or analysis in transform domains.

6 Computational Considerations

6.1 Stability vs. direct formulae

While \(P = A(A^\top A)^{-1}A^\top\) is theoretically standard, computing it via explicit inverses can be numerically unstable. Forming \(A^\top A\) squares the condition number, amplifying roundoff errors. Stable implementations typically use QR or SVD to obtain an orthonormal basis and then form \(P\) as \(QQ^\top\) or apply it indirectly.

6.2 Handling rank deficiency and conditioning

Rank deficiency changes the effective dimension of the subspace and affects which directions are reliably captured. With pseudoinverses or truncated SVD, small singular values are treated carefully—either exactly (when truly zero) or approximately (when below a tolerance). This yields projectors onto the numerically meaningful column space and avoids unstable behavior caused by nearly dependent columns.

6.3 Efficiency for large-scale problems (sparse/iterative methods)

For large \(n\), explicitly constructing \(P\) can be expensive in memory and time. Instead, applications compute the projected vector \(Px\) using factorizations or iterative methods that exploit structure, such as sparsity or matrix-vector products. When only projection of multiple vectors is needed, one may reuse factorization results to reduce repeated computation.

6.4 Verifying projection behavior in practice

Common checks include:

  • Symmetry check: \(P^\top \approx P\) within tolerance.
  • Idempotence check: \(P^2 \approx P\).
  • Orthogonality of residual: for regression-style setups, verify \(A^\top (x - Px) \approx 0\).

Because floating-point arithmetic introduces small errors, these tests use tolerances rather than exact equality.

7.1 Oblique projections vs. orthogonal projections

An oblique projection maps onto a subspace along directions that are not orthogonal to it. In contrast to orthogonal projectors, oblique projectors are not symmetric in general and may have more complicated spectral properties. They still satisfy idempotence \(P^2=P\) when defined as a projection operator, but the defining orthogonality condition is replaced by geometric constraints tied to the chosen direction.

7.2 Projections in weighted inner products

In a weighted setting, orthogonality is defined using an inner product \(\langle x, y\rangle_W = x^\top Wy\), where \(W\) is symmetric positive definite. The corresponding projector depends on \(W\) and yields different matrices than in the unweighted case. Such formulations appear naturally in generalized least squares and other applications where noise variance or scaling varies across components.

7.3 Projection in Hilbert spaces and discretizations

The concept extends beyond finite-dimensional Euclidean spaces. In Hilbert spaces, orthogonal projection is defined onto closed subspaces, and the minimizing property of best approximation remains valid. In numerical analysis, discretizations approximate the continuous operator, often preserving projector behavior approximately while controlling approximation and stability.

8 Worked Derivations

8.1 Deriving the matrix formula from orthogonality conditions

Let \(S=\text{col}(A)\) with full column rank. Seek \(\hat{x}\in S\) of the form \(\hat{x}=Ac\) such that \(x-\hat{x}\) is orthogonal to \(S\). Orthogonality to \(\text{col}(A)\) means \(A^\top(x-Ac)=0\). Solving gives \((A^\top A)c = A^\top x\), so \(c=(A^\top A)^{-1}A^\top x\). Therefore \(\hat{x}=A(A^\top A)^{-1}A^\top x\), and the projector is \(P=A(A^\top A)^{-1}A^\top\).

8.2 Verifying idempotence and symmetry

With \(P=A(A^\top A)^{-1}A^\top\):

  • Symmetry follows because \((A(A^\top A)^{-1}A^\top)^\top = A(A^\top A)^{-1}A^\top\), using that \(A^\top A\) is symmetric.
  • Idempotence can be checked by substitution:

\[ P^2 = A(A^\top A)^{-1}A^\top A(A^\top A)^{-1}A^\top = A(A^\top A)^{-1}(A^\top A)(A^\top A)^{-1}A^\top = P. \] These identities confirm that the operator corresponds to a true orthogonal projection in exact arithmetic.

8.3 Alternative derivation via optimization (minimization viewpoint)

Consider the optimization problem \(\min_{s\in S}\|x-s\|^2\). Writing \(s=Ac\), this becomes \(\min_c \|x-Ac\|^2\). Differentiating the objective with respect to \(c\) and setting the gradient to zero yields the normal equations \(A^\top(Ac-x)=0\). The solution \(c\) leads directly to \(\hat{x}=Ac=Px\). This connects the projector to the geometry of least squares through optimality conditions.

9 Common Pitfalls and Interpretation Guides

9.1 Confusing projection matrix with general linear transforms

A projection matrix is not merely any linear map. It must satisfy the projector properties (notably idempotence) and, for orthogonal projection, symmetry. A generic linear transform may distort directions, change lengths, and shift subspace components in ways that do not preserve the “closest point” interpretation.

9.2 Using non-orthonormal bases incorrectly

If a basis for the subspace is not orthonormal, the simple formula \(P=QQ^\top\) does not apply. Using it with a non-orthonormal \(Q\) produces an incorrect operator that does not project orthogonally. The correct approach requires either orthonormalizing the basis or using the pseudoinverse-based construction.

9.3 Interpreting numerical errors and tolerance checks

Finite precision means computed projectors may not satisfy \(P^2=P\) exactly. Interpreting deviation as “wrong” can be misleading. Tolerance thresholds should be based on the scale of the problem and expected rounding errors, and residual orthogonality should be assessed relative to norms to avoid false alarms.

9.4 Mistaking projection direction for coordinate changes

Projection is a geometric operation on vectors, not a change of coordinates. While one may represent projected vectors using different coordinate systems, the projector itself is an operator defined by how it maps each original vector to its closest point in the subspace. Confusing these two can lead to incorrect conceptual interpretations of results.