1 Preliminaries: Symmetric/Hermitian Matrices

1.1 Eigenvalues, eigenvectors, and spectral decomposition

Let \(A\) be a real symmetric matrix (\(A^\top=A\)) or a complex Hermitian matrix (\(A^*=A\)). Such matrices have real eigenvalues and admit an orthonormal basis of eigenvectors. In the symmetric/Hermitian setting, the spectral theorem guarantees a decomposition \[ A = \sum_{i=1}^n \lambda_i\, u_i u_i^*, \] where \(\{u_i\}\) is an orthonormal basis (orthonormal in \(\mathbb{R}^n\) or \(\mathbb{C}^n\)) and \(\lambda_i\in\mathbb{R}\).

1.2 Rayleigh quotient and quadratic forms

Given \(x\neq 0\), the Rayleigh quotient of \(A\) at \(x\) is \[ \rho_A(x)=\frac{x^*Ax}{x^*x}. \] When \(A\) is symmetric/Hermitian, the numerator is a real number and \(\rho_A(x)\) equals the value of a normalized quadratic form. Eigenvectors satisfy \(\rho_A(x)=\lambda\) precisely when \(x\) is an eigenvector with eigenvalue \(\lambda\).

1.3 Subspace language and orthogonality

Many extremal statements are formulated using subspaces of \(\mathbb{R}^n\) or \(\mathbb{C}^n\). A subspace \(S\) has an orthogonal complement \(S^\perp\), consisting of vectors orthogonal to every vector in \(S\). For subspaces of the same dimension, orthonormal bases can be used to describe constraints like \(x\in S\) or \(x\perp S\), which is central to the theorem’s min–max structure.

1.4 Ordering conventions for eigenvalues

Eigenvalues are ordered increasingly or decreasingly depending on the formula. A common convention is \[ \lambda_1 \le \lambda_2 \le \cdots \le \lambda_n. \] With this convention, “the \(k\)-th eigenvalue” refers to \(\lambda_k\). The Courant–Fischer theorem provides equivalent extremal characterizations of \(\lambda_k\) in terms of the Rayleigh quotient over subspaces and vectors.

2 Statement of the Courant–Fischer Theorem

2.1 Max–min (minimax) characterization

2.1.1 k-th eigenvalue as a maximum over subspaces

For a Hermitian/symmetric \(n\times n\) matrix \(A\) with eigenvalues \(\lambda_1\le\cdots\le\lambda_n\), the \(k\)-th eigenvalue admits the characterization \[ \lambda_k=\max_{\substack{S\subseteq \mathbb{C}^n\\ \dim S=k}}\ \min_{\substack{x\in S\\ x\neq 0}} \rho_A(x). \] Here the outer optimization ranges over all \(k\)-dimensional subspaces \(S\), and for each such \(S\), the inner optimization selects the smallest Rayleigh quotient attained within \(S\).

2.1.2 k-th eigenvalue as a minimum over vectors within a subspace

A closely related extremal form uses orthogonal complements. One standard variant is \[ \lambda_k=\min_{\substack{T\subseteq \mathbb{C}^n\\ \dim T=n-k+1}}\ \max_{\substack{x\in T\\ x\neq 0}} \rho_A(x). \] The structure is the same—Rayleigh quotient extremized—but the quantifiers are arranged differently: the “max over vectors” is performed inside a subspace whose dimension is tuned to \(n-k+1\).

2.2 Min–max (maximin) characterization

2.2.1 Equivalent extremal formulations

The theorem also yields the complementary pairing of the previous descriptions, often written as \[ \lambda_k=\min_{\substack{\dim S=k}}\ \max_{\substack{x\perp S\\ x\neq 0}} \rho_A(x), \] or in other equivalent subspace/orthogonality parameterizations. The key point is that for each \(k\), the eigenvalue \(\lambda_k\) appears as the equilibrium value of two nested optimizations: one over subspaces and one over vectors constrained by membership or orthogonality.

2.2.2 Relationship between the two forms

The “max–min” and “min–max” labels reflect the order of the two extremizations. In symmetric/Hermitian problems, these orders lead to the same numerical value when the subspace dimensions are chosen correctly. This equivalence is not a general feature of arbitrary functions, but it holds here due to the linear-algebraic structure inherited from orthonormal eigenbases and the behavior of quadratic forms on subspaces.

3 Variational Interpretation and Geometry

3.1 Understanding extrema of the Rayleigh quotient

The Rayleigh quotient \(\rho_A(x)\) measures how the quadratic form \(x^*Ax\) compares to the squared norm \(x^*x\). Geometrically, it is the value of a function on the unit sphere \( \{x:\|x\|=1\}\). The theorem interprets eigenvalues as particular saddle-type or extremal values realized under constrained directions: restricting \(x\) to a subspace forces the quotient to range over an interval, and the nested optimizations extract a specific endpoint that matches \(\lambda_k\).

3.2 Connection to best approximations in eigenspaces

Eigenvectors span the directions where the Rayleigh quotient takes exact eigenvalue values. When a vector is restricted to a subspace \(S\), the minimum (or maximum) Rayleigh quotient within \(S\) indicates the “closest” way, inside \(S\), to align with eigen-directions that produce smaller (or larger) spectral values. The Courant–Fischer formula then selects the subspace whose enforced constraints make that best alignment optimal for the chosen index \(k\).

3.3 Role of orthogonal complements in the variational formulas

Orthogonal complements separate “allowed” and “forbidden” directions in a way that harmonizes with eigenvector orthogonality. Because eigenvectors for distinct eigenvalues are orthogonal, specifying \(x\in S\) or \(x\perp S\) effectively restricts which eigendirections may contribute. As a result, nested optimizations over \(S\) and \(S^\perp\) naturally reproduce the ordering of eigenvalues: excluding a subspace removes certain eigenvector components, shifting the range of attainable Rayleigh quotients in a controlled manner.

4 Applications and Consequences

4.1 Eigenvalue bounds via trial subspaces

The theorem provides a practical route to bounds. If one selects a candidate subspace \(S\) of dimension \(k\), then \[ \min_{x\in S,\,x\neq 0}\rho_A(x) \] is a computable lower bound on \(\lambda_k\) (by the “max over subspaces” form). Conversely, picking a subspace in the complementary formulation yields upper bounds. This principle underlies many estimation strategies: even without knowing the exact eigenspaces, trial subspaces can bracket eigenvalues.

4.2 Monotonicity under rank-one or principal submatrix changes

Because eigenvalues are characterized by extremal Rayleigh-quotient behavior on subspaces, changes in \(A\) that have localized effects—such as perturbations that modify quadratic forms along certain directions—can be tracked by how those extremal values shift. In particular, when passing to principal submatrices or adding structured updates, eigenvalue comparisons often follow from how the set of feasible vectors/subspaces is altered, yielding monotonic trends in the ordered spectrum.

4.3 Interlacing inequalities for principal submatrices

A classical consequence is the interlacing property: eigenvalues of a principal submatrix of \(A\) sit between consecutive eigenvalues of \(A\). The Courant–Fischer framework explains this through subspace embeddings. Restricting to coordinates (equivalently, considering vectors supported on a subset of indices) corresponds to limiting the Rayleigh quotient to a subspace, which forces the extremal values to move in a way compatible with the ordering of \(\lambda_k\).

4.4 Extremal eigenvalues: largest and smallest eigenvalues

The theorem’s extremal characterizations recover the simplest bounds directly:

  • The largest eigenvalue \(\lambda_n\) is the maximum of \(\rho_A(x)\) over nonzero vectors.
  • The smallest eigenvalue \(\lambda_1\) is the minimum of \(\rho_A(x)\) over nonzero vectors.

These are the \(k=1\) and \(k=n\) instances of the general min–max/max–min structure, highlighting how the Courant–Fischer theorem extends the usual “extreme Rayleigh quotient” principle to all intermediate eigenvalues.

5.1 Relationship to the spectral theorem

The Courant–Fischer theorem can be viewed as a refinement of the spectral theorem. While the spectral theorem states that eigenvalues and eigenvectors exist and provide an orthonormal basis, Courant–Fischer describes how those eigenvalues can be recovered purely from variational information about the quadratic form. The extremal subspaces that achieve the maxima/minima align with the span of eigenvectors associated with the smallest or largest eigenvalues, making the variational viewpoint consistent with the spectral decomposition.

In functional analysis, similar min–max ideas appear for self-adjoint operators, including in the study of eigenvalue problems for differential operators. Although the setting may involve infinite-dimensional Hilbert spaces, the core mechanism remains: eigenvalues are characterized by extremizing quadratic forms on finite-dimensional subspaces. The Courant–Fischer theorem thus serves as the matrix-level prototype for broader variational principles.

5.3 Comparisons with other eigenvalue characterizations

Several alternative characterizations exist, such as those based on determinants, characteristic polynomials, or matrix norms. Compared with these, the variational approach is especially informative for ordering: it directly ties \(\lambda_k\) to constraints of dimension \(k\) and naturally supports bounds, monotonicity, and interlacing. In many contexts, it provides more immediate qualitative insight than algebraic formulas.

5.4 Courant nodal domain theorem (conceptual relation)

The Courant nodal domain theorem concerns the number of nodal domains of eigenfunctions of certain elliptic problems and is often informally described as “Courant’s theorem.” While it is not the same result as Courant–Fischer, both share the theme of connecting an index (like the eigenvalue number) to geometric or variational features. In this conceptual sense, they form a thematic pair in the study of eigenvalue-associated structure.

6 Computational and Practical Aspects

6.1 Using the theorem for eigenvalue estimation

To estimate eigenvalues, one can compute Rayleigh quotient extrema over subspaces obtained from heuristics or algorithms. For instance, if a subspace \(S\) approximates the span of eigenvectors associated with the smallest \(k\) eigenvalues, then the minimum Rayleigh quotient over \(S\) tends to approximate \(\lambda_k\) from below in accordance with the Courant–Fischer maximization logic. Similarly, complementary subspaces can yield upper estimates.

6.2 Variational methods (high-level)

Variational methods seek eigenvalues by transforming the original problem into one of minimizing or maximizing functionals. In the matrix setting, the Courant–Fischer theorem provides justification for why restricting to carefully chosen subspaces captures eigenvalue information. In applied problems, this translates into iterative procedures that update trial subspaces to improve the Rayleigh quotient extremization outcome.

6.3 Krylov subspace intuition and subspace iteration (overview)

Krylov subspace methods generate subspaces of the form \[ \mathrm{span}\{v, Av, A^2v,\ldots\}, \] which often capture dominant spectral directions efficiently. Subspace iteration similarly constructs sequences of subspaces intended to approximate invariant subspaces. The Courant–Fischer viewpoint explains why increasing the subspace size or refining its composition can systematically improve eigenvalue estimates: extremal Rayleigh quotients over larger subspaces can be squeezed toward the true eigenvalues according to the min–max ordering.

7 Generalizations and Extensions

7.1 Complex Hermitian vs. real symmetric cases

The theorem holds uniformly for real symmetric and complex Hermitian matrices. The primary adjustment is the use of the conjugate transpose \(A^*\) and the Hermitian inner product \(x^*y\). Once that framework is in place, the Rayleigh quotient remains real-valued and the extremal and ordering arguments carry through.

7.2 Positive definite matrices and generalized Rayleigh quotients

When \(A\) is positive definite, the Rayleigh quotient may be studied alongside generalized eigenvalue problems such as \((A,B)\) pairs, where one considers quotients involving two quadratic forms. The generalized Courant–Fischer framework expresses eigenvalues of the corresponding operator pencil through extremal properties of the ratio of quadratic forms, with the positivity condition ensuring well-defined denominators and stable variational behavior.

7.3 Semidefinite forms and constrained minimization (overview)

For positive semidefinite or indefinite settings, careful handling of constraints becomes necessary because the Rayleigh quotient can involve directions where the quadratic form (or an associated denominator) degenerates. Extensions typically incorporate constraints ensuring admissibility—such as restricting to subspaces where a form is nonzero or enforcing orthogonality conditions tied to kernels. In such cases, the min–max structure persists, though the precise statement may require modifying the feasible set to reflect degeneracy.