1. Definition and Basic Properties

1.1 Inner product and orthogonality

In an inner product space, an inner product \(\langle \cdot,\cdot\rangle\) assigns a scalar to each pair of vectors, capturing both “length” and “angle.” Two vectors \(u\) and \(v\) are orthogonal if \(\langle u,v\rangle=0\). This algebraic condition generalizes perpendicularity from Euclidean geometry to more abstract settings.

1.2 Normalization and unit length

A vector is normalized when it has unit norm, meaning \(\|v\|=1\). In an inner product space, the norm is determined by the inner product via \(\|v\|=\sqrt{\langle v,v\rangle}\). Normalization is useful because it turns projections and coefficients into simple inner-product expressions.

1.3 Orthonormality condition and examples

A set \(\{v_1,\dots,v_n\}\) is orthonormal when each vector is normalized and distinct vectors are orthogonal. Equivalently, \[ \langle v_i,v_j\rangle= \begin{cases} 1, & i=j,\\ 0, & i\neq j. \end{cases} \] Common examples include the standard basis vectors in \(\mathbb{R}^n\), and in \(\mathbb{C}^n\) with the usual Hermitian inner product, the canonical coordinate vectors remain orthonormal.

1.4 Consequences for norms and distances

Orthonormality simplifies the computation of norms of linear combinations. If \(x=\sum_{i=1}^n a_i v_i\), then \[

\|x\|^2=\sum_{i=1}^na_i^2,

\] often called the “Pythagorean” behavior of orthonormal expansions. More generally, if two vectors are expressed in orthonormal coordinates, their squared distance can be computed by summing squared coefficient differences.

2. Orthonormal Sets and Linear Independence

2.1 Proof of linear independence

An orthonormal set is automatically linearly independent. Suppose \(\sum_{i=1}^n a_i v_i=0\). Taking the inner product with \(v_k\) yields \[ \left\langle \sum_{i=1}^n a_i v_i,\, v_k\right\rangle=\sum_{i=1}^n a_i\langle v_i,v_k\rangle=a_k=0. \] Since this holds for each \(k\), all coefficients vanish.

2.2 Cardinality bounds in finite-dimensional spaces

In a \(d\)-dimensional inner product space, any orthonormal set has at most \(d\) vectors. The reason is that linear independence cannot exceed the dimension. Thus an orthonormal set’s size is bounded by the ambient space’s dimension.

2.3 Relationship to spanning and bases

When an orthonormal set has \(d\) vectors in a \(d\)-dimensional space, it spans the whole space and forms an orthonormal basis. In lower dimensions, an orthonormal set may span a proper subspace, in which case it acts like a “coordinate system” restricted to that subspace.

2.4 Maximal orthonormal sets

A maximal orthonormal set (not extendable by adding more orthonormal vectors) corresponds to an orthonormal basis for the span of all vectors available in the space under consideration. In finite dimensions, every maximal orthonormal set is a basis, and its size equals the dimension.

3. Matrix and Coordinate Formulations

3.1 Orthonormal vectors as columns of a matrix

Given an orthonormal set \(\{v_1,\dots,v_n\}\) in \(\mathbb{R}^m\) or \(\mathbb{C}^m\), one can form a matrix \(V=[v_1\ \cdots\ v_n]\), whose columns are the vectors. Orthonormality is then encoded by the relation \[ V^\ast V=I_n, \] where \(V^\ast\) denotes the transpose in the real case and the conjugate transpose in the complex case.

3.2 Orthogonal (unitary) matrices and Gram matrix

If \(V\) is square (\(n=m\)) and \(V^\ast V=I\), then \(V\) is orthogonal (real case) or unitary (complex case). If columns are not exactly orthonormal, the matrix \(G=V^\ast V\) is the Gram matrix, whose \((i,j)\) entry equals \(\langle v_i,v_j\rangle\). For orthonormal columns, \(G\) becomes exactly the identity.

3.3 Change-of-basis interpretations

An orthonormal basis provides a particularly stable change-of-basis map. If \(V\) is the matrix of basis vectors, then coordinates of a vector \(x\) in that basis are obtained efficiently: the coordinate vector is \(V^\ast x\) (in the appropriate inner-product sense). This avoids solving general linear systems in many standard tasks.

3.4 Numerical stability considerations

In computation, “almost orthonormal” sets may arise due to floating-point error. Checking \(V^\ast V\approx I\) helps diagnose loss of orthogonality. Orthonormal formulations often reduce sensitivity in projections and least-squares steps, though the actual accuracy can still depend on how the basis was constructed.

4. Projections onto Subspaces

4.1 Orthogonal projection using an orthonormal set

Let \(S=\operatorname{span}\{v_1,\dots,v_n\}\). The orthogonal projection of \(x\) onto \(S\) is the vector \(\hat{x}\in S\) such that \(x-\hat{x}\) is orthogonal to every vector in \(S\). With an orthonormal spanning set, the projection has the explicit form \[ \hat{x}=\sum_{i=1}^n \langle x,v_i\rangle v_i. \]

4.2 Coefficient formulas and reconstruction

The coefficients in the projection are simply the inner products \(\langle x,v_i\rangle\). Because the vectors have unit length and are mutually orthogonal, there is no need for additional scaling factors, and the reconstruction uses the same set of vectors that defines the subspace.

4.3 Error characterization for projections

The residual \(r=x-\hat{x}\) is orthogonal to the subspace \(S\). In particular, for orthonormal \(v_i\), \[

\|x\|^2=\|\hat{x}\|^2+\|r\|^2.

\]

This identity provides a clear error measure: \(\|r\|\) quantifies the distance from \(x\) to the subspace.

4.4 Pythagorean theorem in inner product spaces

The norm identity above is a generalization of the Pythagorean theorem. When components are taken along mutually orthogonal directions, squared lengths add without cross terms, reflecting the vanishing inner products among orthogonal vectors.

5. Expansion in an Orthonormal Basis

5.1 Complete orthonormal sets

A complete orthonormal set spans the entire space, so every vector can be written as a finite linear combination of the basis vectors. In finite-dimensional settings, a complete orthonormal set is exactly an orthonormal basis.

5.2 Coefficients via inner products

If \(\{v_1,\dots,v_d\}\) is an orthonormal basis and \(x\) is any vector, then \[ x=\sum_{i=1}^d \langle x,v_i\rangle v_i. \] The inner product directly yields the expansion coefficients because orthonormality isolates each component.

5.3 Parseval-type identities (finite-dimensional)

A central consequence is the norm preservation in coefficient space: \[

\|x\|^2=\sum_{i=1}^d\langle x,v_i\rangle^2.

\] This mirrors energy conservation: the total squared magnitude of \(x\) equals the sum of squared magnitudes of its coordinates in an orthonormal basis.

5.4 Uniqueness of expansion

Expansions in a basis are unique. Since an orthonormal basis is a basis, the coefficients \(\langle x,v_i\rangle\) determined by inner products are forced to be the only coefficients that reproduce \(x\).

6. Construction of Orthonormal Sets

6.1 Gram–Schmidt process

Given linearly independent vectors \(u_1,\dots,u_n\), the Gram–Schmidt process produces orthonormal vectors \(v_1,\dots,v_n\). One begins by normalizing: \[

v_1=\frac{u_1}{\|u_1\|},

\] and then removes the components along previously constructed vectors. The \(k\)-th step takes \[

w_k=u_k-\sum_{i=1}^{k-1}\langle u_k,v_i\rangle v_i,\quad v_k=\frac{w_k}{\|w_k\|}.

\] The construction ensures \(v_k\) is orthogonal to all earlier \(v_i\) and normalized.

6.2 Modified Gram–Schmidt and implementation details

The straightforward Gram–Schmidt algorithm can suffer from loss of orthogonality when implemented numerically. Modified Gram–Schmidt rearranges computations to improve stability by orthogonalizing step-by-step with better-conditioned intermediate values. In practice, this often yields closer-to-orthonormal results, especially when vectors are nearly dependent.

6.3 Handling dependent or near-dependent vectors

If the input vectors are linearly dependent, a step may produce \(w_k\) that is (exactly or approximately) zero. In exact arithmetic this signals that an orthonormal vector cannot be extended in that direction; in numerical arithmetic, small \(w_k\) values indicate near-dependence. Algorithms typically monitor norms and may discard or replace problematic vectors based on tolerances.

6.4 Orthogonality verification and normalization

After construction, orthogonality can be checked by evaluating inner products \(\langle v_i,v_j\rangle\) and verifying they are near \(0\) for \(i\neq j\). Normalization is verified by ensuring \(\|v_i\|\approx 1\). This verification is particularly important when orthonormal sets are used in sensitive computations like iterative methods or large-scale linear algebra.

7. Orthonormality in Common Discrete Settings

7.1 Finite-dimensional vector spaces over \(\mathbb{R}\) and \(\mathbb{C}\)

Orthonormality is defined in any finite-dimensional inner product space, including spaces over \(\mathbb{R}\) and \(\mathbb{C}\). In the complex case, the inner product uses conjugation in the first argument so that \(\langle v_i,v_j\rangle=0\) still correctly characterizes orthogonality.

7.2 Discrete inner products and weighted norms

In discrete contexts, inner products often take the form \[ \langle x,y\rangle=\sum_{k} w_k\, x_k\,\overline{y_k}, \] where \(w_k\) are weights. Orthonormality then depends on these weights, producing basis vectors orthogonal under the weighted rule rather than under the unweighted sum.

7.3 Orthogonality of indicator vectors

Indicator vectors for disjoint subsets provide a simple orthogonality example. In \(\mathbb{R}^n\), let \(v_A\) be the vector that equals \(1\) on indices in \(A\) and \(0\) elsewhere. If two sets \(A\) and \(B\) are disjoint, then \(\langle v_A,v_B\rangle=0\). By scaling these indicators to have unit norm, one obtains an orthonormal family.

7.4 Connection to discrete orthogonal polynomials (overview)

Discrete orthogonal polynomials form families that satisfy orthogonality relations under an appropriate inner product (often involving weights on integers). After normalization, these polynomials behave like orthonormal vectors in a finite or countable-dimensional space, which enables efficient expansions and recurrence-based computations.

8. Applications in Discrete Mathematics and Algorithms

8.1 Least-squares solutions with orthonormal columns

In linear regression and least-squares fitting, orthonormal columns simplify the solution. If a matrix \(A\) has orthonormal columns, i.e., \(A^\ast A=I\), then the least-squares minimizer for \(\|Ax-b\|\) can be obtained directly by \(x=A^\ast b\). This avoids more complex normal-equation manipulations.

8.2 Spectral views: diagonalization intuition (overview)

Orthonormal bases underpin spectral reasoning for symmetric or normal operators. While the full theory involves eigenvalues and operator structure, the intuition is that an orthonormal basis makes the action of an operator “coordinate-separable,” turning certain transformations into near-diagonal behavior in the right basis.

8.3 Efficient computation with orthonormal bases

When computations are expressed in orthonormal coordinates, dot products and projections become efficient: inner products replace matrix inverses, and orthogonality removes cross terms. This is frequently leveraged in algorithms that alternate between coordinate representations and geometric interpretations.

8.4 Data compression and dimensionality reduction (conceptual)

Orthonormal bases enable compact representations by truncating expansions. If a vector is expanded in an orthonormal basis and only the largest-magnitude coefficients are kept, the discarded part has a controlled contribution to the error in squared norm. This principle motivates dimensionality reduction methods that preserve as much “energy” as possible in a smaller space.

9.1 Orthogonal vs. orthonormal sets

An orthogonal set requires pairwise inner products to vanish but does not require unit lengths. Orthonormality adds the normalization condition, which removes scaling factors and yields cleaner formulas for projections, coordinate extraction, and Parseval-type identities.

9.2 Orthonormal bases vs. orthonormal frames

An orthonormal basis covers the entire space, while an orthonormal frame is typically a smaller orthonormal set whose span is a subspace. Frames are useful for representing vectors approximately or for working with features in a reduced-dimensional component space.

9.3 Unitary operators and general inner product spaces

In complex inner product spaces, an operator \(U\) is unitary if it preserves inner products: \(\langle Ux,Uy\rangle=\langle x,y\rangle\). Unitary operators map orthonormal bases to orthonormal bases, maintaining orthonormality under transformation.

9.4 Biorthogonal and nearly orthonormal sets (overview)

Sometimes two different families of vectors appear, each orthogonal to the other in a cross sense (biorthogonality). More broadly, “nearly orthonormal” sets arise under numerical approximation; their Gram matrices are close to identity, enabling perturbation-based analysis of errors in projections and coefficient computations.

10. Worked Examples

10.1 Constructing an orthonormal set in \(\mathbb{R}^3\)

Let \[ u_1=(1,0,0),\quad u_2=(1,1,0),\quad u_3=(1,1,1). \] First normalize \(v_1=u_1=(1,0,0)\). Next, \[ w_2=u_2-\langle u_2,v_1\rangle v_1=(1,1,0)-(1)(1,0,0)=(0,1,0), \]

so \(v_2=w_2/\|w_2\|=(0,1,0)\). Then

\[ w_3=u_3-\langle u_3,v_1\rangle v_1-\langle u_3,v_2\rangle v_2=(1,1,1)-(1)(1,0,0)-(1)(0,1,0)=(0,0,1), \] so \(v_3=(0,0,1)\). The set \(\{v_1,v_2,v_3\}\) is orthonormal in \(\mathbb{R}^3\).

10.2 Projection example with a small orthonormal system

In \(\mathbb{R}^3\), take the orthonormal set \(v_1=(1,0,0)\), \(v_2=(0,1,0)\), and let \(x=(2,3,5)\). The subspace is the \(xy\)-plane. The projection is \[ \hat{x}=\langle x,v_1\rangle v_1+\langle x,v_2\rangle v_2=2(1,0,0)+3(0,1,0)=(2,3,0). \] The residual is \((0,0,5)\), orthogonal to both \(v_1\) and \(v_2\).

10.3 Verifying orthonormality from a given matrix

Suppose \[ V= \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\[4pt] \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix}. \] Compute \(V^\top V\): \[ V^\top V= \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix} \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix} =I_2. \] Therefore the columns of \(V\) form an orthonormal set.

10.4 Gram–Schmidt example with a dependent-free list

Let \[ u_1=(1,1,0),\quad u_2=(1,0,1),\quad u_3=(0,1,1). \] Normalize the first vector: \[

v_1=\frac{u_1}{\|u_1\|}=\frac{1}{\sqrt{2}}(1,1,0).

\] Compute the second orthogonal component: \[ w_2=u_2-\langle u_2,v_1\rangle v_1. \] Here \(\langle u_2,v_1\rangle=\frac{1}{\sqrt{2}}(1\cdot 1+0\cdot 1+1\cdot 0)=\frac{1}{\sqrt{2}}\), so \[ w_2=(1,0,1)-\frac{1}{\sqrt{2}}\cdot\frac{1}{\sqrt{2}}(1,1,0)=(1,0,1)-\frac12(1,1,0)=\left(\frac12,-\frac12,1\right). \] Normalize: \[

v_2=\frac{w_2}{\|w_2\|},\quad \|w_2\|=\sqrt{\left(\frac12\right)^2+\left(-\frac12\right)^2+1^2}=\sqrt{\frac14+\frac14+1}=\sqrt{\frac32}.

\] For the third, subtract projections onto both \(v_1\) and \(v_2\) to obtain \(w_3\), then normalize to get \(v_3\). By construction, \(v_1,v_2,v_3\) are mutually orthogonal and unit length.