1 Definition and basic properties

A Gaussian process is a probability model over functions. It specifies, for each input point in a domain, a random output value, with the defining feature that every finite collection of those values follows a multivariate normal distribution. This makes the framework well suited to describing uncertain smooth curves, surfaces, and other continuous phenomena.

Gaussian processes are commonly used when one wants not only a prediction but also a measure of uncertainty around that prediction. Their behavior is controlled primarily by a mean function and a covariance function, which together encode expected trends and the degree to which nearby or related inputs vary together.

1.1 Formal mathematical definition

Let \(X\) be an index set, often a subset of \(\mathbb{R}^d\). A stochastic process \(f = \{f(x): x \in X\}\) is a Gaussian process if, for any finite choice of points \(x_1, \dots, x_n \in X\), the random vector \((f(x_1), \dots, f(x_n))\) has a multivariate normal distribution.

It is customary to write

\[ f(x) \sim \mathcal{GP}(m(x), k(x, x')), \]

where \(m(x)\) is the mean function and \(k(x, x')\) is the covariance function. The notation indicates that the entire process is determined by these two functions.

1.2 Finite-dimensional distributions

The finite-dimensional distributions are the ordinary multivariate Gaussian distributions obtained by evaluating the process at a chosen set of inputs. For points \(x_1, \dots, x_n\), the mean vector is

\[ \big(m(x_1), \dots, m(x_n)\big), \]

and the covariance matrix has entries \(k(x_i, x_j)\). These distributions must be mutually compatible as the set of points changes, so that they can arise from a single underlying process.

1.3 Mean function

The mean function \(m(x) = \mathbb{E}[f(x)]\) gives the average value of the process at each input. In many applications, it is taken to be zero, especially when the covariance function is thought to capture most of the structure. More informative mean functions can encode prior trends such as linear growth, seasonality, or known physical behavior.

1.4 Covariance function

The covariance function \(k(x, x') = \mathrm{Cov}(f(x), f(x'))\) determines how values of the process co-vary across the domain. It controls smoothness, scale, periodicity, and many other qualitative features of sampled functions. Different choices of covariance function lead to very different modeling behavior.

1.4.1 Positive semidefinite kernels

A valid covariance function must be positive semidefinite. This means that for any finite set of inputs and any real coefficients \(a_1, \dots, a_n\),

\[ \sum_{i=1}^n \sum_{j=1}^n a_i a_j k(x_i, x_j) \ge 0. \]

This condition ensures that the covariance matrices formed from the kernel are legitimate covariance matrices of a multivariate normal distribution.

1.4.2 Stationary and non-stationary kernels

A stationary kernel depends only on the difference \(x - x'\), not on the absolute locations of the inputs. Such kernels are useful when the statistical behavior is assumed to be translation invariant. Non-stationary kernels, by contrast, allow the covariance structure to vary across the domain and are appropriate when smoothness or variability changes with location.

1.5 Sample path interpretation

A sample path is one realized function drawn from the process. Although the process is specified probabilistically, each draw is an ordinary function. Depending on the kernel, sample paths may appear very smooth, moderately rough, periodic, or irregular. This interpretation is central to the use of Gaussian processes as prior models over unknown functions.

2 Theoretical foundations

Gaussian processes sit at the intersection of probability theory, functional analysis, and statistical modeling. Their formal properties are closely linked to multivariate normal distributions and to consistency conditions that guarantee the existence of a process with prescribed finite-dimensional laws.

2.1 Relation to multivariate normal distributions

A Gaussian process can be viewed as an infinite-dimensional extension of the multivariate normal distribution. In the finite case, a vector is characterized by its mean vector and covariance matrix; in the process setting, these become functions over the index set. The process is Gaussian precisely because all finite subcollections behave like multivariate normals.

2.2 Kolmogorov consistency conditions

The finite-dimensional distributions of a process must satisfy consistency conditions. If one marginalizes a joint distribution over some variables, the result must match the distribution assigned to the remaining variables. Kolmogorov’s extension theorem provides a framework under which a family of consistent finite-dimensional distributions determines a stochastic process.

2.3 Mean-square continuity and differentiability

Continuity and differentiability of Gaussian process sample paths are often studied in the mean-square sense. Mean-square continuity means that the expected squared difference between nearby values becomes small as the inputs approach one another. Mean-square differentiability is similarly defined through limits involving difference quotients. These properties are strongly influenced by the smoothness of the covariance function.

2.4 Reproducing kernel Hilbert space connections

Covariance kernels are closely related to reproducing kernel Hilbert spaces, which are function spaces equipped with an inner product compatible with evaluation at points. This connection helps explain why kernels are useful in both probabilistic and deterministic learning methods. It also provides tools for understanding smoothness, regularization, and representational power.

3 Construction and parameterization

In practice, a Gaussian process is built by selecting a mean function and a covariance kernel, then tuning associated parameters to match observed data. The choices made here largely determine the model’s flexibility and computational behavior.

3.1 Choosing a mean function

A mean function may be simple, such as a constant or zero function, or more structured, such as a parametric trend. When prior knowledge suggests a baseline shape, encoding it in the mean can reduce the burden on the covariance model. In many cases, however, a zero mean is used and the kernel is tasked with explaining the data structure.

3.2 Choosing a covariance kernel

The covariance kernel is the most important design choice in many applications. It encodes assumptions about smoothness, length scale, periodicity, and long-range dependence. Different kernels are often selected to reflect different underlying phenomena.

3.2.1 Squared exponential kernel

The squared exponential kernel, also called the radial basis function kernel, produces very smooth sample paths. It is widely used as a default because of its simplicity and flexibility. However, it can be overly smooth for data with abrupt changes or rough behavior.

3.2.2 Matérn kernel

The Matérn kernel introduces a parameter that controls smoothness. Compared with the squared exponential kernel, it can represent rougher functions and is often preferred when realistic trajectories are expected to be less regular. Its adaptability makes it a common choice in spatial statistics.

3.2.3 Periodic kernel

The periodic kernel is designed to model repeating structure. It is useful for phenomena that recur at regular intervals, such as seasonal patterns or oscillatory signals. By encoding periodicity directly, it can capture cycles more naturally than a generic smooth kernel.

3.2.4 Rational quadratic kernel

The rational quadratic kernel can be interpreted as a scale mixture of squared exponential kernels with different length scales. It is useful when a process exhibits variability at multiple resolutions. This makes it effective for data containing both broad trends and finer local fluctuations.

3.3 Hyperparameters and kernel learning

Kernel functions typically contain hyperparameters such as amplitude, length scale, and smoothness. These parameters are learned from data, often by maximizing the marginal likelihood. Hyperparameter learning allows the model to adapt its complexity to the observed signal while retaining a principled probabilistic structure.

3.4 Kernel composition

Complex kernels can be formed by adding or multiplying simpler ones. Addition combines features, such as trend plus periodicity, while multiplication can create localized or modulated patterns. Kernel composition is a major strength of the framework because it enables expressive models from interpretable building blocks.

4 Gaussian process regression

Gaussian process regression uses the process as a prior over unknown functions and updates it after observing data. The result is a posterior distribution over functions that can be used for prediction and uncertainty assessment.

4.1 Prior and posterior distributions

Before observing data, the process defines a prior over possible functions. After data are incorporated, the posterior reflects both the prior assumptions and the evidence provided by observations. Because the model is Gaussian and the observations are often assumed to be Gaussian as well, the posterior can frequently be derived in closed form.

4.2 Conditioning on observations

Conditioning on observations adjusts the distribution of the function at unobserved points based on the observed values. Nearby points are influenced more strongly when the kernel assigns high correlation to close inputs. This conditioning step is the core mechanism by which Gaussian process regression performs interpolation and extrapolation.

4.3 Predictive mean and variance

The predictive mean gives the central estimate for the function at new inputs, while the predictive variance quantifies uncertainty. In regions with many observations, uncertainty usually decreases. Far from observed data, the predictive variance often grows, reflecting weaker support from the training set.

4.4 Noise models

Observed data are often assumed to differ from the latent function because of measurement noise or other variability. A noise model specifies how observations deviate from the underlying process. The simplest and most common assumption is additive Gaussian noise, though more complex models are also used.

4.4.1 Homoscedastic noise

Homoscedastic noise has constant variance across the input domain. This assumption simplifies inference and is appropriate when measurement error is roughly uniform. It is the standard choice in basic Gaussian process regression.

4.4.2 Heteroscedastic noise

Heteroscedastic noise varies with the input. It is useful when uncertainty in the observations changes across the domain, such as in instruments with location-dependent error or data with region-specific variability. Modeling this effect is more complex but can produce more realistic uncertainty estimates.

4.5 Numerical computation

Exact Gaussian process regression requires operations on covariance matrices, including matrix inversion or factorization. For large data sets, these computations can become expensive. Numerical stability is typically maintained using methods such as Cholesky decomposition and careful regularization.

5 Gaussian process classification

Gaussian process classification extends the framework to discrete outcomes, such as binary labels. Because class labels are not Gaussian, exact inference is generally unavailable and approximation methods are required.

5.1 Latent function formulation

Classification is often modeled through a latent real-valued function drawn from a Gaussian process. The latent function is then mapped to class probabilities. This approach preserves the flexibility of the process while accommodating discrete outputs.

A link function transforms latent values into probabilities. Common choices include the logistic and probit links. These functions ensure that predicted probabilities remain in the interval from zero to one, allowing the model to represent uncertainty in class membership.

5.3 Approximate inference methods

Because the posterior distribution is not Gaussian after the likelihood for classification is applied, approximate inference is necessary. These methods seek a tractable approximation to the true posterior while preserving as much accuracy as possible.

5.3.1 Laplace approximation

The Laplace approximation replaces the posterior with a Gaussian centered at its mode. It is relatively simple and often effective, though it may be less accurate when the posterior is strongly non-Gaussian.

5.3.2 Expectation propagation

Expectation propagation iteratively refines an approximation by matching moments of local factors. It can yield accurate results in many classification problems, though it is computationally more involved than simpler methods.

5.3.3 Variational inference

Variational inference formulates the approximation problem as optimization over a family of tractable distributions. It offers a flexible framework and often scales well to larger data sets, especially when combined with sparse representations.

6 Inference and optimization

Inference in Gaussian process models includes estimating latent functions, learning hyperparameters, and computing posterior summaries. Optimization often focuses on quantities derived from the marginal likelihood, which balances fit and complexity.

6.1 Marginal likelihood

The marginal likelihood measures how well the model explains the observed data after integrating out the latent function. It is a central objective for learning kernel parameters because it naturally penalizes overly complex models that do not improve explanatory power.

6.1.1 Exact marginal likelihood

In Gaussian process regression with Gaussian noise, the marginal likelihood can be computed exactly. This closed-form expression is one of the framework’s most attractive features, as it supports principled model selection and parameter estimation.

6.1.2 Gradient-based hyperparameter optimization

Gradients of the marginal likelihood with respect to hyperparameters are commonly used for optimization. This allows efficient numerical search for good kernel settings. Since the objective may have local optima, initialization and regularization can play important roles.

6.2 Posterior sampling

Posterior sampling generates function realizations from the posterior distribution. Such samples are useful for uncertainty visualization, simulation, and decision making under uncertainty. Sampling methods may be exact for small problems or approximate for larger ones.

6.3 Sparse approximations

Sparse approximations reduce computational cost by summarizing the process with a smaller set of representative variables. They are especially important when the number of observations is large, since exact methods scale poorly with data size.

6.3.1 Inducing point methods

Inducing point methods introduce a limited number of auxiliary locations that summarize the function. These points act as a compressed representation of the process, allowing faster computation while retaining much of the original model’s flexibility.

6.3.2 Low-rank approximations

Low-rank methods approximate the covariance matrix by a matrix of smaller effective rank. They can substantially reduce memory usage and runtime. The tradeoff is that the approximation may miss fine-scale structure if the rank is too low.

6.4 Computational complexity

Standard Gaussian process methods scale poorly with the number of observations because covariance matrix operations typically require cubic time and quadratic memory. This limitation motivates approximate methods, specialized kernels, and structured matrix techniques. Managing complexity is one of the main practical challenges in the field.

7 Applications

Gaussian processes are used in many areas where flexible function estimation and quantified uncertainty are valuable. Their probabilistic nature makes them suitable for prediction, interpolation, and adaptive experiment design.

7.1 Regression and function approximation

In regression, Gaussian processes provide a smooth interpolant that also reports uncertainty. They are effective when data are limited and prior assumptions about continuity or smoothness are reasonable. Their predictions often serve as strong baselines in scientific and engineering problems.

7.2 Time series modeling

For time series, Gaussian processes can represent trend, periodicity, and local variation. They are useful for forecasting when irregular sampling or uncertain noise levels make simpler methods less suitable. Kernel design is often tailored to capture temporal structure.

7.3 Spatial statistics and geostatistics

In spatial modeling, Gaussian processes describe values that vary over geographic or physical space. They are closely associated with kriging and are widely used for interpolation of environmental, geological, and experimental measurements. Spatial kernels encode how correlation decreases with distance.

7.4 Bayesian optimization

Bayesian optimization uses a Gaussian process surrogate model to guide the search for the optimum of an expensive objective function. The model balances exploration and exploitation by quantifying uncertainty over unsampled regions. This makes it useful when each evaluation of the objective is costly.

7.5 System identification

In system identification, Gaussian processes help infer unknown input-output relationships in dynamic systems. They can model nonlinear effects without requiring a rigid parametric form. This is valuable when the underlying mechanism is only partially known.

7.6 Uncertainty quantification

Gaussian processes are a natural tool for uncertainty quantification because they produce distributions over functions rather than single estimates. This allows practitioners to assess confidence, identify regions of sparse evidence, and propagate uncertainty into downstream analyses.

8 Extensions and variants

Many extensions broaden the basic Gaussian process framework to handle multiple outputs, hierarchical structure, latent variables, and nonstandard observation models. These variants expand the range of settings in which the method can be used.

8.1 Multi-output Gaussian processes

Multi-output models describe several related functions jointly. They can capture correlations between outputs, such as temperature and pressure or multiple sensor channels. Shared structure can improve prediction when data for one output help inform another.

8.2 Deep Gaussian processes

Deep Gaussian processes compose multiple Gaussian process layers to build hierarchical representations. This increases expressiveness and can model complex nonlinear structure. Inference is more difficult than in a single-layer model, so approximation methods are typically required.

8.3 Gaussian process state-space models

State-space formulations represent a Gaussian process using latent dynamical states. They are useful for time series and sequential data, particularly when recursive filtering or online updating is desired. This representation can improve scalability for temporal problems.

8.4 Non-Gaussian process alternatives

When Gaussian assumptions are too restrictive, other stochastic process models may be used. These alternatives can accommodate heavier tails, asymmetry, or discrete structures. They broaden the modeling toolkit for data that do not fit the Gaussian paradigm well.

8.5 Gaussian process latent variable models

Latent variable models use Gaussian processes to describe mappings from a low-dimensional latent space to observed data. They are often applied to dimensionality reduction and manifold learning. The latent-space formulation can uncover hidden structure in complex data sets.

9 Advantages and limitations

Gaussian processes offer a powerful balance of flexibility and interpretability, but they also face practical and modeling constraints. Their usefulness depends on the scale of the data and the suitability of the chosen kernel.

9.1 Interpretability

The kernel-based structure makes the model relatively interpretable compared with some black-box approaches. Specific kernel components can correspond to smooth trends, cycles, or local variation. This transparency is one reason the method is popular in scientific settings.

9.2 Data efficiency

Gaussian processes often perform well with limited data because they incorporate prior assumptions directly into the model. They can make sensible predictions even before many observations are available. This is especially valuable when data collection is expensive.

9.3 Scalability challenges

The main drawback is computational cost. Exact inference becomes difficult as data sets grow, especially in higher dimensions or when many hyperparameters are used. Approximate methods reduce this burden but may introduce additional modeling choices.

9.4 Sensitivity to kernel choice

Performance can depend strongly on the selected kernel and its parameters. An unsuitable kernel may oversmooth the data, miss important structure, or produce misleading uncertainty estimates. Careful kernel design and model checking are therefore important.

Gaussian processes are connected to several major ideas in probability, statistics, and machine learning. These related topics help situate the framework within a broader mathematical and applied context.

10.1 Brownian motion

Brownian motion is a classic stochastic process with Gaussian increments. It is one of the best-known examples of a Gaussian process and serves as a foundational model in probability theory.

10.2 Kalman filtering

Kalman filtering is a recursive estimation method for linear dynamical systems with Gaussian noise. It is closely related to Gaussian process modeling through shared dependence on Gaussian distributions and sequential updating.

10.3 Kernel methods

Kernel methods are learning techniques that use positive semidefinite kernels to measure similarity. They are closely linked to Gaussian processes through shared kernel functions and common mathematical structure.

10.4 Kriging

Kriging is a spatial interpolation method widely used in geostatistics. It is mathematically equivalent to certain forms of Gaussian process regression, making the two concepts strongly connected.

10.5 Bayesian nonparametrics

Bayesian nonparametrics studies models whose complexity can grow with the data rather than being fixed in advance. Gaussian processes are a central example of this approach because they define distributions over infinite-dimensional function spaces.