1 Introduction to Surrogate Models

1.1 Motivation and problem setting

Surrogate models address a practical bottleneck: many scientific, engineering, and analytical systems are too computationally expensive, slow to evaluate, or otherwise impractical to run repeatedly. Instead of invoking the original simulator or experiment at every step, a surrogate learns an approximate mapping from inputs to outputs from a limited set of observations. Once trained, it can provide rapid predictions, support optimization loops, and quantify uncertainty without repeatedly calling the costly source.

This approximation is especially useful in iterative workflows such as design exploration, parameter tuning, calibration, and uncertainty propagation. In these settings, the cost of collecting new high-quality data is often dominant, making the efficient use of existing samples a central concern.

1.2 Definitions and key concepts

A surrogate model (also called an emulator in some contexts) approximates a target function \(y=f(x)\) where \(x\) denotes inputs (parameters, design variables, or conditions) and \(y\) denotes outputs (responses, measurements, or model predictions). The surrogate is trained on a dataset \(\{(x_i, y_i)\}_{i=1}^n\), then used to predict \(y\) for new inputs \(x^\*\).

Key concepts include:

  • Training data: a finite set of input–output pairs used to fit the surrogate.
  • Generalization: accuracy on unseen inputs.
  • Interpolation vs extrapolation: predicting within the span of sampled inputs versus outside it.
  • Uncertainty estimation: providing not only predictions but also measures of confidence or variability.

1.3 Evaluation criteria (accuracy, efficiency, robustness)

Surrogate performance is typically assessed along several axes:

  • Accuracy: closeness of predicted outputs to true outputs, often summarized by error metrics and diagnostic plots.
  • Efficiency: runtime and memory usage for both training and prediction, including feasibility under practical budgets.
  • Robustness: stability under noisy measurements, changing data distributions, and moderate perturbations of inputs.
  • Uncertainty quality: whether uncertainty estimates meaningfully reflect prediction error, including calibration behavior.

A model can be accurate but unreliable in uncertainty, or fast but fragile under distribution shift. Good practice evaluates these dimensions jointly rather than in isolation.

2 Problem Formulation and Data

2.1 Inputs, outputs, and notation

Consider a supervised learning formulation where:

  • Inputs \(x \in \mathbb{R}^d\) represent \(d\) variables.
  • Outputs \(y\) may be scalar or vector-valued, \(y \in \mathbb{R}^m\).
  • Training data are \(\{(x_i, y_i)\}_{i=1}^n\).

For deterministic simulators, training targets can be treated as noise-free (or nearly so). For real experiments or stochastic simulators, outputs may include noise. Some settings also involve constraints, latent structure, or outputs with known physical properties.

2.2 Training data generation (design of experiments)

Because surrogate accuracy depends strongly on how data are collected, design of experiments (DoE) plays a foundational role. The goal is to choose input locations that provide informative coverage while respecting a limited evaluation budget.

Common strategies include space-filling designs that spread samples across the input domain, designs tailored to expected gradients or sensitivity, and sequential plans that add points where prediction uncertainty is high or where optima might lie.

2.3 Data preprocessing and scaling

Preprocessing can materially affect model behavior. Typical steps include:

  • Scaling: standardizing input features so distance-based methods behave sensibly.
  • Transformation: applying log or other transforms to stabilize variance or handle skewed outputs.
  • Dimensionality handling: reducing effective complexity (e.g., via feature selection or projections) when \(d\) is large.

For vector outputs, preprocessing may also include separate scalings per output dimension or transformations that make different outputs comparable in magnitude.

2.4 Handling noisy and heteroscedastic data

Many practical problems have noise that is non-constant across the input space (heteroscedasticity). Robust surrogate modeling accounts for this by:

  • Modeling an explicit noise term (e.g., in probabilistic frameworks).
  • Using loss functions less sensitive to outliers.
  • Incorporating replicate measurements or modeling observation variance.

When noise is ignored, uncertainty estimates can become overconfident and predictions may appear accurate in-sample while failing under new conditions.

3 Model Classes

3.1 Interpolating surrogates

Interpolating surrogates are constructed so they reproduce training data exactly (or nearly so, depending on implementation). They can be effective when the relationship is smooth and the dataset provides adequate coverage.

3.1.1 Radial basis function networks

Radial basis function (RBF) networks represent the function as a weighted sum of localized kernels centered at selected points. Their flexibility makes them popular for emulation of smooth responses, and they can handle irregular spacing of samples. However, performance depends on kernel choice and regularization, especially in higher dimensions.

3.1.2 Kriging / Gaussian process emulators

Kriging, often framed as Gaussian process (GP) emulation, models the output as a realization of a stochastic process with a mean function and covariance structure. This yields predictions along with a principled uncertainty measure. By learning hyperparameters of the covariance kernel, GP emulators can capture smoothness and correlations across inputs, making them suitable for both emulation and optimization.

3.2 Regression-based surrogates

Regression surrogates fit a parametric (or semi-parametric) model to approximate the mapping without necessarily matching all training points exactly. They can be advantageous when data are noisy, when exact interpolation would overfit, or when the true relationship is well captured by a lower-dimensional form.

3.2.1 Polynomial regression and splines

Polynomial regression uses basis expansions with coefficients estimated from data. Splines generalize this idea by using piecewise polynomials with continuity constraints, allowing controlled flexibility. These methods can be interpretable, but polynomial degree or spline complexity must be managed to avoid instability and overfitting.

3.2.2 Regularized linear models

Regularized linear models incorporate penalties on coefficient magnitude (or sparsity), producing solutions that generalize better under limited data. They are common baselines when the relationship is approximately linear after feature engineering, and they can be computationally efficient.

3.3 Physics-informed and hybrid approaches

Some surrogate constructions incorporate domain knowledge through constraints, known invariances, or structural relationships. This can improve accuracy and reduce data requirements by restricting the hypothesis space.

3.3.1 Embedding constraints or invariances

Constraints may include conservation laws, monotonicity, symmetries, boundary conditions, or functional forms implied by theory. Hybrid approaches blend data-driven learning with these restrictions, often by designing specialized feature mappings, modifying loss functions, or enforcing constraints architecturally.

3.4 Neural network surrogates

Neural networks provide flexible function approximation and scale well when training data are sufficiently large. Their success depends on architecture, regularization, and careful handling of uncertainty if probabilistic outputs are needed.

3.4.1 Feedforward networks

Feedforward networks (multilayer perceptrons) approximate a function using stacked nonlinear transformations. They can model complex interactions among variables, especially when paired with appropriate regularization and training schedules.

3.4.2 Mixture density and probabilistic networks

For noisy objectives or multimodal responses, probabilistic networks can represent output distributions rather than single point estimates. Mixture density networks output parameters of a mixture distribution, enabling richer predictive behavior. Other probabilistic methods estimate uncertainty via variational approximations or ensemble-style strategies.

4 Uncertainty and Probabilistic Prediction

4.1 Predictive uncertainty vs model error

Uncertainty in a surrogate prediction can be interpreted in multiple ways. Some methods estimate predictive uncertainty that combines epistemic uncertainty (due to limited data) and aleatoric uncertainty (data noise). Others report only a proxy related to expected variance under the model.

A key distinction is that high predictive variance does not automatically mean the model is wrong; rather, it indicates the surrogate lacks confidence given the training evidence and model assumptions.

4.2 Calibration of uncertainty estimates

Calibration assesses whether reported uncertainty aligns with observed error frequencies. Well-calibrated uncertainty means that intervals contain the true values with frequencies consistent with their stated coverage (e.g., a nominal 90% interval covers about 90% of cases under representative test conditions).

Calibration can be evaluated using reliability diagrams, coverage tests, or scoring rules such as proper scoring functions tailored to probabilistic forecasts.

4.3 Confidence intervals and credible intervals

  • Confidence intervals typically refer to frequentist notions based on sampling variability and repeated-sampling interpretation.
  • Credible intervals refer to Bayesian posterior uncertainty, conditional on prior assumptions and observed data.

In practice, many surrogate frameworks produce interval estimates that can be treated similarly for decision-making, but their statistical interpretations differ. Choosing which interpretation matters depends on downstream use, such as risk-sensitive design versus scientific inference.

4.4 Propagation of uncertainty through surrogates

Once a surrogate provides a distribution over outputs, uncertainty can be propagated to derived quantities (e.g., thresholds, aggregated metrics, or posterior distributions in inference tasks). Common strategies include:

  • Monte Carlo sampling using surrogate-predicted distributions.
  • Analytic approximations where feasible.
  • Sampling schemes that preserve correlations among outputs.

Propagation quality depends on how well the surrogate’s uncertainty structure captures dependencies across the input space and among output components.

5 Training and Hyperparameter Tuning

5.1 Loss functions and objective choices

Training objectives vary by model class:

  • Deterministic regression uses squared error, absolute error, or robust alternatives.
  • Probabilistic models may maximize likelihood or minimize negative log predictive density.
  • Physics-informed models may combine data fit with constraint penalties.

The choice of objective affects both accuracy and how uncertainty behaves, especially in the presence of noise and outliers.

5.2 Cross-validation and resampling strategies

Because data are finite, validation must be performed carefully. Cross-validation partitions data into training and validation folds to estimate generalization error. For smaller datasets, resampling schemes such as repeated cross-validation can reduce variance in error estimates.

For surrogates used in optimization, validation often includes checks over regions that are likely to be queried later, not just uniformly sampled test points.

5.3 Hyperparameter selection

Hyperparameters include kernel parameters in Gaussian processes, regularization strengths, network depth and width, learning rates, and mixture components. Selection typically uses validation performance, but may also incorporate model complexity constraints and prior knowledge about expected smoothness.

Bayesian hyperparameter optimization is sometimes used to tune these values efficiently when each training run is expensive.

5.4 Dealing with overfitting and underfitting

  • Overfitting occurs when the surrogate captures noise or idiosyncrasies of the training set, leading to poor generalization.
  • Underfitting occurs when the model class is too rigid or insufficiently trained, yielding systematic bias.

Regularization, early stopping, and careful model selection help manage this trade-off. Diagnostics such as training–validation error gaps and residual patterns are commonly used to identify whether the model is too complex or too simplistic.

6 Model Validation and Diagnostics

6.1 Error metrics (global and local)

Global metrics summarize performance across the input domain, such as mean absolute error, root mean square error, or normalized variants. Local metrics evaluate behavior in specific regions, which is critical for surrogates intended for optimization or for queries concentrated in certain neighborhoods.

When outputs have differing scales, normalization or per-dimension evaluation can be needed to avoid misleading averages.

6.2 Residual analysis

Residuals (differences between observed and predicted outputs) reveal systematic errors not captured by a single scalar metric. Useful diagnostics include:

  • Residual-vs-prediction plots to detect heteroscedasticity.
  • Residual maps over the input space to identify regions with poor coverage.
  • Autocorrelation checks for structured inputs.

Residual behavior can guide targeted improvements, such as adding data where residuals are consistently large.

6.3 Sensitivity checks on influential samples

Some training points can disproportionately affect a surrogate, especially in interpolating methods. Sensitivity diagnostics examine how predictions change if particular samples are removed or downweighted. This helps identify whether the model relies on isolated points or whether it captures general trends.

Influential-sample analysis is also useful for detecting outliers or mislabeled measurements.

6.4 Out-of-distribution detection and safeguards

Surrogates often assume that future queries resemble the training distribution. When inputs move far from observed data, predictions may become unreliable. Out-of-distribution detection methods attempt to quantify this risk using distance in feature space, predictive variance, or learned uncertainty signals.

Safeguards may include fallback strategies (e.g., rerunning the expensive simulator), restricting optimization to trusted regions, or using conservative uncertainty-aware decision rules.

7 Experimental Design for Surrogate Modeling

7.1 Sampling strategies

7.1.1 Random sampling

Random sampling draws inputs according to a prescribed distribution. It is easy to implement and unbiased, but may provide uneven coverage in finite samples, especially in higher dimensions.

7.1.2 Latin hypercube sampling

Latin hypercube sampling (LHS) ensures that each marginal dimension is sampled evenly. It tends to improve space coverage compared to pure randomness and is widely used as a general-purpose DoE method.

7.1.3 Quasi-random sequences

Quasi-random sequences (e.g., low-discrepancy sequences) aim for uniformity using deterministic rules. They often yield better coverage than random draws for the same sample budget, making them attractive for initial surrogate training.

7.2 Adaptive sampling (sequential design)

Adaptive sampling selects new input points based on current surrogate performance. The procedure iteratively trains a surrogate, evaluates candidate locations, then adds points that improve accuracy efficiently.

Sequential design is particularly effective when the objective is to reduce error near regions of interest, such as near predicted optima or decision boundaries.

7.3 Space-filling and coverage objectives

Coverage-based methods choose points to maximize uniformity or minimize distances among samples. Metrics may include:

  • Maximin criteria (maximize minimum pairwise distance),
  • Minimization of projection gaps,
  • Measures based on discrepancy or filling radius.

Such approaches are designed to reduce the likelihood of large unexplored regions that undermine generalization.

7.4 Budget-aware design and stopping rules

Practical workflows require stopping decisions based on a budget limit or a target performance level. Stopping rules can be triggered by:

  • Insufficient improvement in validation error,
  • Saturation of uncertainty reduction,
  • Achievement of a predefined tolerance for predictions in critical regions.

Budget-aware design also involves choosing whether to allocate resources to new data points versus increasing model complexity.

8 Surrogate-Based Optimization

8.1 Optimization workflows

Surrogate-based optimization replaces repeated evaluations of the expensive target function with repeated queries to the surrogate. A typical workflow is:

  1. Collect an initial dataset via DoE.
  2. Train a surrogate model.
  3. Run optimization on the surrogate.
  4. Optionally validate promising points using the original simulator/experiment.
  5. Update the surrogate and iterate.

This loop balances speed with reliability, ensuring that the surrogate guides search effectively without producing misleading optima.

8.2 Acquisition functions and exploration-exploitation

8.2.1 Expected improvement

Expected improvement measures the expected gain over the current best observed value at candidate points. It balances the desire to explore uncertain regions with the desire to exploit areas predicted to yield better outcomes.

8.2.2 Upper confidence bound

Upper confidence bound methods select points that maximize a trade-off between predicted mean performance and uncertainty magnitude. Large uncertainty can encourage exploration, while high predicted mean promotes exploitation.

8.3 Constrained optimization with surrogates

When optimization includes constraints, surrogates can model constraints as additional outputs. Feasibility is then estimated, and optimization methods incorporate penalties or constrained acquisition functions. Accurate constraint modeling is important because small constraint prediction errors can lead to infeasible solutions.

8.4 Multi-fidelity optimization concepts

8.4.1 Blending low- and high-fidelity models

Multi-fidelity approaches combine cheaper, lower-accuracy evaluations with more expensive, high-accuracy ones. Blending models seeks to leverage broad coverage from low-fidelity data while correcting bias using targeted high-fidelity samples. The result is often faster convergence under a total budget constraint than using only one fidelity level.

9 Surrogate Models in Bayesian Optimization

9.1 Bayesian optimization overview

Bayesian optimization is a sequential strategy for finding extrema of objective functions that are expensive to evaluate. It builds a probabilistic surrogate over the objective and uses an acquisition function to determine the next evaluation point. The process continues until a stopping criterion is met, such as a target objective value or a maximum number of evaluations.

9.2 Role of probabilistic surrogates

Probabilistic surrogates—particularly Gaussian process emulators—provide both predictive means and uncertainties. These quantities are essential for acquisition functions that explicitly rely on confidence or expected improvement. Even when uncertainty estimates are imperfect, the framework often produces a principled exploration strategy compared to purely greedy search.

9.3 Practical considerations for noisy objectives

Noisy objectives require modifications in both surrogate modeling and acquisition. Surrogates must distinguish observation noise from systematic signal. Acquisition functions may need to incorporate uncertainty appropriately so that the method does not overreact to random fluctuations.

Replicate evaluations at selected points can improve noise estimates and stabilize decision-making.

9.4 Convergence behavior and limitations

Bayesian optimization often achieves strong performance with relatively few evaluations in smooth problems. However, convergence can degrade when:

  • the surrogate assumptions are mismatched,
  • the dimensionality is high,
  • the objective is highly discontinuous,
  • the acquisition function repeatedly samples uninformative regions.

Additionally, the quality of convergence depends on hyperparameter choices and the representativeness of initial data.

10 Multi-Fidelity and Transfer Learning

10.1 When multi-fidelity is useful

Multi-fidelity modeling is useful when different evaluation levels vary in cost and accuracy. Low-fidelity simulations might be faster approximations or simplified physics, while high-fidelity models are more accurate but slower. The goal is to reduce the number of expensive evaluations needed to reach a reliable result.

10.2 Co-kriging and correlated surrogate methods

Co-kriging models outputs at multiple fidelities jointly by learning correlations between levels. If low- and high-fidelity responses are related, the high-fidelity prediction can borrow strength from abundant low-fidelity data while using limited high-fidelity samples to correct discrepancies.

10.3 Domain adaptation and transfer approaches

Transfer learning aims to improve predictions in a target domain using information from related source domains. In surrogate modeling, this can mean transferring knowledge across experimental regimes, different parameter ranges, or related physical configurations. Techniques may include reweighting training samples, learning shared representations, or fitting hierarchical models.

10.4 Combining datasets of different quality

When data quality differs (noise levels, resolution, or bias), combining them requires careful weighting and uncertainty handling. Common approaches include:

  • modeling each dataset’s noise characteristics explicitly,
  • using hierarchical Bayesian formulations,
  • incorporating reliability weights in the loss function.

If datasets are combined naively, biases can overwhelm the more accurate signals, leading to worse performance rather than improvement.

11 Computational Aspects

11.1 Training cost vs inference cost

Training can range from inexpensive (e.g., simple regression on small feature sets) to costly (e.g., Gaussian process training with large datasets). Inference—predicting for new inputs—is often much cheaper once training is completed. Selecting an approach therefore depends on the expected number of queries and the feasibility of training given available resources.

11.2 Scalability for large datasets

Scaling issues arise in memory usage, kernel computations, and optimization of hyperparameters. Mitigation strategies include:

  • sparse or inducing-point approximations for probabilistic models,
  • reduced-rank representations,
  • mini-batch training for neural networks,
  • dimensionality reduction before modeling.

Even with approximations, scalability must preserve uncertainty quality if uncertainty is important for decision-making.

11.3 Approximate Gaussian process methods (overview)

For larger datasets, approximate GP methods use techniques such as inducing points, variational inference, or structured covariance approximations. These methods reduce computational complexity while retaining key GP properties like uncertainty estimates and smoothness modeling. Performance depends on approximation quality and choice of inducing locations.

11.4 Implementation tips and common pitfalls

Common pitfalls include:

  • neglecting input scaling, causing distance-based kernels to behave poorly,
  • using overly complex models relative to dataset size,
  • ignoring constraints on output ranges when they are known,
  • assuming uncertainty estimates are automatically calibrated.

Implementation best practices include consistent preprocessing pipelines, careful hyperparameter management, and storing artifacts needed for reproducibility (data splits, seeds, and model configurations).

12 Applications and Use Cases

12.1 Fast emulation of simulation codes

Surrogates can emulate complex simulation software by learning input–output behavior from a set of simulator runs. This enables rapid what-if analysis and reduces turnaround time for iterative tasks such as design revisions or parameter sweeps.

12.2 Emulators for parameter studies

In parameter studies, a surrogate speeds up repeated evaluations across many parameter settings. This allows researchers to explore sensitivity patterns, identify promising regimes, and generate results that would be too costly if computed directly.

12.3 Design space exploration and what-if analysis

Surrogate models facilitate exploration of design spaces by enabling fast evaluation of candidate designs. Analysts can scan regions for favorable performance metrics, examine trade-offs, and perform scenario analysis with manageable computational effort.

12.4 Surrogates for inference and calibration tasks

In inference and calibration, the expensive forward model is evaluated repeatedly within statistical procedures. Surrogates can approximate the forward mapping, allowing faster posterior exploration or parameter estimation. Accuracy and uncertainty calibration remain critical because errors in the surrogate can propagate into final inference results.

13 Best Practices and Guidelines

13.1 Choosing the surrogate type

Choice depends on:

  • data size and dimensionality,
  • smoothness and expected functional structure,
  • noise presence and heteroscedasticity,
  • need for probabilistic outputs,
  • interpretability requirements.

Interpretable regression models may be appropriate for low-dimensional problems with limited data, while probabilistic emulators are common when uncertainty and active learning are important. Neural networks can be suitable when enough data exist and flexible modeling is needed.

13.2 Data requirements and target accuracy

Surrogate accuracy is constrained by sample coverage. Best practice sets a target accuracy aligned with downstream decisions. It is often more effective to improve data placement (or increase sample density near regions of interest) than to switch to a significantly more complex model.

13.3 Reproducibility and reporting standards

Reproducibility includes documenting:

  • training data generation method and ranges,
  • preprocessing steps,
  • model specification and hyperparameter selection,
  • validation protocol (splits, metrics, and uncertainty calibration checks),
  • computational environment and random seeds.

Clear reporting enables others to compare methods fairly and understand performance variability.

13.4 Common failure modes and mitigation

Typical failure modes include:

  • extrapolation into regions far from training data,
  • uncertainty miscalibration that misguides sequential sampling,
  • overfitting due to limited datasets and high-capacity models,
  • misaligned preprocessing or inconsistent feature encoding.

Mitigations involve improving experimental design, using uncertainty-aware acquisition, validating locally in regions of interest, and applying regularization and robust losses where appropriate.

14 Further Reading and References

14.1 Foundational texts and surveys

Foundational material on surrogate modeling spans statistical learning, design of experiments, and Gaussian process methodology. Surveys covering emulator construction, probabilistic emulation, and uncertainty quantification provide a broad conceptual foundation and help connect model classes to modeling assumptions.

14.2 Benchmark datasets and tools (general)

Benchmark suites often include synthetic functions and standardized engineering or scientific problems used to compare emulation accuracy and optimization performance. Tool ecosystems for Gaussian processes, Bayesian optimization, and general regression offer reusable implementations and provide baselines for experimentation.

14.3 Suggested learning path for practitioners

A practical learning progression is:

  1. Start with regression and interpolation basics, including error metrics.
  2. Move to probabilistic surrogates to understand uncertainty and calibration.
  3. Learn experimental design concepts for robust training data placement.
  4. Study Bayesian optimization workflows and acquisition functions.
  5. Explore multi-fidelity modeling and computational approximations.
  6. Practice validation diagnostics and uncertainty propagation in end-to-end tasks.