Reproducibility in numerical computing

1 Foundations of reproducibility in numerical computing

1.1 Definitions: repeatability vs reproducibility

In numerical computing, repeatability describes the ability to obtain the same results when the computation is run again by the same user or system under the same conditions. Reproducibility extends this idea: results should be obtainable by others using the same general methods, ideally with the same or equivalent data and documented procedures.

Both notions rely on specifying enough information to remove ambiguity: the mathematical problem statement, the numerical method, the software and library stack, the computational environment, and the exact inputs (including preprocessing steps). In practice, reproducibility is often partial, meaning results match within agreed tolerances rather than bit-for-bit.

1.2 Sources of irreproducibility in numerical workflows

Irreproducibility can arise from many points in a workflow. Common culprits include changes in data preprocessing, nondeterministic data loading order, differing defaults in libraries, and unrecorded parameter choices (such as solver tolerances or regularization weights). Even with identical code, outcomes can diverge due to floating-point behavior, parallel execution effects, and variations in hardware or compiler optimizations.

Another frequent source is “configuration drift,” where small differences in environment variables, build flags, or dependency versions subtly alter numerical routines. Finally, workflows that include hidden state—caches, random sampling without fixed seeds, or dynamically generated artifacts—may yield results that are difficult to recreate.

1.3 Floating-point and numerical sensitivity basics

Floating-point arithmetic approximates real numbers using a finite mantissa and exponent, creating rounding error at every operation. Many numerical algorithms are sensitive to such perturbations, particularly when they involve ill-conditioned problems, subtraction of nearly equal quantities, or iterative procedures where errors can accumulate or amplify.

Numerical sensitivity does not necessarily imply that results are wrong; it indicates that small changes in inputs, rounding, or operation order can lead to noticeable output differences. Understanding sensitivity typically requires attention to the conditioning of the underlying mathematical problem and to the stability of the chosen algorithm.

1.4 Determinism vs statistical randomness in computation

A computation may be deterministic in the strict sense (given identical inputs and environment it always produces the same outputs), yet still not reproducible in a broader scientific sense if the randomness model is not controlled or documented. Conversely, stochastic algorithms can be reproducible when randomness is generated from a specified pseudorandom generator and seed, and when all other sources of nondeterminism are addressed.

Distinguishing deterministic numerical differences from randomness-driven variation is essential. For example, two runs with fixed seeds should be expected to follow the same random sampling path, while runs with different seeds should be compared using variance-aware metrics rather than a single equality check.

2 Experimental design for repeatable numerics

2.1 Problem specification and model configuration

Repeatable numerics begin with precise problem definition. This includes the mathematical objective or constraints, the model architecture (if applicable), and the method-specific configuration such as discretization choices, solver selection, and hyperparameters.

A well-specified experiment also clarifies what counts as the “input”: raw data, processed features, normalized variables, and any derived quantities. If the same paper or notebook can be interpreted multiple ways, later attempts at reproduction may implement different implicit assumptions.

2.1.1 Unit consistency, scaling, and parameter documentation

Numerical results are frequently sensitive to scaling and unit conventions. Ensuring consistent units across all stages—data acquisition, preprocessing, model equations, and reporting—reduces avoidable discrepancies. Documented scaling choices (such as normalization ranges, standardization constants, or nondimensionalization factors) support faithful re-creation.

Parameter documentation should cover not only values but also their meanings and transformations. For instance, recording whether a tolerance is absolute or relative, or whether a learning rate was scheduled, helps others match the intended computational protocol.

2.2 Data provenance and dataset versioning

Data provenance records where data originated, how it was transformed, and which versions were used. Reproducibility improves when datasets are treated as versioned artifacts, with immutable snapshots corresponding to reported experiments.

Provenance typically includes links to source files, checksums for integrity, documentation of schema changes, and logs describing preprocessing steps. If preprocessing depends on external resources (such as lookup tables or remote services), those dependencies also need version tracking.

2.2.1 Handling missing data and preprocessing pipelines

Missing values and preprocessing choices can significantly alter numerical outcomes. Reproducible handling of missing data requires explicit policies (imputation method, treatment of sentinel values, or row/feature removal rules) and consistency across training and evaluation splits.

A preprocessing pipeline should be implemented as deterministic code wherever possible. When nondeterminism is unavoidable (e.g., randomized augmentation), it should be controlled via fixed randomness parameters and documented clearly.

2.3 Random seeds and controlled stochastic components

Stochastic elements include random sampling, data augmentation, randomized initializations, and certain solver strategies. Reproducibility requires managing these sources systematically through seed specification and controlled generator usage.

A robust approach records the seed(s) used, the pseudorandom generator algorithm (or at least the library and version), and how random streams are consumed (for example, separate generators per component). This prevents accidental shifts in random consumption order that can change outcomes even when seeds appear unchanged.

2.3.1 Pseudorandom generators and seed management

Pseudorandom generators should be selected and configured in a way that is consistent across platforms. Some libraries differ in generator implementations or default seeding behaviors, so documentation should name the exact generator or specify the API-level choices used.

Seed management also includes recording whether seeds are set once globally or per epoch/job, and whether multiple workers share or split randomness. For parallel workloads, careful design helps avoid race-condition-driven changes in random draw order.

2.4 Numerical tolerances and acceptance criteria

Because floating-point computations often cannot be expected to match exactly, reproducibility frequently uses tolerance-based acceptance. Acceptance criteria must be agreed upon in advance and justified by the error magnitudes expected from the algorithm and problem scale.

Tolerance selection should consider both absolute and relative scales, as well as the type of quantities being compared (residuals, solution vectors, objective values, or derived metrics). Proper criteria reduce false failures due to insignificant numerical variation while still catching genuine errors.

2.4.1 Relative vs absolute error norms and benchmarks

Relative error focuses on scale by comparing differences to the magnitude of the reference values, while absolute error measures differences directly. Error norms (such as L2 or infinity norms) summarize vector discrepancies and support standardized comparisons.

Benchmarks used for tolerance design should reflect the typical operating regime of the computation. Using unrealistic benchmarks can either hide real problems (tolerances too loose) or cause frequent unnecessary mismatches (tolerances too strict).

3 Software and environment controls

3.1 Dependency management and version pinning

Software reproducibility depends on locking the versions of libraries and their transitive dependencies. Even minor version changes can alter numerical kernels, default solver settings, or algorithmic details such as convergence checks.

Version pinning typically uses mechanisms such as lockfiles or explicit module manifests. For compiled components, pinning may include compiler runtime libraries and platform-specific builds. Documenting the full dependency graph supports later reconstruction of the exact environment.

3.1.1 Lockfiles, module manifests, and containerization

Lockfiles record precise package versions and resolution choices, limiting the variability introduced by dependency solvers. Module manifests serve a similar purpose for curated environments, specifying the versions required to run the computation.

Containerization packages the software stack together with runtime libraries and often system dependencies. While containers can improve reproducibility substantially, they still require careful documentation of host-level differences (e.g., drivers for GPU workloads) and careful pinning of base images.

3.2 Build settings and compiler/library differences

Build configuration influences numerical behavior. Compiler optimizations can change instruction sequences, rearrange computations, or affect how intermediate precision is handled. Library implementations may also differ in mathematical function approximations or parallelization strategies.

Recording build flags, compilation toolchains, and the exact math libraries in use helps isolate discrepancies. For compiled languages, documenting whether fast-math options were enabled is particularly important because such options can trade strict numerical guarantees for speed.

3.2.1 CPU instruction sets and math library behavior

Instruction sets (for example, SIMD variants) can alter operation ordering and the handling of rounding. Likewise, math libraries may vary in how they implement transcendental functions and fused operations.

Reproducibility efforts often require controlling or at least reporting these factors. When exact bit-for-bit matching is not feasible, consistent instruction set configurations still reduce variability and make tolerance-based comparisons more meaningful.

3.3 Operating system and hardware considerations

Operating systems influence file handling, threading behavior, and available runtime capabilities. Hardware differences—including CPU models, memory behavior, and accelerators—can affect both performance and numerical results, especially in the presence of nondeterminism.

Hardware-level nondeterminism can include subtle differences in floating-point execution pathways or reductions performed across parallel units. Therefore, documenting machine characteristics and ensuring consistent resource configurations supports comparability.

3.4 Reproducible execution workflows

A reproducible workflow specifies not only code and dependencies but also the execution script: command-line arguments, environment variables, input selection logic, output naming conventions, and the order of steps.

Workflow reproducibility improves when runs are orchestrated via scripts rather than manual interactive execution. Capturing the exact command sequence makes it easier to rerun experiments and diagnose discrepancies quickly.

3.4.1 Deterministic settings for parallel computing

Parallel computing introduces nondeterministic effects, particularly when reductions occur in different orders. Many frameworks offer determinism or reproducible settings, but they may reduce performance.

Documenting whether determinism flags were enabled, how many threads or workers were used, and which scheduling policy was applied helps ensure that others can replicate the conditions under which results were produced.

4 Numerical methods and stability practices

4.1 Algorithm selection and conditioning awareness

Choosing a numerical algorithm is inseparable from considerations of stability and conditioning. Conditioning describes how sensitively the true solution responds to perturbations in the input, while stability describes how errors introduced by the algorithm behave during computation.

Awareness of ill-conditioning motivates techniques such as regularization, rescaling, or alternative formulations. When two researchers choose different algorithms for the “same” problem, they may obtain systematically different results even if both are internally consistent and numerically reasonable.

4.1.1 Ill-conditioned problems and sensitivity mitigation

Ill-conditioned problems amplify errors and can make iterative methods converge slowly or to different approximations depending on rounding. Sensitivity mitigation includes improving scaling, using more robust factorizations, applying regularization, and verifying that stopping criteria align with desired accuracy.

Another practical technique is to evaluate error measures beyond raw output inspection. For example, checking residuals and invariants can reveal when differences reflect instability rather than a software mistake.

4.2 Precision choices (float32/float64) and rounding

Precision selection affects both accuracy and reproducibility. Single-precision computations generally have larger rounding errors than double precision, and they may amplify differences across hardware and compiler implementations.

Rounding behavior also depends on whether operations use strict IEEE semantics or allow relaxed precision. Documenting the expected precision throughout the pipeline—data storage, intermediate computations, and output casting—helps others replicate the numerical trajectory.

4.2.1 Mixed-precision tradeoffs for reproducibility

Mixed-precision methods use lower precision for some operations while retaining higher precision for others. This can improve speed but introduces additional sources of variability, such as how gradients or updates are accumulated and when casting occurs.

To improve reproducibility, mixed-precision experiments should clearly specify which operations run in which precision and how accumulation is handled. Acceptance criteria should reflect the expected numerical differences introduced by mixed precision.

4.3 Convergence criteria and stopping rules

Iterative solvers depend on stopping criteria that may involve residual thresholds, relative changes, or maximum iteration counts. Slight differences in tolerances or how norms are computed can cause solvers to terminate at different iteration numbers and yield different outputs.

Reproducible design requires documenting solver stopping conditions and the precise interpretation of tolerance parameters. It also includes specifying whether convergence is checked at each iteration, at fixed intervals, or based on transformed residuals.

4.3.1 Backward/forward error perspectives for checks

Backward error assesses how much the computed result would need to change to satisfy the problem exactly under the algorithm’s assumptions. Forward error measures the difference between the computed and true solutions, which can be difficult to compute directly.

While many systems report residual norms rather than forward error, an informed interpretation using backward/forward concepts can guide the choice of meaningful checks. It also helps calibrate tolerances used to compare results across runs.

4.4 Regression testing for numerical outputs

Regression testing verifies that new code changes do not alter numerical outputs beyond acceptable bounds. For numerics, tests often use tolerance checks rather than exact equality, especially for floating-point vectors and derived metrics.

Good regression tests target the most important quantities: residual behavior, convergence rates, objective values, or key intermediate statistics. Tests should cover typical scenarios and edge cases, ensuring that numerical stability remains intact as the software evolves.

4.4.1 Golden tests, baselines, and update policies

Golden tests compare current outputs against stored reference results (“baselines”). Baselines can be stored as full outputs or compact summaries, depending on the cost of comparison.

Update policies should specify when and how baselines may be refreshed. For example, baseline updates may be allowed only when changes are intentional and accompanied by documentation of numerical impact. Without such policies, regression testing can degrade into a mechanism for ignoring discrepancies.

5 Verification, validation, and auditing

5.1 Correctness checks beyond exact equality

Numerical verification aims to confirm correctness despite floating-point variation. Exact equality checks are often too strict; instead, computations can be evaluated using residuals, conservation laws, bounds, and consistency relations.

Sanity bounds are particularly useful: verifying that intermediate values remain finite, that probabilities sum within tolerance, or that energy-like quantities follow expected trends can catch many failures early. Invariant tests provide a way to validate the computation’s internal coherence.

5.1.1 Invariant tests, conservation laws, and sanity bounds

Invariant tests check that certain properties remain unchanged under the computation, such as mass conservation in a discretized model or symmetry relations in derived matrices. Conservation laws often provide strong diagnostic power because they reflect underlying model structure.

Sanity bounds verify that key quantities remain within plausible ranges. These checks can be combined with numerical diagnostics like condition estimates or monitoring of norm growth to detect divergence or numerical overflow.

5.2 Cross-implementation comparisons

Cross-implementation comparison evaluates whether results agree when computed by different software stacks or independent code paths. Agreement within tolerance supports confidence, while consistent disagreement may indicate modeling errors or algorithmic mismatches.

A “same result” strategy relies on controlled inputs and agreed comparison metrics. Independent implementations reduce the risk that a shared bug affects both versions, but they also require careful documentation of differences in defaults and preprocessing.

5.2.1 Independent implementations and “same result” strategies

Independent implementations should share the same mathematical specification even if their code differs internally. To make comparisons meaningful, experiments should align on preprocessing, scaling, solver settings, and stopping rules.

“Same result” strategies often involve comparing outputs at multiple stages: final solutions, intermediate residual norms, and objective evaluations. This helps distinguish cases where errors cancel out in final values from those where the entire computation diverged.

5.3 Unit tests vs integration tests for numerics

Unit tests validate small components—individual functions, transforms, or linear algebra utilities—often with deterministic inputs chosen to make expected behaviors clear. Integration tests validate the full pipeline, ensuring that components interact correctly and that end-to-end results meet acceptance criteria.

For numerical systems, integration tests are valuable because real workflows combine multiple sources of variation: data handling, solver selection, configuration loading, and output formatting. Unit tests alone may miss issues introduced by orchestration or parameter wiring.

5.4 Provenance capture and computational auditing

Computational auditing records the information needed to interpret and verify results after the fact. Provenance capture includes parameters, solver options, random seeds, input hashes, and details about generated artifacts such as checkpoints or intermediate files.

Auditing also benefits from structured logs that trace the computation: what decisions were made, what defaults were used, and what outputs were produced at each step. When combined with versioned code and data, provenance forms a defensible record that enables independent verification.

5.4.1 Recording parameters, solver options, and artifacts

Parameter recording should include every quantity that can change numerical behavior: tolerances, iteration limits, preconditioner choices, regularization settings, and normalization constants. Solver options often hide in configuration files, so they should be surfaced explicitly in logs or metadata.

Artifacts such as trained models, computed matrices, or solver checkpoints should be saved with identifiers and hashes. These allow later comparisons to use the exact intermediate objects associated with reported results.

6 Parallelism and performance-related variability

6.1 Thread scheduling and reduction-order effects

Parallel execution can change the order of operations, especially in reductions like summing arrays. Since floating-point addition is not associative, different reduction orders can produce different rounding outcomes.

Thread scheduling introduces additional variability when work partitioning or synchronization points vary run to run. Even when the overall computation is “the same,” nondeterministic scheduling can change the sequence of floating-point operations.

6.1.1 Non-associativity of floating-point addition

Floating-point addition satisfies neither associativity nor distributivity in general due to rounding. As a result, grouping terms differently leads to distinct rounding paths and potentially different final results.

Reproducibility strategies aim to control operation order or to bound the variability through deterministic reduction schemes, compensated summation, or tolerance-based comparisons. The choice depends on performance constraints and the required strictness of matching.

6.2 GPU determinism considerations

GPUs often execute operations in ways that can be nondeterministic, particularly for parallel reductions and certain atomic operations. Frameworks may provide determinism controls, but they can require disabling optimizations or changing kernel choices.

Reproducibility in GPU settings depends on careful alignment of driver versions, framework versions, and determinism settings. Additionally, mixed precision and tensor-core optimizations can further influence numerical outcomes.

6.3 Scaling studies and reproducibility under load

Scaling experiments examine how results change with problem size and resource allocation. Under load, systems may exhibit different scheduling, memory behavior, and communication patterns, which can affect reproducibility.

A reproducible scaling study documents resource configuration: number of nodes, GPUs, workers, thread counts, batch sizes, and communication topology. It should also separate “algorithmic changes” from “runtime variability,” so that changes due to scaling do not confound numerical comparisons.

6.4 Benchmarking methodology for stable comparisons

Benchmarking aims to compare performance or numerical behavior reliably. For reproducibility, benchmarks should use controlled inputs, fixed seeds where stochasticity exists, and consistent run counts.

Stable benchmarking also requires handling warm-up effects (such as just-in-time compilation), caching behavior, and system noise. Reporting both timing and numerical summary metrics can reveal whether performance changes coincide with numerical drift.

7 Documentation and sharing for reproducibility

7.1 Reproducibility checklists for computational studies

Checklists provide a systematic way to ensure essential reproducibility elements are covered. A typical checklist includes items such as data version identifiers, code version hashes, dependency versions, randomness controls, parameter lists, and environment details.

For numerical studies, checklists should also include the comparison method: the exact tolerance criteria and error metrics used for evaluating agreement. This converts reproducibility from a vague goal into a measurable statement.

7.2 Minimum information standards (data, code, parameters)

Minimum information standards define what must be disclosed so others can recreate results. Usually this includes the dataset snapshot or a way to obtain the exact data, the executable code or instructions to build it, and the parameters that configure the run.

For numerics, additional required information often includes solver settings, stopping rules, and precision choices. Without these, reproductions can unintentionally run a different numerical procedure than the one intended.

7.3 Repository organization and artifact publishing

Repository organization improves discoverability and reduces ambiguity. Clear directory structures for code, configuration files, data processing scripts, and run outputs make it easier to follow the computational path from raw data to published results.

Artifact publishing may include compiled binaries, container images, checkpoints, and metadata files. Including checksums and version identifiers helps ensure that users fetch the intended artifacts rather than near equivalents.

7.4 Automated scripts for environment setup and runs

Automation reduces the risk of manual mistakes. Scripts can install dependencies, configure environment variables, set seeds, prepare data, and execute runs with specified arguments.

A good setup script is itself versioned, and it should produce logs and structured outputs. When possible, the script should verify environment constraints (such as required versions or precision modes) before execution.

8 Tooling and best practices

8.1 Version control workflows (code and configs)

Version control tracks code and configuration changes that affect numerical outcomes. Reproducibility improves when configuration is stored alongside code and changes are reviewed and tagged.

A common best practice is to associate each published result with a specific commit identifier and a set of configuration files. This allows later retrieval of the exact settings used to generate results.

8.2 Workflow managers and experiment tracking

Workflow managers and experiment tracking systems coordinate multi-step computations and maintain records of runs. They can capture inputs, outputs, parameters, and logs in a structured manner, which supports auditing and comparison.

Experiment tracking tools often help aggregate results across multiple runs and hyperparameter settings. When configured properly, they can also enforce metadata completeness and prevent missing critical configuration.

8.2.1 Metadata schemas for experiments

Metadata schemas specify which fields must be recorded, such as dataset identifiers, solver options, seeds, precision mode, hardware details, and evaluation metrics. Schemas encourage consistency across projects and teams.

A well-designed schema also captures units, scaling factors, and the meaning of each tolerance or threshold. This reduces interpretive errors when reading results later.

8.3 Automated formatting, linting, and style enforcement

Automated formatting and linting do not directly guarantee numerical reproducibility, but they help prevent accidental changes that alter behavior. Consistent code style can reduce the likelihood of subtle bugs introduced during refactoring and can improve reviewability of changes.

In numerics, style tools can also enforce best practices like explicit casting, consistent use of reduction operations, or avoidance of ambiguous defaults that vary by interpreter settings.

8.4 Continuous integration for numerical reproducibility

Continuous integration (CI) runs automated checks on each code change. For reproducibility, CI can execute numerical regression tests and verify that outputs remain within specified tolerances.

CI should control environment variability as much as feasible: use pinned dependencies, fixed seeds, and deterministic settings. When full determinism is impossible, CI can still use robust tolerance-based comparisons and report drift trends over time.

8.5 Common pitfalls and how to avoid them

Common pitfalls include comparing floating-point results with strict equality, failing to log seeds and tolerances, and relying on implicit defaults in libraries. Another recurring issue is using nondeterministic operations in parallel contexts without documenting determinism settings.

Avoidance strategies include explicit logging, tolerance-based checks with justification, use of deterministic computation modes when required, and validation steps that confirm numerical invariants. Additionally, avoiding hidden state (like cached preprocessing outputs without versioning) helps prevent “it worked once” scenarios.

9 Case studies and templates

9.1 Reproducing a linear solver experiment

Reproducing a linear solver experiment requires capturing the full problem specification: the matrix or operator definition, right-hand side vectors, any scaling applied, and the solver configuration. Key solver details include the iterative method, preconditioner choice, and convergence criteria.

A complete reproduction record also documents how matrices are formed and stored. Differences in sparse formats or ordering can change numerical behavior and performance, making reproducibility more reliable when storage conventions are recorded.

9.1.1 Capturing solver settings and stopping criteria

Solver settings should include maximum iterations, residual computation method, norm type (if applicable), and the meaning of tolerances. For example, the distinction between absolute and relative tolerance affects termination behavior.

Stopping criteria should be logged with enough detail to recreate the exact iteration of acceptance. Recording both the achieved residual and the number of iterations helps interpret how tolerant comparisons relate to convergence quality.

9.2 Reproducing a stochastic simulation

Stochastic simulation reproduction depends on controlling randomness and documenting the statistical interpretation of outputs. Reporting a single run can be insufficient; reproducible analysis often uses multiple seeds and summarizes variability with confidence intervals or variance estimates.

The simulation record should include the seed strategy, the number of trials, and how random choices are distributed across components. It should also note any stochasticity in initialization, sampling, or environmental transitions.

9.2.1 Seed strategy and variance reporting

A seed strategy might use a base seed with deterministic offsets per trial or generate seeds via a reproducible scheme. The simulation should document how seeds map to trials and which generator implementations are used.

Variance reporting should specify metrics used to summarize outcomes and the units of measurement. Comparing results across reproductions typically involves matching both means (within tolerances) and dispersion measures, rather than requiring exact trajectories.

9.3 Reproducing an optimization run

Optimization run reproducibility requires logging objective functions, constraints, data splits, and training schedules. Critical details include optimizer type, learning rate policy, batch size, gradient accumulation behavior, and any regularization.

Because optimization can be sensitive to initialization and stochastic mini-batch order, reproducibility often relies on fixed seeds and deterministic data shuffling. When determinism is limited, the expected outcome should be expressed in terms of tolerances and performance distributions.

9.3.1 Logging optimization states and hyperparameters

Logging should capture hyperparameters and their schedules over time, including initial settings and any per-epoch adjustments. Optimization states such as momentum buffers, adaptive optimizer statistics, and checkpoint metadata help interpret differences after resumed runs.

To support reproduction, the run record should also store evaluation metrics at defined checkpoints. Recording achieved hyperparameter values, not just their base forms, helps others align with the exact optimization trajectory taken.