1 Definition and scope of numerical reproducibility

Numerical reproducibility describes the extent to which a computational workflow yields the same numerical outcome when the computation is repeated under comparable conditions. “Comparable conditions” typically includes identical or closely matched inputs, algorithmic settings, and execution environment features that influence floating-point operations and control flow. Because computers use finite-precision arithmetic and may execute instructions differently across runs, exact agreement is not always expected; many workflows instead aim for agreement within specified tolerances or within statistically consistent bounds.

The concept is central to scientific computing, engineering simulation, machine learning experimentation, and any domain where numerical results support decisions. Reproducibility is often discussed in terms of whether differences are explainable, quantifiable, and stable under controlled changes.

1.1 Reproducibility vs. repeatability in computing

In computing practice, *repeatability* usually refers to obtaining the same result when the same procedure is executed multiple times under essentially unchanged conditions. *Reproducibility* is broader: it emphasizes obtaining consistent outcomes even when runs involve different systems, different compilation settings, or different implementations, provided the experiments are specified and controlled appropriately. For numerical work, repeatability may be easier to achieve than cross-platform reproducibility because hardware and runtime details can change.

A key distinction is that reproducibility focuses on the ability to replicate results using the published workflow description, whereas repeatability focuses on repeated execution within a single setup.

1.2 Determinism and the role of execution environment

Numerical computations depend on determinism at multiple layers. At the algorithmic level, the sequence of operations must be fixed by the inputs and control flow. At the software level, the runtime must behave consistently (for example, library functions must follow the same numerical pathways). At the hardware level, instruction selection, parallel execution patterns, and floating-point unit behavior can change the order and manner in which intermediate results are computed.

An execution environment includes processor features (such as support for fused multiply-add), the compiler toolchain, runtime libraries, operating system scheduling behavior, and parallel communication libraries. Even when the algorithm is deterministic, the environment can introduce nondeterministic execution or different arithmetic sequences.

1.3 Metrics: exact match, tolerance-based match, and statistical similarity

Reproducibility is rarely measured by a single notion of “equality.” Common evaluation modes include:

  • Exact match (bitwise equivalence): The computed outputs match at the representation level, including sign bits, NaN payloads, and rounding effects.
  • Tolerance-based match: Differences between outputs remain within absolute or relative tolerances, sometimes expressed in units of least precision (ULPs) or as norms of error vectors.
  • Statistical similarity: For stochastic methods, the distribution of outcomes (means, variances, quantiles, or other summary statistics) remains consistent, rather than each individual run producing the same value.

Choosing a metric depends on the role of the result: whether it is used as an intermediate quantity, a final reported number, or a component inside a larger pipeline.

1.4 Sources of nondeterminism in numerical software

Nondeterminism can arise from many sources, including:

  • Concurrency and parallel execution: Scheduling can alter the order of operations, especially in reductions and aggregations.
  • Library-specific behavior: Some numerical libraries select implementations at runtime based on detected hardware or environment variables.
  • Randomness: Stochastic algorithms and data augmentation require careful seeding and controlled sampling.
  • Floating-point environment differences: Variations in rounding mode, denormal handling, or exception behavior can change results.
  • Build and optimization differences: Compiler flags can alter instruction sequences, re-associate operations, or enable transformations that affect rounding.

Understanding the origin of variability is a prerequisite for designing experiments that can be repeated with predictable outcomes.

2 Why numerical results can differ

Even when “inputs” are identical in a human sense, numerical results may diverge because finite-precision arithmetic and execution details can cause the same mathematical problem to be realized through slightly different numerical paths. Over many operations, small discrepancies may amplify, particularly in unstable algorithms or poorly conditioned problems.

2.1 Floating-point arithmetic and finite precision

Computers typically represent real numbers using floating-point formats with limited mantissa precision. Operations are rounded to fit the format, and these rounding steps may occur at different points depending on how expressions are compiled and executed.

2.1.1 Rounding modes and precision formats

Rounding behavior affects every floating-point operation. Most systems use round-to-nearest behavior, but variations exist via rounding modes and environment settings.

2.1.1.1 IEEE 754 implications (e.g., NaNs, subnormals, infinities)

IEEE 754 defines standardized behavior for special values and edge cases:

  • NaNs: Not-a-Number values can carry payloads; comparisons and propagation may differ depending on how libraries handle them.
  • Subnormals (denormals): Very small magnitudes may be represented differently or treated as zero depending on hardware and performance settings.
  • Infinities: Division by zero and overflow produce infinities in a standardized way, yet subsequent operations can still diverge if intermediate values differ slightly.
  • Exception flags and signaling behavior: Handling of underflow, overflow, or invalid operations can influence control flow and function results in some software stacks.

These details are typically not visible to users unless they inspect raw representations or carefully craft test cases near numerical extremes.

2.2 Order of operations and associativity

Floating-point addition and multiplication are not associative in general. As a result, reordering operations can change rounding points and yield different final results even for the same algebraic expression.

2.2.1 Summation and reduction effects (e.g., catastrophic cancellation)

Summations are particularly sensitive. When adding numbers with widely varying magnitudes, rounding can eliminate contributions of smaller terms. In expressions involving subtraction of nearly equal values, catastrophic cancellation can reduce significant digits, increasing the impact of subsequent rounding. Different reduction trees (common in parallel code) can therefore change results.

2.3 Parallelism and concurrency effects

Parallel execution can alter the sequence of operations. In many parallel numeric programs, partial results are computed independently and then combined, producing variability if combination order changes.

2.3.1 Thread scheduling and race conditions

Even in code that avoids explicit data races, scheduling can influence when partial results are combined, especially if synchronization points are not uniform. Race conditions can produce incorrect results outright, but subtler nondeterminism can occur when operations are logically safe yet order-dependent in floating-point arithmetic.

2.3.2 MPI reductions and reduction tree differences

In distributed computations, collective operations such as reductions are implemented using trees or algorithms that may differ between runs or configurations. The order of pairwise combination depends on communicator topology, message ordering, and implementation details, leading to distinct rounding outcomes.

2.4 Hardware-specific behaviors

Hardware capabilities influence numerical instruction sequences and precision pathways.

2.4.1 SIMD, fused multiply-add (FMA), and instruction selection

Modern processors use SIMD vector instructions and may support fused multiply-add. With FMA, the multiplication and addition can be performed with a single rounding step, which can differ from separate multiply then add operations. If the compiler or library chooses different instruction sequences on different platforms, results may shift.

2.4.2 Compiler optimizations and re-association

Compilers can apply optimizations that change evaluation order, such as re-associating arithmetic expressions or using wider registers. Some toolchains default to flags that allow transformations under floating-point math assumptions, which can change reproducibility. Link-time optimization and automatic vectorization can also change the computation path.

2.5 Randomness in numerical algorithms

Randomness introduces variability by design. Even with identical code and data, sampling variability produces different trajectories and outcomes.

2.5.1 Pseudorandom number generators and seeding

Pseudorandom number generators depend on initial seeds, algorithm version, and usage pattern (how many random numbers are consumed, and in what order). Parallel generation can complicate seeding, especially when threads or ranks consume random streams in differing orders.

2.5.2 Stochastic solvers and sampling variability

Stochastic optimization, Monte Carlo sampling, and randomized linear algebra techniques yield results as random variables. Reproducibility in this context often means that repeated runs produce statistically consistent distributions given the same specified RNG behavior, not that each individual sample matches.

3 Reproducibility categories in numerical experiments

Different experiments demand different notions of “the same result.” Categorizing reproducibility helps clarify what level of agreement is targeted and how it should be assessed.

3.1 Bitwise reproducibility

Bitwise reproducibility requires exact agreement of all output bits for each run. This is the strongest form and can be difficult to achieve across different hardware, compilers, or parallel configurations. It typically requires controlling floating-point environment, enforcing deterministic reduction order, disabling nondeterministic library paths, and ensuring identical instruction sequences.

3.2 Reproducibility within numerical tolerances

Tolerance-based reproducibility allows small differences while ensuring that deviations remain within an acceptable range. The tolerance may be defined in absolute terms, relative terms, or using measures tied to floating-point resolution such as ULPs. This category is common in engineering workflows, where downstream decisions rely on accuracy rather than exact bit patterns.

3.3 Statistical reproducibility for stochastic methods

For stochastic algorithms, statistical reproducibility targets consistency of output distributions. This can be operationalized by comparing estimated means, variances, confidence intervals, or other distributional metrics under repeated runs. Proper comparison often requires attention to sample size and the variance introduced by the random process.

3.4 Cross-platform and cross-implementation reproducibility

Cross-platform reproducibility aims for consistent outcomes across different machines, operating systems, or hardware generations, often including different compiler toolchains or numerical backends. It typically relies on tolerance-based or statistical notions because exact bitwise matching may be infeasible.

3.5 Reproducibility under changing compilation settings

Compilation settings—optimization level, floating-point model flags, vectorization settings, and math library versions—can affect evaluation order and precision. Reproducibility under changing compilation settings addresses how outcomes remain stable when such settings vary within a controlled range, highlighting whether results are robust to tooling differences.

4 Designing reproducible computations

Improving reproducibility requires deliberate design choices across the full computational pipeline: how inputs are specified, how arithmetic behavior is controlled, how parallelism is managed, and how randomness and environments are documented.

4.1 Input specification and data provenance

Reproducible outcomes begin with unambiguous inputs and histories of how those inputs were created.

4.1.1 Versioning datasets, parameters, and preprocessing

Data provenance includes dataset versions, preprocessing scripts, normalization steps, feature extraction logic, and parameter files. Versioning allows a repeated experiment to use the same data transformations rather than merely the same raw files. Parameter versioning matters as well because small differences in tolerances, stopping criteria, or regularization constants can change numerical trajectories.

4.2 Controlling numerical behavior

Reproducibility improves when numerical behavior is constrained so that computation paths remain stable.

4.2.1 Fixed-point of algorithmic choices (e.g., solvers, tolerances)

Algorithmic choices form a “fixed point” of the workflow: selecting a particular solver, preconditioner, discretization scheme, and tolerance policy. Two runs that use different but mathematically similar settings may follow divergent numerical paths. Documenting and locking these choices reduces variability.

4.2.2 Consistent floating-point settings (where available)

When software and hardware permit, fixed floating-point configuration—such as rounding mode, denormal handling, and exception behavior—can improve matching. In practice, these settings are not always accessible or consistent across environments, so workflows often pair configuration control with tolerance-based evaluation.

4.3 Managing parallel execution

Parallelism must be approached with awareness that reductions and scheduling influence floating-point rounding order.

4.3.1 Deterministic reduction strategies

Deterministic reductions enforce a consistent combination order, often by fixing reduction trees or using numerically careful aggregation methods. Some approaches use long accumulators or compensated summation to reduce sensitivity to ordering, while others enforce a fixed sequence of combining partial results.

4.3.2 Stable work partitioning

Stable partitioning refers to ensuring that each thread or process handles the same subset of data in a repeatable manner. If work distribution changes across runs, floating-point order and the sequence of operations may also change. Designing deterministic partition logic reduces this source of variability.

4.4 Controlling randomness

Randomness must be made reproducible by defining RNG behavior precisely.

4.4.1 Seeding conventions across ranks/threads

A common approach is to use seeding conventions that map seeds deterministically to ranks and threads, such as deriving per-rank seeds from a master seed. This reduces dependency on thread scheduling order, though it requires careful design so that each parallel worker consumes its random stream predictably.

4.4.2 Recording RNG state and algorithm parameters

Beyond initial seeds, it can be important to record RNG state at key checkpoints, especially for restartable jobs. Recording algorithm parameters that affect random consumption—such as batch sizes, sampling schedules, and augmentation configuration—helps ensure the same number and ordering of RNG calls.

4.5 Build and environment capture

Reproducibility depends heavily on tooling. Capturing the build and runtime environment reduces unintentional differences.

4.5.1 Containerization and dependency pinning

Containers and pinned dependencies help keep operating system packages, library versions, and runtime components consistent. The goal is to reduce “hidden” changes such as different BLAS implementations or different math library behavior.

4.5.2 Compiler flags and library version tracking

Exact compiler flags, link options, and library versions should be recorded. Floating-point related flags and optimization levels are particularly influential. Tracking the versions of numerical libraries (e.g., BLAS/LAPACK providers, MPI, and FFT libraries) improves the ability to replicate arithmetic behavior.

5 Assessment and testing for reproducibility

Evaluating reproducibility involves defining reference results and selecting comparison metrics that reflect acceptable numerical variation.

5.1 Establishing reference results

Reference results can be obtained by running a trusted configuration on a designated system, or by using higher-precision or more conservative settings as a baseline. The reference should be generated with a clearly documented method, since the act of defining “truth” influences what counts as reproducible.

5.2 Comparing outputs: norms, relative/absolute tolerances, and ULPs

Comparison typically uses metrics such as:

  • Absolute and relative differences per element or aggregated by norms.
  • ULP-based checks where appropriate, reflecting discrepancies at the floating-point representation level.
  • Intermediate quantity checks (e.g., residuals or conserved quantities) when final outputs are insensitive or when errors propagate nonlinearly.

Tolerance choices should be justified by numerical analysis or by validation against known accuracy requirements.

5.3 Regression tests and continuous integration

Regression tests ensure that changes to code, dependencies, or build settings do not degrade reproducibility or accuracy. Continuous integration can run a small suite of reproducibility checks on representative configurations, flagging deviations early.

Because full bitwise reproducibility may be expensive to enforce, many projects use tolerance-based regression tests and reserve stronger checks for critical components.

5.4 Sensitivity analysis and uncertainty about “acceptable” divergence

Sensitivity analysis examines how outputs respond to controlled perturbations in inputs, parameters, and numerical settings. This helps determine whether observed differences are expected or indicative of a fault. Establishing an “acceptable” divergence requires understanding both numerical behavior and application requirements, including how errors influence final decisions.

5.5 Detecting hidden sources of variance

A reproducibility failure may stem from subtle causes such as environment variables, differences in math library selection, or non-pinned dependencies. Systematic detection can involve bisecting changes, logging library versions, inspecting floating-point settings, and verifying that parallel reduction order matches expectations.

6 Algorithmic strategies to improve reproducibility

Beyond environment control, algorithmic design can reduce sensitivity to rounding order and parallel execution effects.

6.1 Numerically stable formulations

Reformulating computations to improve numerical stability can reduce error growth. Stable formulations limit amplification of rounding errors and reduce dependence on operation ordering. Stability often involves careful treatment of subtraction, division, and scaling.

6.2 Compensated summation and error-reduction techniques

Compensated summation methods attempt to correct lost low-order bits during accumulation. These techniques can make results less sensitive to reduction ordering and floating-point precision, improving both tolerance-based agreement and, in some setups, stronger reproducibility.

6.3 Reproducible linear algebra practices

Linear algebra routines benefit from practices that control pivoting, ordering, and scaling decisions.

6.3.1 Stable ordering in matrix operations

Operations such as sparse matrix traversal and dense matrix multiplication may iterate indices in different orders depending on storage format or library implementation. Stable iteration order and consistent sparse data handling reduce variability. When exact agreement is critical, using deterministic sparse kernels can help.

6.4 Reproducible iterative methods

Iterative solvers accumulate rounding error across iterations. Controlling iteration behavior improves reproducibility.

6.4.1 Consistent stopping criteria

Stopping criteria based on residual norms, tolerances, or maximum iteration counts should be defined consistently. Small differences in computed residuals can change the iteration where the method stops, leading to larger downstream changes. Using deterministic criteria and recording intermediate thresholds helps reduce this effect.

6.5 Reproducible interpolation and discretization choices

Discretization affects numerical pathways through mesh generation, quadrature rules, and interpolation. Ensuring the same discretization strategy and deterministic ordering of grid traversal contributes to repeatable outcomes. For adaptive methods, reproducibility also depends on deterministic refinement decisions driven by computed error estimates.

7 Documentation and reporting standards

Documentation translates reproducibility goals into actionable records so that others can replicate the computation reliably.

7.1 Reporting computational environment

A report should identify the computing environment: processor type, operating system, runtime versions, and key numerical libraries. When possible, environment summaries include CPU instruction capabilities and whether specialized math modes are enabled.

7.2 Describing numerical settings and tolerances

Numerical settings include solver configuration, convergence tolerances, iteration limits, discretization parameters, and any custom error controls. Reporting these values helps readers understand what level of numerical variation was permitted.

7.3 Sharing code and run scripts

Sharing code artifacts and run scripts supports replication by capturing the exact sequence of commands used to produce results. It also helps isolate sources of nondeterminism, such as environment variables, scheduling parameters, or data loading logic.

7.4 Including reproducibility checklists in publications

Many projects adopt structured checklists to document reproducibility-critical items: data versions, preprocessing steps, RNG seeds, parallel configuration, compiler flags, and dependency versions. Checklists improve completeness and reduce the likelihood that reviewers miss key details.

8 Common pitfalls and troubleshooting

Reproducibility failures are often systematic and can be resolved through targeted investigation.

8.1 “Same inputs” not being the same computation

Two runs can differ even if input data appears identical. Causes include differences in preprocessing, data shuffling order, implicit type conversions, or unit scaling. Verifying the full input pipeline—raw data through final tensors or arrays—is often necessary.

8.2 Unintended nondeterminism from libraries

Some libraries may use nondeterministic algorithms unless configured otherwise, particularly for parallel operations or performance modes. Troubleshooting includes checking library documentation for deterministic modes and verifying whether runtime environment variables change algorithm selection.

8.3 Effects of different BLAS/LAPACK backends

Different BLAS/LAPACK implementations can produce different floating-point results due to distinct blocking strategies, reduction order, and instruction selection. Even within the same nominal version, vendor variants may differ. Tracking and standardizing the backend can reduce variability.

8.4 Divergence due to ill-conditioning

Ill-conditioned problems amplify perturbations and rounding error, causing trajectories to diverge more than expected. In such cases, reproducibility may be limited by the underlying mathematical sensitivity. Addressing this may involve improving conditioning through scaling, regularization, or algorithmic changes to stable formulations.

8.5 Debugging workflow for reproducibility failures

A practical debugging workflow includes:

  1. Confirm whether differences are bitwise, tolerance-based, or statistical.
  2. Compare deterministic components first by running in single-threaded mode.
  3. Check data preprocessing and type conversions.
  4. Verify RNG seeding and consumption patterns for stochastic code.
  5. Inspect build flags and library versions.
  6. Isolate the earliest point where outputs diverge using logging or checkpoints.

This staged approach narrows the search from broad causes to localized arithmetic or control-flow differences.

9 Tools and workflow patterns

Workflow tooling can help manage the complexity of reproducing numerical computations across systems and over time.

9.1 Container and virtualization approaches

Containers package runtime dependencies and system libraries, enabling consistent execution environments. Virtualization can provide broader control of system characteristics, though it may introduce performance differences. Together, these tools can reduce the “it worked on my machine” problem.

9.2 Workflow managers and experiment tracking

Workflow managers orchestrate multi-step pipelines and can enforce deterministic execution graphs. Experiment tracking systems record hyperparameters, seeds, dataset versions, and metrics over time, which helps compare outcomes and reproduce specific runs.

9.3 Checkpointing and restart strategies

Checkpointing stores intermediate states so computation can resume deterministically from a known point, provided the RNG state and relevant solver variables are captured. Restart strategies also help when long jobs are interrupted, reducing the chance that a resumed run follows a different path due to changed scheduling or consumed randomness.

9.4 Reproducible build practices

Reproducible builds aim for consistent binary artifacts given the same source and configuration. For numerical reproducibility, capturing compiler versions, flags, and linked dependencies is crucial because the generated instruction sequence impacts floating-point behavior.

9.5 Automated provenance capture

Automated provenance capture records metadata during execution, including git commit identifiers, package hashes, configuration files, and runtime parameters. This approach reduces manual bookkeeping and supports systematic auditing when reproducibility issues arise.

Numerical reproducibility is closely related to several concepts that address accuracy, stability, and experimental rigor. These concepts often overlap but emphasize different properties.

10.1 Numerical stability vs. reproducibility

Numerical stability concerns how errors behave as computations proceed, especially regarding rounding error amplification. Reproducibility concerns whether results match under repeated computation. A stable algorithm can still vary across runs if execution order changes, while a reproducible workflow can yield inaccurate results if the underlying method is unstable.

10.2 Conditioning, error propagation, and backward error

Conditioning measures sensitivity of the mathematical problem to perturbations in inputs. Error propagation describes how those perturbations evolve through an algorithm. Backward error frames the computed result as the exact solution of a nearby problem. Together, these ideas help explain why small arithmetic differences may or may not lead to large output changes, informing tolerance selection.

10.3 Verification, validation, and benchmarking

Verification checks that the implemented model solves the equations correctly (within numerical tolerances). Validation checks that the model represents real-world phenomena or target behavior. Benchmarking evaluates performance and accuracy across methods or implementations. Reproducibility supports all three by ensuring that comparisons reflect genuine algorithmic differences rather than uncontrolled variability.