1 Fundamental concepts
Automatic differentiation is a collection of techniques for computing derivatives of functions encoded as computer programs. It applies the rules of differential calculus directly to the operations performed during execution, rather than manipulating algebraic formulas or estimating slopes from nearby sample points. As a result, it can obtain exact derivatives up to ordinary floating-point roundoff error.
The term covers methods for scalar-valued and vector-valued functions, as well as routines that produce gradients, Jacobians, Hessians, and related derivative products. It is used whenever a program can be decomposed into elementary operations with known derivatives.
1.1 Definition and scope
Automatic differentiation, often abbreviated autodiff, refers to any method that systematically differentiates a program by tracking how each intermediate value depends on its inputs. The technique works on ordinary numerical code, provided the operations used are differentiable or have defined derivative rules.
Its scope is broader than a single algorithm. In practice, it includes forward-mode, reverse-mode, and mixed-mode methods, along with implementation strategies such as operator overloading, source transformation, and tracing. These approaches are found in scientific software, optimization toolkits, and machine learning frameworks.
1.2 Relationship to calculus
Automatic differentiation is grounded in standard calculus. Every computation is viewed as a composition of simpler maps, and the derivative of the whole program is assembled from the derivatives of its parts. This makes the method mathematically exact in the sense of calculus, even when the program itself is large or complex.
1.2.1 Chain rule
The chain rule is the central principle behind automatic differentiation. If one operation depends on the output of a previous operation, then the derivative of the composite function is obtained by multiplying the local derivative of each step by the derivative flowing in from earlier steps.
In a program, this rule is applied repeatedly across the full sequence of instructions. The result is a derivative calculation that mirrors the original computation, step by step.
1.2.2 Computational graphs
A computational graph represents a program as nodes and edges, where nodes correspond to operations or intermediate values and edges represent data dependencies. Such a graph makes the application of the chain rule explicit.
By traversing the graph in the appropriate direction, an autodiff system can accumulate derivatives efficiently. This representation is especially useful for programs with branching reuse of intermediate results.
1.3 Comparison with symbolic and numerical differentiation
Symbolic differentiation manipulates formulas using algebraic rules and can produce closed-form expressions. It is powerful for compact mathematical expressions, but it may cause expression growth and can struggle with code that is not naturally written as a symbolic formula.
Numerical differentiation approximates derivatives by evaluating the function at nearby points, often with finite differences. It is easy to implement, but its accuracy depends on step size and can be degraded by rounding error and cancellation. Automatic differentiation avoids these limitations by computing derivatives directly from the program’s operations.
2 Core modes of automatic differentiation
Automatic differentiation is commonly divided into forward mode and reverse mode. These two modes differ in the direction in which derivative information is propagated and in how they scale with the number of inputs and outputs.
2.1 Forward mode
Forward mode propagates derivative information alongside the original computation from inputs toward outputs. Each variable is paired with a derivative-like quantity that records how that variable changes with respect to a chosen input direction.
This mode is often efficient when a function has relatively few inputs and one wishes to differentiate with respect to those inputs individually.
2.1.1 Tangent propagation
In tangent propagation, each intermediate value carries a tangent, which represents its instantaneous rate of change. As the program executes, derivatives are updated using local rules for each operation.
This approach is straightforward and mirrors the original control flow closely. It is especially natural for computing directional derivatives.
2.1.2 Dual numbers
Dual numbers provide a compact algebraic device for forward-mode differentiation. A dual number has a real part and a nilpotent infinitesimal part, and arithmetic on these objects encodes derivative information automatically.
By evaluating a program with dual numbers instead of ordinary scalars, one obtains the derivative in the infinitesimal component. This method is widely used in language libraries and educational implementations.
2.2 Reverse mode
Reverse mode propagates derivative information backward from outputs to inputs. It is particularly efficient for functions with many inputs and a single scalar output, because one reverse pass can compute the gradient with respect to all inputs.
This mode stores intermediate results during the forward computation and uses them later when traversing the program in reverse.
2.2.1 Adjoint propagation
Adjoint propagation uses adjoints, also called sensitivities, to measure how much a final output changes with respect to each intermediate quantity. During the reverse sweep, these quantities are accumulated from the output back to the inputs.
The method is central to optimization and machine learning because it can efficiently produce gradients of scalar objectives.
2.2.2 Backpropagation
Backpropagation is a widely known application of reverse-mode differentiation in neural networks. It computes error signals layer by layer from the output toward the input, using derivative rules for each operation or layer.
Although often associated specifically with deep learning, backpropagation is a general reverse-mode technique and predates modern neural network frameworks.
2.3 Mixed-mode differentiation
Mixed-mode differentiation combines forward and reverse methods to suit more complex derivative tasks. For example, a system may use reverse mode to obtain gradients and then apply forward mode to differentiate those gradients again.
This hybrid approach is useful for higher-order derivatives, Hessian-vector products, and situations where neither pure forward mode nor pure reverse mode is ideal.
3 Mathematical foundations
The mathematics of automatic differentiation rests on local derivative rules, function composition, and sensitivity analysis. These ideas allow a program’s overall derivative to be built from the derivative of each elementary step.
3.1 Derivatives of elementary operations
Most autodiff systems begin with a library of derivative rules for basic arithmetic and common functions such as exponentials, logarithms, trigonometric functions, and comparisons where differentiable. Each rule describes how a small operation transforms both values and derivative information.
Because complex code can be expressed as combinations of these primitives, derivative computation reduces to repeatedly applying a finite set of local formulas.
3.2 Composition of functions
A computer program may be viewed as a composition of many smaller functions. Automatic differentiation exploits this structure by differentiating each component and combining the results according to the chain rule.
This compositional viewpoint makes the method applicable to long programs, nested procedures, and modular software designs. It also clarifies why autodiff is exact for the operations included in the computation.
3.3 Sensitivity analysis
Automatic differentiation is closely related to sensitivity analysis, the study of how outputs respond to small changes in inputs. In scientific computing and engineering, sensitivities are used to understand parameter influence, stability, and system response.
Autodiff provides these sensitivities efficiently and with high numerical reliability. It is therefore useful in simulation-based modeling and design studies.
3.4 Higher-order derivatives
Higher-order derivatives, such as second and third derivatives, can also be computed by automatic differentiation. This is typically done by applying differentiation more than once, sometimes using mixed-mode strategies to improve efficiency.
Second-order information is important for curvature analysis, Newton-type methods, and uncertainty estimation. However, higher-order calculations may increase computational and memory demands.
4 Implementation methods
Automatic differentiation can be implemented in several ways, depending on the programming language and performance requirements. The main differences concern how derivative rules are inserted into code and how intermediate computations are recorded.
4.1 Source transformation
Source transformation rewrites a program into another program that computes derivatives. A compiler or preprocessing step analyzes the original code and generates derivative code with explicit derivative variables and operations.
This approach can produce efficient execution because the derivative code is specialized and does not require runtime inspection of every operation.
4.2 Operator overloading
Operator overloading extends arithmetic operators and functions so they act on special derivative-aware objects. When the program executes, these overloaded operations record or propagate derivative data automatically.
This method is easy to integrate into existing languages and is common in user-facing libraries. Its simplicity, however, can introduce runtime overhead.
4.3 Program tracing
Program tracing records the operations performed during execution, then differentiates the recorded trace. The trace may be reused to perform derivative calculations after the forward pass has completed.
Tracing is particularly effective when the control flow is fixed during execution. It supports dynamic programs, though it may require care when branches or loops depend on input values.
4.4 Graph-based execution
Graph-based execution organizes computations into explicit nodes and edges that can be differentiated later. The graph may be built before execution or assembled during runtime, depending on the system design.
This approach is common in machine learning frameworks, where a graph can support optimization, reuse, and deployment. It also helps separate the mathematical structure of a computation from the mechanics of evaluation.
5 Data structures and representations
The efficiency and flexibility of autodiff depend heavily on how computations are represented internally. Different data structures emphasize readability, execution speed, memory use, or support for dynamic control flow.
5.1 Expression trees
Expression trees represent computations as hierarchical structures in which each node corresponds to an operation and each leaf corresponds to an input or constant. They are useful for symbolic-style reasoning and for building derivative rules recursively.
Although expression trees can be intuitive, they may duplicate shared subcomputations unless additional mechanisms are used.
5.2 Computational graphs
Computational graphs encode dependencies among intermediate values and are better suited than trees for programs with reused results. Each node may feed multiple downstream nodes, reflecting the true data flow of the computation.
This representation supports efficient derivative accumulation and is widely used in autodiff systems for modern software.
5.3 Tape-based systems
Tape-based systems store a sequence of executed operations, often called a tape, during the forward pass. In reverse mode, the tape is replayed backward so that derivative information can be accumulated in the opposite direction.
The tape acts as a record of the computation, including enough information to reconstruct local derivative contributions. This design is common in reverse-mode libraries.
5.4 Static and dynamic graphs
Static graphs are built in advance and then executed, while dynamic graphs are created on the fly as the program runs. Static representations can enable optimization and compilation, whereas dynamic graphs offer greater flexibility for data-dependent control flow.
Both styles are used in practice, and many systems support a mixture of the two. The choice affects usability, performance, and ease of debugging.
6 Output types and derivative products
Automatic differentiation can generate several forms of derivative output. The appropriate product depends on the function’s input and output sizes and on the needs of the application.
6.1 Gradients
A gradient is the vector of partial derivatives of a scalar-valued function with respect to its inputs. It indicates the direction of steepest increase and is fundamental in optimization and learning.
Reverse mode is especially well suited to gradient computation when there are many inputs.
6.2 Jacobians
A Jacobian is the matrix of first-order partial derivatives for a vector-valued function. It describes how each output changes with each input and is important in nonlinear systems and constrained optimization.
Depending on the dimensions of the function, autodiff may compute the full Jacobian or only selected products involving it.
6.3 Hessians
A Hessian is the matrix of second derivatives of a scalar function. It captures local curvature and is used in second-order optimization methods and stability analysis.
Because explicit Hessian formation can be costly, systems often compute Hessian-vector products rather than constructing the full matrix.
6.4 Vector-Jacobian products
A vector-Jacobian product multiplies a vector by a Jacobian from the left. Reverse-mode differentiation naturally computes this quantity and uses it as a building block for gradients.
This product is efficient because it avoids forming the entire Jacobian when only its action on a vector is needed.
6.5 Jacobian-vector products
A Jacobian-vector product multiplies a Jacobian by a vector from the right. Forward mode is naturally suited to this operation, making it useful for directional derivatives and sensitivity calculations.
Like vector-Jacobian products, it provides a compact alternative to explicit matrix construction.
7 Applications
Automatic differentiation is used wherever derivatives are required for large or complex computations. Its reliability and flexibility make it a standard tool in modern numerical software.
7.1 Optimization
In optimization, derivatives guide iterative search methods toward minima or maxima. Autodiff provides gradients and Hessians that can improve convergence and reduce the need for manually derived formulas.
It is widely used in unconstrained and constrained problems, including parameter fitting and design optimization.
7.2 Machine learning
Machine learning relies heavily on gradient-based training. Automatic differentiation supplies the derivatives needed to update model parameters, evaluate loss functions, and implement learning algorithms efficiently.
Its integration with neural network frameworks has made it a core technology for deep learning.
7.3 Numerical simulation
Scientific simulations often involve differential equations, physical models, and iterative solvers. Automatic differentiation helps compute sensitivities of simulation outputs with respect to model parameters or initial conditions.
These derivatives can support calibration, uncertainty quantification, and model comparison.
7.4 Parameter estimation
Parameter estimation seeks values that make a model match observed data. Autodiff enables efficient computation of objective-function derivatives, which are central to least-squares and maximum-likelihood methods.
It is especially valuable when the model is implemented as a substantial computer program rather than a closed-form equation.
7.5 Control and inverse problems
In control theory and inverse problems, one often seeks inputs or parameters that produce desired system behavior. Derivative information helps optimize control policies and reconstruct hidden quantities from measurements.
Automatic differentiation supports these tasks by supplying accurate sensitivities for complex dynamical models.
8 Software and libraries
Many programming environments provide automatic differentiation tools, either as standalone packages or as features built into larger frameworks. These systems differ in language integration, performance, and supported derivative modes.
8.1 General-purpose autodiff frameworks
General-purpose frameworks are designed to differentiate broad classes of programs. They are commonly used in scientific computing and machine learning and often support both forward and reverse modes.
Examples include systems that integrate with Python, Julia, C++, and other languages through libraries or compiler-based tooling.
8.2 Domain-specific tools
Some autodiff tools are tailored to particular domains, such as robotics, physics simulation, or statistical modeling. These packages may provide specialized derivative rules for the structures most common in their field.
By narrowing the problem domain, such tools can offer better performance or more convenient interfaces.
8.3 Language support
Programming language design influences how easily automatic differentiation can be expressed. Languages with operator overloading, multiple dispatch, macros, or compile-time metaprogramming often support autodiff more naturally.
In some environments, derivative computation is exposed as a first-class feature, while in others it is provided by external libraries.
8.4 Performance considerations
Performance depends on execution speed, memory reuse, compiler optimization, and the cost of recording intermediate states. Reverse mode can be fast for scalar outputs but may require substantial storage, while forward mode can become expensive when many input directions are needed.
Efficient implementations often combine caching, specialization, and graph optimization to reduce overhead.
9 Advantages and limitations
Automatic differentiation offers substantial benefits, but it is not universally optimal. Its strengths and weaknesses depend on the structure of the program and the derivative task at hand.
9.1 Accuracy and stability
A major advantage of autodiff is that it avoids the truncation errors associated with finite differences. The resulting derivatives are typically accurate to machine precision, aside from standard floating-point effects.
This makes the method attractive for delicate numerical computations where small derivative errors could affect convergence or stability.
9.2 Computational cost
The cost of autodiff is usually a small multiple of the cost of evaluating the original program, but the exact factor depends on the mode and implementation. Reverse mode can be especially efficient for scalar outputs, while forward mode may be preferable for limited numbers of input directions.
For very large problems, derivative evaluation may still be expensive enough to require careful algorithmic design.
9.3 Memory usage
Reverse-mode methods often need to retain intermediate values from the forward pass so they can be used during the backward sweep. This can create significant memory pressure for long or deeply nested computations.
Various techniques, such as checkpointing, reduce storage demands by recomputing some intermediates instead of keeping all of them.
9.4 Code compatibility and restrictions
Not all code is equally compatible with autodiff. Operations that are discontinuous, non-differentiable, or defined only approximately can complicate derivative computation. Similarly, some forms of dynamic control flow or mutation may require special handling.
Practical systems therefore impose rules on supported operations or provide custom derivative definitions for problematic cases.
10 Related topics
Automatic differentiation is connected to several broader areas in mathematics and computing. These related methods and concepts help place autodiff in context.
10.1 Symbolic computation
Symbolic computation manipulates mathematical expressions directly using algebraic rules. It is related to autodiff because both deal with derivatives, but symbolic methods operate on formulas rather than executed programs.
10.2 Finite difference methods
Finite difference methods approximate derivatives by evaluating a function at nearby points. They are simpler than autodiff but generally less accurate and more sensitive to numerical issues.
10.3 Backpropagation in neural networks
Backpropagation is the standard reverse-mode method used to train neural networks. It is one of the most familiar practical applications of automatic differentiation.
10.4 Differentiable programming
Differentiable programming is a programming paradigm in which entire programs are designed to be differentiable or easily equipped with derivatives. Automatic differentiation is one of its enabling technologies.