1 Definition and Basic Identity

1.1 The core log-sum-exp transformation

The log-sum-exp function (LSE) maps a collection of real numbers \(\{x_i\}\) to \[ \operatorname{LSE}(x)=\log\left(\sum_i e^{x_i}\right). \] It is a fundamental transformation in numerical analysis because it rewrites “log of a sum of exponentials” in a form that can be computed stably even when the \(x_i\) are large in magnitude.

1.2 Stabilization via the max-shift trick

A direct evaluation of \(\sum_i e^{x_i}\) may overflow when some \(x_i\) are large, and may underflow when they are very negative. The max-shift trick addresses both issues by factoring out the largest exponential term.

1.2.1 Derivation of the numerically stable form

Let \(m=\max_i x_i\). Then \[ \sum_i e^{x_i}=\sum_i e^{m+(x_i-m)}=e^{m}\sum_i e^{x_i-m}. \] Taking logs gives \[ \operatorname{LSE}(x)=\log\left(e^{m}\sum_i e^{x_i-m}\right)=m+\log\left(\sum_i e^{x_i-m}\right). \] Because every \(x_i-m\le 0\), the terms \(e^{x_i-m}\) lie in \((0,1]\), which substantially reduces overflow risk.

1.3 Connection to sums in log space

Many probabilistic quantities are naturally expressed in log space, where products become sums and sums require careful handling. LSE provides the standard way to combine terms that are already logged: \[ \log\left(\sum_i a_i\right)\quad \text{with } a_i>0 \] can be computed by setting \(x_i=\log a_i\) and evaluating \(\log\left(\sum_i e^{x_i}\right)\). This avoids converting back to raw scale prematurely.

2 Numerical Stability and Implementation

2.1 Overflow/underflow concerns with exponentials

Exponentials amplify magnitude: if \(x_i\) exceeds the logarithm of the largest representable floating-point number, \(e^{x_i}\) overflows to \(\infty\). Conversely, if \(x_i\) is very negative, \(e^{x_i}\) can underflow toward \(0\), and the sum may lose contributions from small but non-negligible terms. Both effects distort the final logarithm.

2.2 Stable computation strategies

A stable LSE implementation typically uses the max-shift form and computes the exponential of non-positive offsets.

2.2.1 Choosing a reference shift (e.g., \(\max_i x_i\))

Using \(m=\max_i x_i\) minimizes the largest offset \(x_i-m\) to \(0\), ensuring the largest exponential term equals \(1\). Other reference shifts are possible (for example, a pre-chosen constant or a quantile-based approximation), but the max is the most common because it is cheap and provably safe.

2.3 Handling edge cases (empty sums, infinities, NaNs)

Real implementations must specify behavior for unusual inputs:

  • Empty sums: \(\sum_i e^{x_i}\) is undefined for an empty index set; many libraries return \(-\infty\) or raise an error depending on convention.
  • Infinities: if some \(x_i=+\infty\), then \(\sum_i e^{x_i}=+\infty\) and LSE is \(+\infty\). If all \(x_i=-\infty\), then the sum equals \(0\) and \(\log 0=-\infty\).
  • NaNs: any NaN in \(x_i\) can make the result NaN, though some implementations may offer options for NaN-handling behavior.

3 Mathematical Properties

3.1 Convexity and smoothness

LSE is convex as a function of \(\{x_i\}\). This follows from the fact that log-sum-exp is the log of a sum of exponentials, and exponentials are convex. The function is also smooth (infinitely differentiable) on \(\mathbb{R}^n\), which makes it suitable as a differentiable approximation to non-smooth operations such as \(\max\).

3.2 Relationship to the softmax function

The softmax function normalizes exponentials: \[ \operatorname{softmax}(x)_i=\frac{e^{x_i}}{\sum_j e^{x_j}}. \] LSE and softmax are tightly coupled: LSE appears as the log-partition function, and softmax emerges naturally when differentiating LSE.

3.3 Log-sum-exp as a soft maximum

LSE behaves like a “softened” version of \(\max_i x_i\), with the smoothness controlled by how concentrated the exponentials become.

3.3.1 Bounds between LSE and \(\max\)

Let \(m=\max_i x_i\). Then \[ m \le \operatorname{LSE}(x)\le m+\log n, \] where \(n\) is the number of terms. The lower bound holds because \(\sum_i e^{x_i}\ge e^{m}\). The upper bound follows from \(\sum_i e^{x_i}\le n e^{m}\).

3.3.2 Limiting behavior under scaling

Introduce a scale parameter \( \alpha>0 \) and consider \[ \operatorname{LSE}(\alpha x)=\log\left(\sum_i e^{\alpha x_i}\right). \] As \(\alpha\to\infty\), \[ \frac{1}{\alpha}\operatorname{LSE}(\alpha x)\to \max_i x_i, \] because the largest exponential dominates the sum. This provides a rigorous basis for using LSE as a smooth approximation to \(\max\).

4 Differentiation and Gradients

4.1 Gradient of log-sum-exp with respect to inputs

For \(x\in\mathbb{R}^n\), \[ \frac{\partial}{\partial x_i}\operatorname{LSE}(x)=\frac{e^{x_i}}{\sum_j e^{x_j}}. \] Thus, the gradient components form a probability vector.

The gradient equals \(\operatorname{softmax}(x)_i\). This interpretation is widely used in optimization: maximizing LSE encourages larger \(x_i\), while the gradient distributes credit among entries proportionally to their exponentiated magnitude.

4.2 Jacobian and higher-order derivatives

The Hessian of LSE has a characteristic form: \[ \frac{\partial^2}{\partial x_i\partial x_k}\operatorname{LSE}(x)=p_i(\delta_{ik}-p_k), \] where \(p_i=\operatorname{softmax}(x)_i\) and \(\delta_{ik}\) is the Kronecker delta. This matrix is positive semidefinite, consistent with convexity. Higher derivatives can be expressed in terms of softmax moments, though they are less frequently needed in standard machine learning pipelines.

4.3 Applications in automatic differentiation contexts

Because LSE is smooth and composed of elementary differentiable operations (sum, exponentials, log), modern autodiff systems can compute its gradients accurately and efficiently. In practice, many frameworks also implement a dedicated stable kernel for LSE so that gradient computation inherits numerical stability.

5.1 Log of mean exponential (log-mean-exp)

A common variant replaces the sum with an average: \[ \operatorname{logmeanexp}(x)=\log\left(\frac{1}{n}\sum_i e^{x_i}\right)=\operatorname{LSE}(x)-\log n. \] This shift can be convenient when comparing across different numbers of terms.

5.2 Weighted log-sum-exp

Given nonnegative weights \(w_i\), one may define \[ \log\left(\sum_i w_i e^{x_i}\right). \] Weighted LSE appears in mixture models, attention mechanisms, and any setting where contributions are not uniform. For numerical stability, the same max-shift idea applies by factoring out \(\max_i x_i\) (or a weighted reference if convenient).

5.3 Temperature-scaled log-sum-exp

Temperature scaling introduces a factor \(\beta\) (often written as \(1/T\)): \[ \operatorname{LSE}_\beta(x)=\log\left(\sum_i e^{\beta x_i}\right). \] As \(\beta\) increases, the function approaches a hard max; as \(\beta\) decreases, it becomes more uniform, with behavior closer to an average in log space.

5.4 Pairwise and incremental forms

Instead of computing LSE over all terms at once, one can combine subsets: \[ \operatorname{LSE}(x_1,\dots,x_n)=\operatorname{LSE}\big(\operatorname{LSE}(x_{1..k}), x_{k+1..n}\big), \] with appropriate grouping. This supports streaming or hierarchical computation, where partial aggregates are updated as new terms arrive.

6 Applications in Machine Learning and Statistics

6.1 Stable softmax and log-softmax computations

Softmax and log-softmax are central in neural classification. The naive computation can be unstable for large logits. LSE provides a stable normalization term: \[ \log\left(\operatorname{softmax}(x)_i\right)=x_i-\operatorname{LSE}(x), \] so log-softmax can be computed using the same stable LSE routine.

6.2 Log-likelihood aggregation

In statistical models, the likelihood of independent events often multiplies probabilities, turning into sums of log-probabilities. When marginalization requires summing probabilities over latent states, it becomes a log-sum-exp computation: \[ \log\left(\sum_z p(z)\,p(y\mid z)\right). \] LSE ensures these marginal log-likelihoods can be evaluated without numerical distortion.

6.3 Loss functions using log-sum-exp

Many common losses can be written in terms of LSE, benefiting from its stability and smoothness.

6.3.1 Examples in classification and ranking

  • Multiclass cross-entropy: typically involves log-softmax, which is \(x-\operatorname{LSE}(x)\).
  • Ranking and contrastive objectives: often aggregate positive and negative scores using log-sum-exp to form smooth approximations to max-margin behavior.

7 Computational Complexity and Practical Considerations

7.1 Complexity of naive vs stabilized computation

Both naive and stabilized LSE require exponentiating and summing across \(n\) terms, yielding \(O(n)\) time. The stabilized form adds a pass (or reduction) to find the maximum, which is also \(O(n)\). Overall, the asymptotic complexity is unchanged, while numerical reliability improves.

7.2 Vectorized and batched evaluation

In practice, LSE is computed over tensors along specified axes. Efficient implementations exploit parallelism (SIMD/GPU kernels) and reuse reductions (e.g., computing the max and the shifted exponentials). Batched LSE is especially common in deep learning, where computations occur for many samples and many candidate classes simultaneously.

7.3 Precision considerations (float32/float64)

Floating-point precision affects both overflow thresholds and rounding error:

  • float32 has a narrower exponent range, making stability tricks more critical.
  • float64 is more forgiving but can still benefit from max-shift for accuracy, especially when values are extreme.

Additionally, summation order can influence rounding; some libraries use techniques such as compensated summation or optimized reduction trees to reduce error.

8 Inequalities, Bounds, and Approximations

8.1 Common upper/lower bounds

The basic max-based bounds \[ m \le \operatorname{LSE}(x)\le m+\log n \] are frequently used to reason about magnitude. More refined bounds can be derived when additional structure is known (e.g., known gaps between top elements), providing tighter control on the discrepancy between LSE and the max.

8.2 Approximating max with log-sum-exp

Because LSE is smooth, it is often used in optimization where \(\max\) is non-differentiable. The approximation quality improves when the inputs are scaled so that the largest term dominates. Analysts commonly tune a temperature or scaling parameter to balance approximation fidelity against gradient smoothness.

8.3 Error behavior and tuning for accuracy

The difference between LSE and \(\max\) depends on how concentrated the exponentials are. If the largest element is much greater than the rest, LSE is close to the max. If multiple entries are near the top, the gap increases, bounded by \(\log n\). In applications, tuning the temperature or scaling factor adjusts this trade-off, affecting both numerical behavior and the sharpness of gradients.