In applied mathematics, a sigmoid function is a mathematical function having a characteristic "S"-shaped curve (sigmoid curve). Typically, a sigmoid function is real-valued, differentiable, and has a non-negative or non-positive first derivative, a single inflection point, and asymptotic behavior at both ends (e.g., approaching 0 and 1, or -1 and 1). The logistic function is the most common example, widely used in logistic regression, neural networks (as activation functions), and population growth models. Sigmoid functions map any real input to a bounded output, often between 0 and 1, making them useful for probability estimation and smooth thresholding.
1 Definition and Properties
1.1 Formal Definition
A sigmoid function is defined as any continuous, real-valued function \( f : \mathbb{R} \to (a, b) \) (where \( a < b \) are finite real numbers) that is differentiable, strictly increasing (or strictly decreasing), has exactly one inflection point, and satisfies \(\lim_{x \to -\infty} f(x) = a\) and \(\lim_{x \to +\infty} f(x) = b\). The simplest and most widely used sigmoid is the logistic function, but many other functions share these properties.
1.2 Key Properties
1.2.1 Monotonicity and Smoothness
Sigmoid functions are monotonic (either increasing or decreasing on their entire domain). They are also infinitely differentiable (smooth), which is a desirable property for optimization algorithms that rely on gradients.
1.2.2 Asymptotic Behavior
The function approaches finite horizontal asymptotes at both extremes. For a sigmoid mapping to \((0,1)\), \( \lim_{x \to -\infty} f(x) = 0 \) and \( \lim_{x \to +\infty} f(x) = 1 \). For a symmetric sigmoid such as \(\tanh\), the asymptotes are \(-1\) and \(1\).
1.2.3 Derivative and Inflection Point
The first derivative of a sigmoid is non-negative (for increasing functions) and unimodal—it rises to a maximum at the inflection point and then decays. The inflection point occurs where the second derivative changes sign; for many standard sigmoids this is at \(x=0\) (given appropriate centering).
1.3 Common Sigmoid Functions
1.3.1 Logistic Function
1.3.1.1 Standard Form (0 to 1)
The logistic function is defined as \( \sigma(x) = \frac{1}{1 + e^{-x}} \). It maps the real line to \((0,1)\), with inflection at \(x=0\) where \( \sigma(0)=0.5\). Its derivative satisfies \( \sigma'(x) = \sigma(x)(1-\sigma(x)) \).
1.3.1.2 Generalized Form
A generalized logistic function is \( f(x) = \frac{L}{1 + e^{-k(x - x_0)}} \), where \(L\) is the maximum value, \(k\) controls the steepness of the curve, and \(x_0\) is the midpoint. This form is common in population modeling and biological growth.
1.3.2 Hyperbolic Tangent (tanh)
The hyperbolic tangent \(\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}\) maps \(\mathbb{R}\) to \((-1,1)\). It is a rescaled and shifted logistic function: \(\tanh(x) = 2\sigma(2x)-1\). It is symmetric about the origin and often used in neural networks because its output is zero-centered.
1.3.3 Arctangent, Error Function (erf), and Others
Other sigmoid functions include the arctangent \(\arctan(x)\) (mapping to \((-\pi/2, \pi/2)\)) and the error function \(\operatorname{erf}(x) = \frac{2}{\sqrt{\pi}}\int_0^x e^{-t^2} dt\) (mapping to \((-1,1)\)). These are less commonly used in machine learning but appear in probability and statistics.
2 Applications in Applied Mathematics
2.1 Logistic Regression and Classification
2.1.1 Binary Classification
In logistic regression, the logistic function models the probability that a binary outcome belongs to a particular class. Given a linear combination of features \(z = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p\), the predicted probability is \(p = 1/(1+e^{-z})\). The model is trained by maximizing the log-likelihood.
2.1.2 Multinomial Logistic (Softmax) as Generalization
For multi-class classification, the softmax function—a generalization of the logistic function—is used. It maps a vector of real scores to a probability distribution over K classes: \( \text{softmax}(z)_i = e^{z_i} / \sum_{j=1}^K e^{z_j} \). Each output is between 0 and 1, and they sum to 1.
2.2 Neural Networks and Machine Learning
2.2.1 Activation Functions in Feedforward Networks
Sigmoid functions, particularly logistic and tanh, were historically used as activation functions in artificial neurons. They introduce nonlinearity and bound the output, mimicking biological neuron firing rates.
2.2.2 Vanishing Gradient Problem
In deep networks, sigmoid activation functions cause gradients to become very small in early layers during backpropagation, because the derivative of a sigmoid is at most 0.25. This "vanishing gradient" hinders training of deep networks, leading to the adoption of alternatives like ReLU.
2.2.3 Alternatives (ReLU, Swish, etc.)
Rectified Linear Unit (ReLU) and its variants (Leaky ReLU, ELU) mitigate the vanishing gradient by having a derivative of 1 for positive inputs. Swish, \(x \cdot \sigma(x)\), is a smooth non-monotonic function that sometimes outperforms ReLU. These functions are not sigmoids but have displaced sigmoids in many modern architectures.
2.3 Population Modeling and Ecology
2.3.1 Logistic Growth Model
The logistic differential equation \( \frac{dP}{dt} = rP\left(1 - \frac{P}{K}\right) \) describes population growth with a carrying capacity \(K\). Its solution is a sigmoid (logistic) curve: \( P(t) = \frac{K}{1 + (K/P_0 - 1)e^{-rt}} \).
2.3.2 Carrying Capacity and Saturation
The sigmoid shape reflects initial exponential growth, followed by saturation as resources become limited. This model is widely used in ecology, epidemiology, and resource economics.
2.4 Control Theory and Signal Processing
2.4.1 Smooth Step Functions
Sigmoid functions serve as smooth approximations to step functions. In control systems, they are used as soft switches or saturation functions to avoid discontinuities in feedback loops.
2.4.2 Sigmoid as a Smooth Approximation to the Heaviside Step
The logistic function with large steepness \(k\) approximates the Heaviside step function. This is useful in numerical simulations where smooth derivatives are required, e.g., in level-set methods or when using gradient-based optimization over discontinuous thresholds.
3 Variants and Extensions
3.1 Bounded Rational Functions
The function \( f(x) = \frac{x}{\sqrt{1+x^2}} \) is a sigmoid with outputs in \((-1,1)\). It is a rational approximation (up to square root) and is sometimes used for its computational simplicity.
3.2 Gompertz Function – Asymmetric Sigmoid
The Gompertz function \( f(x) = a e^{-b e^{-cx}} \) produces an asymmetric S-shaped curve. It is commonly used in mortality modeling, tumor growth, and technology diffusion where growth is not symmetric about the inflection point.
3.3 Hyperbolic Sigmoid and Scaled Versions
A "hyperbolic sigmoid" can refer to tanh or any sigmoid obtained by scaling and shifting the logistic function. For example, \( \sigma_a(x) = \frac{1}{1+e^{-ax}} \) where \(a\) controls slope.
3.4 Parametric Families (e.g., Generalized Logistic)
Richards' curve (generalized logistic) adds an additional parameter \(\nu\) to control asymmetry: \( f(x) = L / (1 + \nu e^{-k(x-x_0)})^{1/\nu} \). The Gompertz function is a limiting case. This family allows fitting a wide variety of sigmoid shapes.
4 Computational Considerations
4.1 Numerical Stability (Overflow/Underflow)
Direct computation of \(e^{-x}\) for large negative \(x\) can cause overflow (or underflow for large positive \(x\) in tanh). For logistic, if \(x\) is very negative, compute \( \sigma(x) = e^x / (1 + e^x) \) to avoid overflow. Many libraries implement a branch: \( \sigma(x) = 1/(1+e^{-x}) \) for \(x \geq 0\), else \( \sigma(x) = e^x/(1+e^x) \).
4.2 Efficient Approximations (Lookup Tables, Piecewise)
For performance-critical applications (e.g., hardware implementation), sigmoids are approximated using piecewise linear or polynomial functions, or small lookup tables. Common approximations include using a linear region around zero and saturation elsewhere.
4.3 Derivative Computation (for Backpropagation)
In neural networks, the derivative of the logistic function is computed as \( \sigma(x)(1-\sigma(x)) \). Since \(\sigma(x)\) is already known from the forward pass, this is cheap. For tanh, the derivative is \(1 - \tanh^2(x)\). These expressions are numerically stable when the activation value is bounded.
5 Historical Context and Related Concepts
5.1 Development in Statistics (Verhulst, Pearl)
The logistic function was introduced by Pierre François Verhulst in 1838–1847 to model population growth with limited resources. He named it the "logistic curve". In the early 20th century, Raymond Pearl and others revived the model for population biology, and later it became fundamental in statistics through logistic regression (developed in the 1940s–50s by Joseph Berkson and others).
5.2 Connection to Cumulative Distribution Functions (CDFs)
The logistic function is the CDF of the logistic distribution. Similarly, the error function is the CDF of the normal distribution (scaled). Many sigmoids arise as CDFs of continuous probability distributions, making them natural choices for modeling probabilities.
5.3 Sigmoid in Physics (Fermi–Dirac Distribution)
In statistical mechanics, the Fermi–Dirac distribution \( f(E) = 1/(1 + e^{(E-\mu)/(kT)}) \) is a logistic function. It describes the probability that a fermion occupies a given energy state and has a sigmoid shape as a function of energy. This connection is often noted in the context of activation functions in artificial neurons.
6 References and Further Reading
- Verhulst, P. F. (1838). "Notice sur la loi que la population poursuit dans son accroissement". *Correspondance Mathématique et Physique*, 10: 113–121.
- Berkson, J. (1944). "Application of the Logistic Function to Bio-Assay". *Journal of the American Statistical Association*, 39(227): 357–365.
- Han, J., & Moraga, C. (1995). "The influence of the sigmoid function parameters on the speed of backpropagation learning". *From Natural to Artificial Neural Computation*, 195–201.
- Cybenko, G. (1989). "Approximation by superpositions of a sigmoidal function". *Mathematics of Control, Signals, and Systems*, 2(4): 303–314.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). *Deep Learning*. MIT Press. (Chapter 6: Deep Feedforward Networks).