Role in neural computation
In artificial neural networks, each neuron computes a weighted sum of its inputs plus a bias, then passes the result through an activation function. This function determines the neuron’s output—often called its activation—and thereby controls the signal transmitted to subsequent layers. By applying a nonlinear transformation, activation functions enable networks to represent complex, hierarchical features.
Need for non‑linearity
Without nonlinear activation functions, a neural network with multiple layers would collapse into a single linear transformation: any composition of linear operations is itself linear. Consequently, such a network could only learn linear decision boundaries, severely limiting its capacity to model real‑world data distributions. Non‑linearity breaks this limitation, allowing networks to approximate arbitrary functions, including those with intricate patterns such as image edges, speech phonemes, or semantic relationships.
Binary step functions
The binary step function outputs either 0 or 1 based on whether the neuron’s input exceeds a threshold. Historically used in perceptrons, it is non‑differentiable (except at the threshold) and offers no graded response, making it unsuitable for gradient‑based learning. It is rarely employed in modern deep learning except in purely theoretical contexts.
Linear activation functions
A linear activation function simply returns its input unchanged (identity) or scaled by a constant. While differentiable, stacking multiple linear layers remains equivalent to a single linear layer, negating the benefit of depth. Linear activations are occasionally used in the output layer for regression tasks but are generally avoided in hidden layers.
Non‑linear activation functions
Sigmoid (logistic) function
The sigmoid function, defined as σ(x) = 1 / (1 + e⁻ˣ), maps any real input to the interval (0, 1). It is smooth, differentiable, and monotonic. Historically popular in shallow networks, it suffers from saturation: for large positive or negative inputs, the gradient approaches zero, impeding learning in deep architectures (vanishing gradient problem). It remains common in binary classification output layers.
Hyperbolic tangent (tanh)
tanh(x) = (eˣ – e⁻ˣ) / (eˣ + e⁻ˣ) outputs values in (–1, 1) and is zero‑centered, which often improves convergence compared to sigmoid. Like sigmoid, it saturates at extremes, leading to vanishing gradients. It is still used in recurrent networks and some older architectures.
Rectified Linear Unit (ReLU)
ReLU(x) = max(0, x) is the most widely used activation in deep learning since the early 2010s. It is computationally cheap, does not saturate for positive inputs, and introduces sparsity by zeroing negative values. However, it can cause “dead neurons” where large negative biases permanently prevent activation. ReLU’s success stems from its ability to mitigate vanishing gradients while preserving non‑linearity.
Variants: Leaky ReLU, Parametric ReLU, ELU
Leaky ReLU allows a small, fixed negative slope (e.g., 0.01) for negative inputs, reducing the risk of dead neurons. Parametric ReLU (PReLU) learns the negative slope during training, offering adaptive flexibility. Exponential Linear Unit (ELU) uses a smooth negative exponential curve, providing nonzero gradients for negative inputs and improving noise robustness. Each variant addresses ReLU’s limitations while retaining computational efficiency.
Exponential linear unit (ELU)
ELU(x) = x for x ≥ 0, and α(eˣ – 1) for x < 0 (with α > 0). It shares ReLU’s linear behavior for positives but has a smooth, saturating negative part. This reduces bias shift and can speed up learning in some architectures. ELU is less common than ReLU due to slightly higher computation cost.
Swish and mish
Swish (also called SiLU) is defined as x · σ(x), where σ is the sigmoid function. It is smooth, non‑monotonic, and has been shown to outperform ReLU on some deep networks, particularly in image classification tasks. Mish is a similar activation: x · tanh(softplus(x)). Both functions exhibit a small “dip” for negative values, which can improve gradient flow. They are gaining adoption in modern architectures such as EfficientNet and Transformer‑based models.
Probabilistic activation functions
Softmax
Softmax converts a vector of real numbers into a probability distribution over K classes: softmax(z_i) = e^(z_i) / Σ_j e^(z_j). It is standard for the output layer of multi‑class classification networks, ensuring outputs sum to 1 and are non‑negative. Softmax is differentiable and closely related to the logistic function.
Softplus
Softplus(x) = ln(1 + eˣ) is a smooth approximation of ReLU, outputting only positive values. It is used as an activation in certain probabilistic models (e.g., variational autoencoders) and occasionally in hidden layers where differentiability everywhere is required. Its gradient does not vanish for large positive inputs, but it is computationally more expensive than ReLU.
Differentiability
Most modern activation functions are differentiable almost everywhere, enabling gradient‑based optimisation via backpropagation. Binary step functions are non‑differentiable at the threshold; ReLU is non‑differentiable at zero but has a subgradient (0 or 1) commonly used in practice. Smooth functions like sigmoid, tanh, Swish, and ELU are fully differentiable.
Monotonicity
Many activation functions are monotonic (non‑decreasing or strictly increasing), such as sigmoid, tanh, ReLU, and ELU. Monotonicity ensures that the function’s output preserves the order of inputs, which aids optimisation and prevents oscillations in error surfaces. Non‑monotonic activations like Swish and mish sometimes yield better performance by allowing negative gradients to escape plateaus.
Output range and symmetry
The output range influences gradient magnitudes and network stability. Sigmoid outputs (0, 1), tanh (–1, 1), ReLU [0, ∞), and ELU (–α, ∞). Zero‑centered activations (tanh) typically lead to faster convergence than strictly positive ones (sigmoid, ReLU), as they prevent biased updates. Symmetry (e.g., tanh’s odd symmetry) can also impact learning dynamics.
Vanishing and exploding gradients
Vanishing gradients occur when activation derivatives are near zero for most inputs (e.g., sigmoid’s tails), causing deep layers to learn very slowly. Exploding gradients happen when derivatives are large (rare in standard activations) or when weights cause exponential growth. ReLU and its variants mitigate vanishing gradients in deep networks by maintaining a constant derivative of 1 for positive inputs, while techniques like gradient clipping address exploding gradients.
Impact on training speed
Activation functions that avoid saturation (ReLU, leaky ReLU, Swish) generally accelerate training because gradients propagate efficiently. Saturated activations (sigmoid, tanh) can slow convergence, especially in deep networks. Computational cost also matters: ReLU is simplest; ELU and Swish involve exponentials, slightly increasing per‑iteration time. In practice, ReLU and its variants are preferred for hidden layers in feedforward and convolutional networks.
Dead neurons and dying ReLU
The dying ReLU problem occurs when a large negative bias causes a neuron’s output to always be zero, and its gradient is also zero, so it never recovers. This phenomenon reduces network capacity. Variants like leaky ReLU, PReLU, and ELU eliminate this by allowing small negative gradients. Proper weight initialisation (e.g., He initialisation) and lower learning rates also help prevent dead neurons.
Compatibility with output layer
Binary classification (sigmoid)
For two‑class problems, a single sigmoid neuron outputs a probability between 0 and 1, interpreted as the likelihood of the positive class. It pairs naturally with binary cross‑entropy loss.
Multi‑class classification (softmax)
For K > 2 classes, the softmax function produces a probability vector over all classes. It is typically used with categorical cross‑entropy loss.
Regression (linear or identity)
For regression tasks predicting unbounded real values, the output layer commonly uses a linear (identity) activation so that the network output can be any real number. If the target is bounded, a sigmoid or tanh may be applied to constrain outputs.
Adaptive activation functions
Some modern designs use activation functions that adapt their shape during training. For example, the Adaptive Piecewise Linear (APL) function learns breakpoints; others adjust slope or curvature based on data statistics. These methods can increase model flexibility but risk overfitting without proper regularisation.
Parametric and learnable activations
In Parametric ReLU (PReLU) and Swish with trainable β, the activation function itself contains learnable parameters (e.g., slope, scaling). These parameters are updated via backpropagation along with network weights, allowing the activation to adapt to the task. Such learnable activations often yield small performance gains in vision and language models.
Activation functions in recurrent networks
Recurrent neural networks (RNNs) face severe vanishing gradient problems due to repeated multiplication of gradients over time steps. The tanh and sigmoid activations were standard but worsened vanishing gradients. The Long Short‑Term Memory (LSTM) and Gated Recurrent Unit (GRU) use gating mechanisms with sigmoid and tanh to control information flow. More recently, ReLU‑based RNNs with careful initialisation have shown stability, but tanh remains prevalent in many RNN implementations.
Activation functions in transformers (GELU)
Transformer architectures, popular in natural language processing, often use the Gaussian Error Linear Unit (GELU). GELU(x) = x · Φ(x), where Φ is the standard Gaussian CDF. It is a smooth, non‑monotonic function that outperforms ReLU on tasks like machine translation and language modelling (e.g., BERT, GPT). GELU’s shape resembles Swish (with shift) and is computationally approximated (e.g., tanh approximation) for efficiency. Its use in transformers highlights the importance of activation function choice for very deep, attention‑based networks.