1 Background and Definition
An activation function is a mathematical function applied to the output of a neuron in an artificial neural network. Its primary purpose is to introduce non-linearity into the model, enabling the network to learn complex patterns and representations beyond simple linear transformations. Activation functions determine whether a neuron should be activated ("fired") based on the weighted sum of its inputs and a bias term. They are a fundamental component of deep learning architectures, influencing training dynamics, convergence speed, and overall model performance.
1.1 Role in Neural Networks
In an artificial neuron, the weighted sum of inputs plus a bias is passed through an activation function to produce the neuron's output. Without activation functions, a neural network of multiple layers would collapse into a single linear transformation, severely limiting its expressive power. By introducing non-linearity, activation functions allow networks to approximate arbitrary continuous functions (the universal approximation theorem). They also control the magnitude and range of neuron outputs, which affects gradient flow during backpropagation.
1.2 Historical Development
Early neural networks, such as the perceptron (1958), used a step function that output 0 or 1. The limitations of step functions (non-differentiability) led to the adoption of sigmoid and tanh functions in the 1980s and 1990s, which are smooth and differentiable. However, these saturating functions caused vanishing gradient problems in deep networks. The Rectified Linear Unit (ReLU), popularized around 2010 by Hinton and others, addressed many of these issues and became a default choice. Subsequent research has produced numerous variants and modern functions like Swish and GELU, aimed at improving training dynamics and performance.
2 Common Activation Functions
2.1 Sigmoid
The sigmoid function maps any real-valued input to a value between 0 and 1. It was historically used in binary classification output layers and in hidden layers of shallow networks.
2.1.1 Mathematical Expression
The sigmoid function is defined as: \[ \sigma(x) = \frac{1}{1 + e^{-x}} \] Its derivative is \(\sigma(x)(1 - \sigma(x))\), which is simple to compute.
2.1.2 Advantages and Limitations
Advantages: Smooth and differentiable; outputs can be interpreted as probabilities; bounded output prevents uncontrolled activation growth.
Limitations: Saturation for large positive or negative inputs causes gradients to become very small, leading to vanishing gradients in deep networks. Outputs are not zero-centered, which can slow convergence. Exponential computation is more expensive than linear operations.
2.2 Tanh (Hyperbolic Tangent)
The tanh function is a scaled and shifted version of the sigmoid, outputting values between -1 and 1.
2.2.1 Mathematical Expression
\[ \tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} = 2\sigma(2x) - 1 \] Its derivative is \(1 - \tanh^2(x)\).
2.2.2 Advantages and Limitations
Advantages: Zero-centered output (mean near zero) helps alleviate the bias shift effect, often leading to faster convergence than sigmoid. Still smooth and differentiable.
Limitations: Still suffers from saturation and vanishing gradients for very large or small inputs. The exponential computation cost remains.
2.3 Rectified Linear Unit (ReLU)
ReLU is defined as \(f(x) = \max(0, x)\). It outputs zero for negative inputs and the identity for positive inputs. Its simplicity and effectiveness made it the default activation for hidden layers in many deep architectures.
Advantages: Computationally cheap (simple thresholding); mitigates vanishing gradient for positive inputs; induces sparsity (many neurons output zero). Empirically accelerates convergence in many tasks.
Limitations: "Dying ReLU" problem—neurons can become permanently inactive when their weights push activations to always be negative, producing zero gradients and preventing recovery.
2.3.1 Variants of ReLU
Several variants have been proposed to address the dying ReLU problem by allowing small gradients for negative inputs.
2.3.1.1 Leaky ReLU
Leaky ReLU replaces the zero part with a small positive slope \(\alpha\), typically 0.01: \[ f(x) = \max(\alpha x, x) \] This ensures a small gradient for negative inputs, keeping neurons potentially recoverable.
2.3.1.2 Parametric ReLU (PReLU)
PReLU generalizes Leaky ReLU by making the negative slope \(\alpha\) a learnable parameter during training. This allows the network to adapt the slope for each neuron or layer, offering more flexibility.
2.3.1.3 Exponential Linear Unit (ELU)
ELU uses an exponential for negative inputs to push outputs closer to zero mean: \[ f(x) = \begin{cases} x & x \geq 0 \\ \alpha(e^x - 1) & x < 0 \end{cases} \] The negative saturation dampens noise while the exponential ensures smooth differentiability. ELU can improve learning speed but involves extra computation.
2.4 Softmax
Softmax is a multi-class generalization of the sigmoid, mapping a vector of real numbers to a probability distribution over classes.
\[ \text{softmax}(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}} \]
It ensures outputs are positive and sum to 1, making it suitable for the output layer of classification networks.
2.4.1 Use in Multi-class Classification
In multi-class classification, the softmax function outputs probabilities for each class. Combined with cross-entropy loss, it provides a well-behaved gradient for training. The function is typically applied to the logits (pre-activation outputs) of the final layer.
2.5 Other Modern Functions
2.5.1 Swish
Swish, introduced by Google researchers in 2017, is defined as: \[ f(x) = x \cdot \sigma(x) \] where \(\sigma(x)\) is the sigmoid. Swish is smooth, non-monotonic, and has been shown to outperform ReLU on deep networks for certain tasks. Its unbounded upper and lower saturation regions help with gradient flow.
2.5.2 Gaussian Error Linear Unit (GELU)
GELU is a smooth approximation of the ReLU-like behavior, defined as: \[ \text{GELU}(x) = x \cdot \Phi(x) \] where \(\Phi(x)\) is the cumulative distribution function of the standard normal distribution. GELU is commonly used in transformer models (e.g., BERT, GPT) and provides smooth transitions around zero. It can be approximated by \(0.5x(1 + \tanh(\sqrt{2/\pi}(x + 0.044715x^3)))\) for efficiency.
3 Mathematical Properties
3.1 Continuity and Differentiability
Most modern activation functions are continuous and differentiable almost everywhere, a requirement for gradient-based optimization via backpropagation. Step functions (used in early perceptrons) are discontinuous and not differentiable, making them unsuitable for deep learning. Functions like ReLU are continuous but not differentiable at zero; however, subgradients or custom gradient definitions (e.g., setting derivative at zero to 0 or 0.5) suffice for training.
3.2 Boundedness and Saturation
Bounded functions (sigmoid, tanh) limit output ranges, which can be useful in output layers but cause saturation—regions where gradients are near zero. Unbounded functions (ReLU, Swish) avoid saturation for large positive inputs but may produce very large activations, potentially causing exploding gradients if not managed with normalization or careful weight initialization.
3.3 Monotonicity
Monotonic activation functions (sigmoid, tanh, ReLU) preserve the sign of the input gradient, simplifying optimization. Non-monotonic functions (Swish, GELU) have a small dip for negative values, which can provide beneficial regularization and help avoid poor local minima. Monotonicity is not strictly required, but most common activations are monotonic for practical reasons.
4 Practical Considerations
4.1 Vanishing and Exploding Gradients
Vanishing gradients occur when activation functions saturate (sigmoid, tanh) or produce very small gradients (e.g., negative region of ReLU), making deep networks difficult to train. Exploding gradients happen when gradients become very large, often due to unbounded activations or poor initialization. Both problems are exacerbated in very deep networks.
4.1.1 Mitigation Strategies
Common strategies include: using non-saturating activations (ReLU, Swish, GELU); proper weight initialization (Xavier/Glorot, He); batch normalization or layer normalization; gradient clipping; skip connections (ResNets); and learning rate scheduling.
4.2 Computational Efficiency
Activation functions vary in computational cost. ReLU is the cheapest (max operation), while sigmoid, tanh, Swish, and GELU require exponentials or more complex operations. On modern hardware with parallel computation, the overhead is often negligible except for large-scale models; however, embedded or mobile deployments may favor simpler functions.
4.3 Choosing an Activation Function
The choice of activation function depends on network architecture, task, and training dynamics. No single function universally outperforms others; empirical tuning is common.
4.3.1 Guidelines for Different Network Types
- Feedforward and convolutional networks: ReLU (or its variants) is a good default. Leaky ReLU, PReLU, or ELU may help if dying ReLU is problematic.
- Recurrent networks (RNNs, LSTMs): tanh or sigmoid are historically common due to bounded outputs; however, ReLU with careful initialization or gradient clipping can work.
- Transformer models: GELU is widely used; Swish is also effective. ReLU and its variants appear in some older architectures.
- Output layers: Sigmoid for binary classification; softmax for multi-class; linear (no activation) for regression.
4.3.2 Impact on Convergence
Activation functions affect convergence speed and stability. Saturated functions (sigmoid, tanh) often converge more slowly in deep networks. ReLU accelerates convergence due to its non-saturating behavior but can cause dead neurons. Modern functions like Swish and GELU provide a trade-off: they are non-saturating for large positive inputs yet have a small negative region that helps regularize. In practice, experimenting with a few candidates (ReLU, Leaky ReLU, Swish, GELU) and monitoring validation performance is recommended.