Overview The exponential linear unit (ELU) is an activation function commonly used in deep learning and artificial neural networks. Proposed by Djork‑Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter in 2015, ELU is defined as \( f(x) = x \) for \( x \ge 0 \) and \( f(x) = \alpha (e^x - 1) \) for \( x < 0 \), where \( \alpha \) is a positive hyperparameter. The function aims to combine the linear regime of ReLU with a smooth, non‑zero negative part to reduce the vanishing gradient problem while improving noise robustness and learning dynamics. ELU has been widely adopted in architectures such as convolutional and recurrent neural networks.
1 Definition
ELU is a piecewise activation function that returns the identity for non‑negative inputs and an exponentially shaped negative value for negative inputs. Its design preserves the advantages of rectified linear units (ReLU) while addressing the problem of dying neurons.
1.1 Mathematical formulation
\[ \text{ELU}(x) = \begin{cases} x & \text{if } x \ge 0,\\ \alpha (e^x - 1) & \text{if } x < 0, \end{cases} \]
where \(\alpha > 0\) is a constant.
1.1.1 Role of the hyperparameter α
The parameter \(\alpha\) controls the saturation point of the negative part. For large negative \(x\), \(\text{ELU}(x) \to -\alpha\). A common default value is \(\alpha = 1\). Tuning \(\alpha\) can adjust the scale of negative outputs, influencing gradient flow and the mean activation of neurons.
1.1.2 Derivative
The derivative of ELU is:
\[ \text{ELU}'(x) = \begin{cases} 1 & \text{if } x \ge 0,\\ \alpha e^x = \text{ELU}(x) + \alpha & \text{if } x < 0. \end{cases} \]
The derivative is continuous for all \(x\) (including at \(x = 0\)), which facilitates gradient‑based optimization.
1.2 Properties
1.2.1 Continuity and differentiability
ELU is continuous and differentiable everywhere. At \(x = 0\), the left‑ and right‑hand derivatives are equal (both equal to 1), ensuring smooth gradient flow.
1.2.2 Boundedness and saturation on the negative side
For negative inputs, ELU saturates to a constant negative value \(-\alpha\). This boundedness on the negative side helps reduce the variance of activations and provides noise robustness, as large negative signals are capped.
2 Advantages and disadvantages
2.1 Advantages
2.1.1 Mitigation of vanishing gradient
Unlike the sigmoid function, ELU does not saturate for positive \(x\). The derivative is 1 for \(x \ge 0\), preventing gradient vanishing in the positive region. For negative inputs, the derivative is positive and non‑zero, allowing gradients to propagate through inactive neurons.
2.1.2 Shifted mean activation toward zero
The negative outputs of ELU push the mean activation of a layer closer to zero. This zero‑centered property accelerates convergence and reduces the bias shift effect, similar to batch normalization in some contexts.
2.1.3 Robustness to noise
The saturation on the negative side makes ELU less sensitive to small negative perturbations than ReLU (which outputs zero). This can improve generalization in noisy environments.
2.2 Disadvantages
2.2.1 Increased computational cost (exponential operation)
Computing \(e^x\) for negative inputs is more expensive than the simple linear or threshold operations of ReLU. This adds overhead, especially in large networks or on resource‑constrained hardware.
2.2.2 Sensitivity to α tuning
The performance of ELU can depend on the choice of \(\alpha\). An inappropriate value may saturate the negative part too early or too late, affecting gradient flow. Manual tuning or learnable variants are sometimes required.
3 Variants and related functions
3.1 Scaled exponential linear unit (SELU)
SELU is a scaled version of ELU introduced by Klambauer et al. (2017). It is defined as:
\[ \text{SELU}(x) = \lambda \begin{cases} x & \text{if } x \ge 0,\\ \alpha (e^x - 1) & \text{if } x < 0, \end{cases} \]
with specific values \(\lambda \approx 1.0507\) and \(\alpha \approx 1.6733\).
3.1.1 Self‑normalizing property
SELU is designed to automatically normalize activations toward zero mean and unit variance during forward propagation. This property, called self‑normalization, reduces the need for batch normalization in deep networks.
3.2 Parametric rectified linear unit (PReLU)
PReLU introduces a learnable parameter for the negative slope:
\[ \text{PReLU}(x) = \begin{cases} x & \text{if } x \ge 0,\\ a x & \text{if } x < 0, \end{cases} \]
where \(a\) is learned during training. Unlike ELU, PReLU does not saturate and uses a linear negative part.
3.3 Leaky ReLU and exponential ReLU
Leaky ReLU is a variant with a fixed small negative slope (e.g., 0.01). Exponential ReLU (also known as ELU) is sometimes confused with other exponential variants, but the term typically refers to the function described in this article.
4 Applications
4.1 Convolutional neural networks (CNNs)
ELU has been applied in CNNs for image classification (e.g., on CIFAR‑10 and ImageNet) where it outperforms ReLU on some architectures by reducing overfitting and improving convergence.
4.2 Recurrent neural networks (RNNs)
In RNNs, ELU helps mitigate vanishing gradients over long sequences. Its smooth derivative and saturation on the negative side contribute to more stable training compared to vanilla tanh or sigmoid.
4.3 Autoencoders and generative models
ELU is used in autoencoders, variational autoencoders (VAEs), and generative adversarial networks (GANs). Its zero‑centered property benefits tasks that require balanced activation distributions.
5 Comparison with other activation functions
5.1 ReLU and its variants
ReLU (\(f(x)=\max(0,x)\)) is computationally cheaper but suffers from dying neurons. ELU avoids this by providing non‑zero gradients for negative inputs. Leaky ReLU and PReLU are cheaper alternatives but lack saturation, which may reduce noise robustness.
5.2 Sigmoid and tanh
Sigmoid and tanh are monotonic, smooth, and saturating. Sigmoid saturates at 0 and 1; tanh at –1 and 1. Both suffer from vanishing gradients in deep networks. ELU’s linear positive part eliminates saturation for positive inputs, making it more suitable for deep architectures.
5.3 Swish and Mish
Swish (\(f(x)=x \cdot \sigma(x)\)) and Mish (\(f(x)=x \cdot \tanh(\ln(1+e^x))\)) are non‑monotonic functions that often outperform ELU in accuracy but are more computationally expensive. ELU remains a simple, interpretable alternative with a solid theoretical foundation.
6 History
6.1 Motivation and introduction (2015)
ELU was proposed in the paper “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)” by Clevert, Unterthiner, and Hochreiter, published at ICLR 2016 (archived in 2015). The authors identified that ReLU’s hard zero on the negative side could block gradient flow and increase bias shift. ELU’s exponential negative part was designed to address these issues while preserving computational efficiency.
6.2 Subsequent influence on activation function design
ELU inspired the development of SELU (2017) and contributed to the broader interest in activation functions with negative saturation. It demonstrated that non‑linearities smoother than ReLU could improve learning dynamics, paving the way for functions like GELU and Mish.
7 References
- Clevert, D.‑A., Unterthiner, T., & Hochreiter, S. (2015). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). *arXiv:1511.07289*.
- Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017). Self‑Normalizing Neural Networks. *Advances in Neural Information Processing Systems*.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). *Deep Learning*. MIT Press.