The rectified linear unit (ReLU) is an activation function commonly used in artificial neural networks, defined as \( f(x) = \max(0, x) \). Introduced in the early 2000s and popularized in deep learning around 2010, ReLU helps mitigate the vanishing gradient problem compared to sigmoid or tanh functions, enabling faster training of deep networks. Its piecewise linear nature introduces non-linearity while maintaining computational simplicity.

1 Definition

1.1 Mathematical formulation

The ReLU function is defined as: \[ f(x) = \max(0, x) = \begin{cases} x & \text{if } x \geq 0 \\ 0 & \text{if } x < 0 \end{cases} \] It is a piecewise linear function that outputs the input directly if it is positive; otherwise, it outputs zero.

1.2 Alternative representations

1.2.1 Piecewise linear form

ReLU can be expressed as a composition of linear functions: \[ f(x) = x \cdot \mathbf{1}(x \geq 0) \] where \(\mathbf{1}(\cdot)\) is the indicator function, or as: \[

f(x) = \frac{x +x}{2}

\] using the absolute value.

1.2.2 Subgradient

Because ReLU is not differentiable at \(x = 0\), the subgradient is used for optimization. The subgradient is any value in the interval \([0, 1]\) at \(x = 0\). In practice, the derivative is often taken as: \[ f'(x) = \begin{cases} 1 & \text{if } x > 0 \\ 0 & \text{if } x < 0 \end{cases} \] with \(f'(0)\) commonly defined as 0 or 1, depending on implementation.

2 Properties

2.1 Advantages

2.1.1 Sparsity induction

ReLU naturally induces sparsity in the hidden representations because negative inputs produce zero outputs. This sparsity can lead to more efficient and interpretable models, as only a subset of neurons are activated for any given input.

2.1.2 Computational efficiency

The ReLU function involves only a simple thresholding operation (comparison and assignment), making it computationally inexpensive compared to exponential-based functions like sigmoid or tanh. This efficiency speeds up both forward and backward propagation in deep networks.

2.2 Limitations

2.2.1 Dying ReLU problem

2.2.1.1 Causes

During training, a neuron can become stuck in a state where it outputs zero for all inputs. This occurs if the weighted sum of its inputs is always negative, causing the gradient to be zero. Once the neuron enters this state, it cannot recover because the gradient is zero for all training examples.

2.2.1.2 Mitigation strategies

To address the dying ReLU problem, several techniques are employed:

  • Using variants such as Leaky ReLU or Parametric ReLU that allow small negative gradients.
  • Employing careful weight initialization and adaptive learning rates.
  • Using batch normalization to keep activations centered.

3 Variants

3.1 Leaky ReLU

Leaky ReLU modifies the negative part with a small, fixed slope \(\alpha\) (commonly 0.01): \[ f(x) = \begin{cases} x & \text{if } x \geq 0 \\ \alpha x & \text{if } x < 0 \end{cases} \] This prevents zero gradients for negative inputs.

3.2 Parametric ReLU (PReLU)

PReLU generalizes Leaky ReLU by making the negative slope \(\alpha\) a learnable parameter during training. This allows the network to adapt the slope per neuron.

3.3 Exponential Linear Unit (ELU)

ELU smooths the transition at zero using an exponential curve: \[ f(x) = \begin{cases} x & \text{if } x \geq 0 \\ \alpha (e^x - 1) & \text{if } x < 0 \end{cases} \] It pushes mean activations closer to zero, improving learning speed.

3.4 Scaled Exponential Linear Unit (SELU)

SELU is a scaled version of ELU with predefined parameters \(\alpha\) and \(\lambda\) such that the activation function induces self-normalizing properties: \[ f(x) = \lambda \begin{cases} x & \text{if } x \geq 0 \\ \alpha (e^x - 1) & \text{if } x < 0 \end{cases} \] It ensures that activations automatically converge to zero mean and unit variance through layers.

3.5 Gaussian Error Linear Unit (GELU)

GELU weights inputs by their value, approximated by: \[ f(x) = x \cdot \Phi(x) \] where \(\Phi(x)\) is the cumulative distribution function of the standard normal distribution. It combines properties of ReLU, dropout, and zoneout.

4 Applications

4.1 Convolutional neural networks (CNNs)

ReLU is the most widely used activation in convolutional neural networks for image classification, object detection, and segmentation. Its non-saturating nature allows deep models like ResNet, VGG, and AlexNet to train effectively.

4.2 Recurrent neural networks (RNNs)

Although less common than in CNNs, ReLU has been used in recurrent architectures. However, due to the risk of exploding gradients, it is often paired with gradient clipping or replaced by variants like Leaky ReLU.

4.3 Generative adversarial networks (GANs)

Both generator and discriminator networks in GANs frequently employ ReLU or its variants. Leaky ReLU is particularly common in discriminators to avoid sparse gradients and improve training stability.

5 Training considerations

5.1 Weight initialization

Proper initialization is critical for ReLU-based networks. Biased initialization can cause dead neurons. He initialization (Gaussian variance scaled by \(2/n_{\text{in}}\)) is recommended to maintain variance of activations.

5.2 Learning rate tuning

High learning rates may cause many neurons to die due to large negative weight updates. ReLU networks often benefit from lower learning rates or adaptive methods like Adam.

5.3 Regularization techniques

Dropout and batch normalization are frequently used with ReLU. Batch normalization helps keep activations in the positive regime, reducing the dying ReLU problem. L2 regularization can also prevent weights from becoming too large.

6 History

6.1 Early biological inspiration

ReLU draws inspiration from the thresholding behavior of biological neurons, where a neuron fires only if the input exceeds a certain threshold. Early neural network models (e.g., the McCulloch-Pitts neuron) used step functions; ReLU can be seen as a smooth, continuous version.

6.2 Adoption in deep learning

Although ReLU was introduced in the early 2000s (e.g., by Hahnloser et al. in 2000, and by Jarrett et al. in 2009), it gained widespread adoption after Alex Krizhevsky’s 2012 ImageNet paper. The success of deep CNNs with ReLU over sigmoid/tanh demonstrated its practical advantages.

6.3 Influence from the neural tangent kernel

The neural tangent kernel (NTK) theory, developed in the late 2010s, provides a theoretical framework for understanding the training dynamics of infinitely wide ReLU networks. It shows that ReLU networks behave like Gaussian processes under certain scaling, deepening theoretical insights into their success.

7.1 Sigmoid and tanh

Sigmoid and tanh are S-shaped functions that saturate at extremes. Unlike ReLU, they suffer from the vanishing gradient problem, making deep network training difficult. They were predominant in shallow networks but have been largely superseded by ReLU in deep learning.

7.2 Softplus

Softplus is a smooth approximation of ReLU: \[ f(x) = \ln(1 + e^x) \] It is differentiable everywhere and avoids the hard zero region, but introduces additional computational cost without significant performance benefits.

7.3 Swish and Mish

Swish (defined as \( x \cdot \sigma(x) \), where \(\sigma\) is the sigmoid) and Mish (\( x \cdot \tanh(\ln(1+e^x)) \)) are newer activation functions that often outperform ReLU in certain deep architectures. They are smooth, non-monotonic, and have been shown to improve accuracy in image classification and other tasks.