1 Fundamental Concepts
1.1 Sequential data and temporal dependencies
Sequential data consists of ordered elements—such as words in a sentence, stock prices over time, or frames in a video—where the order carries meaning. Temporal dependencies refer to relationships between elements at different positions; for example, the meaning of a word often depends on preceding words. Recurrent neural networks are designed to exploit such dependencies by processing inputs one step at a time while retaining information from earlier steps.
1.2 Recurrent connections and hidden states
A recurrent connection is a directed cycle in the network’s graph: the output of a neuron at time step \(t\) feeds back into the same neuron (or others) at time step \(t+1\). This creates an internal hidden state \(\mathbf{h}_t\), a vector that summarizes relevant information about the sequence up to time \(t\). The hidden state is updated at each step using the current input and the previous hidden state:
\[ \mathbf{h}_t = f(\mathbf{W}_{xh}\mathbf{x}_t + \mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{b}_h) \]
where \(f\) is a nonlinear activation function.
1.3 Unfolding computational graphs
To apply standard backpropagation, the recurrent structure is “unfolded” (or “unrolled”) into a feedforward graph with one copy of the network per time step, sharing the same weights across all copies. This unfolded representation makes the temporal dependencies explicit and allows gradient computation over a fixed-length segment of the sequence.
2 Architecture
2.1 Vanilla RNN
The simplest form of RNN—often called the “vanilla” RNN—consists of a single hidden layer with recurrent connections. At each time step, it computes the hidden state and optionally an output \(\mathbf{y}_t\):
\[ \mathbf{h}_t = \tanh(\mathbf{W}_{xh}\mathbf{x}_t + \mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{b}_h) \] \[ \mathbf{y}_t = \mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y \]
2.1.1 Activation functions (tanh, ReLU)
The hyperbolic tangent (tanh) is the traditional activation, squashing values to \((-1, 1)\) and providing stable gradients for moderately sized inputs. The rectified linear unit (ReLU) can also be used but may cause unbounded activations and is less common in vanilla RNNs due to instability in recurrent dynamics.
2.2 Deep RNNs
Deep RNNs extend the depth of the network, either by stacking multiple recurrent layers or by adding depth in the transition from input to hidden (or hidden to output).
2.2.1 Stacked recurrent layers
In a stacked RNN, each layer’s hidden state feeds into the next layer at the same time step. The output of layer \(l\) becomes the input to layer \(l+1\):
\[ \mathbf{h}^{(l)}_t = f(\mathbf{W}^{(l)}_{xh}\mathbf{h}^{(l-1)}_t + \mathbf{W}^{(l)}_{hh}\mathbf{h}^{(l)}_{t-1} + \mathbf{b}^{(l)}) \]
Stacking increases representational capacity but also raises computational cost and the risk of overfitting.
2.2.2 Residual connections in RNNs
Residual (skip) connections add the layer’s input directly to its output, e.g., \(\mathbf{h}^{(l)}_t = \mathbf{h}^{(l-1)}_t + f(\cdots)\). This helps mitigate gradient degradation in very deep RNNs, enabling training with more than a few layers.
2.3 Bidirectional RNNs
A bidirectional RNN (BiRNN) processes the sequence in both forward and backward directions simultaneously, yielding a hidden state that captures context from past and future time steps.
2.3.1 Forward and backward passes
The forward pass runs a standard RNN from left to right, producing hidden states \(\overrightarrow{\mathbf{h}}_t\). The backward pass runs a separate RNN from right to left, producing \(\overleftarrow{\mathbf{h}}_t\). The two states are concatenated (or summed) at each time step to form the final representation.
2.4 Encoder–Decoder Architectures
Encoder–decoder RNNs comprise two RNNs: an encoder that reads the entire input sequence and produces a fixed-length context vector (often the final hidden state), and a decoder that generates an output sequence step by step conditioned on that context. This architecture is the foundation of sequence-to-sequence learning, particularly in machine translation.
3 Training
3.1 Backpropagation Through Time (BPTT)
Backpropagation through time applies the chain rule to the unfolded computation graph. The gradients of the loss with respect to all network weights are computed by accumulating errors across time steps. For a sequence of length \(T\), BPTT requires storing the hidden states for all \(T\) steps, leading to high memory usage.
3.2 Truncated BPTT
To reduce memory and computational cost, truncated BPTT processes the sequence in fixed-length chunks. Gradients are propagated only within each chunk (e.g., \(k=10\) steps), and the hidden state is carried over as initial state for the next chunk. This approximation works well if the truncation length captures the most relevant dependencies.
3.3 Vanishing and Exploding Gradients
During BPTT, the gradient of the loss with respect to earlier time steps involves repeated multiplication by the recurrent weight matrix \(\mathbf{W}_{hh}\). If its largest eigenvalue (spectral radius) is less than 1, gradients shrink exponentially (vanishing); if greater than 1, they grow exponentially (exploding). Vanishing gradients make it hard for RNNs to learn long-range dependencies; exploding gradients can cause numerical overflow and training instability.
3.3.1 Gradient clipping
A common remedy for exploding gradients is gradient clipping: if the total gradient norm exceeds a threshold (e.g., 5.0), it is scaled down proportionally. This prevents sudden weight updates without altering the direction of the gradient.
3.4 Regularization techniques
3.4.1 Dropout in RNNs
Standard dropout—randomly dropping neurons during training—can be applied to RNNs, but naive application to recurrent connections can disrupt the sequential dynamics. A common practice is to apply dropout only to the non‑recurrent connections (e.g., input-to-hidden, hidden-to-output). Variants such as variational dropout use the same dropout mask across all time steps, preserving noise injection while respecting temporal recurrence.
3.4.2 Zoneout
Zoneout is a stochastic regularization technique specific to RNNs: at each time step, a neuron’s hidden state is either updated (as usual) or randomly kept unchanged (i.e., “zoned out”). This acts like a temporal dropout, preventing co‑adaptation and encouraging the network to preserve information over longer timescales.
4 Variants
4.1 Long Short-Term Memory (LSTM)
LSTM networks introduce a cell state \(\mathbf{c}_t\) that flows through the sequence with minimal linear transformations, allowing gradients to propagate more easily. Gating mechanisms control the flow of information into and out of the cell state.
4.1.1 Cell state and gates
The cell state is updated by three gates: forget, input, and output. Each gate is a sigmoid layer that outputs values in \((0, 1)\) representing the degree of allowed flow.
4.1.1.1 Forget gate
The forget gate decides what information to discard from the previous cell state. It takes \(\mathbf{h}_{t-1}\) and \(\mathbf{x}_t\) and outputs a vector \(\mathbf{f}_t\):
\[ \mathbf{f}_t = \sigma(\mathbf{W}_{xf}\mathbf{x}_t + \mathbf{W}_{hf}\mathbf{h}_{t-1} + \mathbf{b}_f) \]
4.1.1.2 Input gate
The input gate decides which new information to store. It produces a vector \(\mathbf{i}_t\) and, together with a candidate cell state computed via tanh, updates the cell state:
\[ \mathbf{i}_t = \sigma(\mathbf{W}_{xi}\mathbf{x}_t + \mathbf{W}_{hi}\mathbf{h}_{t-1} + \mathbf{b}_i) \] \[ \tilde{\mathbf{c}}_t = \tanh(\mathbf{W}_{xc}\mathbf{x}_t + \mathbf{W}_{hc}\mathbf{h}_{t-1} + \mathbf{b}_c) \] \[ \mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t \]
4.1.1.3 Output gate
The output gate controls which parts of the cell state are exposed as the hidden state. It produces \(\mathbf{o}_t\), and the final hidden state is:
\[ \mathbf{o}_t = \sigma(\mathbf{W}_{xo}\mathbf{x}_t + \mathbf{W}_{ho}\mathbf{h}_{t-1} + \mathbf{b}_o) \] \[ \mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t) \]
4.2 Gated Recurrent Unit (GRU)
GRU simplifies LSTM by merging the cell state and hidden state into a single vector and using only two gates: update and reset.
4.2.1 Update gate
The update gate \(\mathbf{z}_t\) decides how much of the previous hidden state to retain versus replace with new candidate information:
\[ \mathbf{z}_t = \sigma(\mathbf{W}_{xz}\mathbf{x}_t + \mathbf{W}_{hz}\mathbf{h}_{t-1} + \mathbf{b}_z) \]
4.2.2 Reset gate
The reset gate \(\mathbf{r}_t\) controls how much of the past hidden state is forgotten when computing the candidate hidden state:
\[ \mathbf{r}_t = \sigma(\mathbf{W}_{xr}\mathbf{x}_t + \mathbf{W}_{hr}\mathbf{h}_{t-1} + \mathbf{b}_r) \] \[ \tilde{\mathbf{h}}_t = \tanh(\mathbf{W}_{xh}\mathbf{x}_t + \mathbf{W}_{hh}(\mathbf{r}_t \odot \mathbf{h}_{t-1}) + \mathbf{b}_h) \] \[ \mathbf{h}_t = (1 - \mathbf{z}_t) \odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t \]
GRU has fewer parameters than LSTM and often matches its performance on many tasks.
4.3 Clockwork RNN
Clockwork RNN (CW-RNN) partitions the hidden layer into modules that operate at different clock rates. Each module is updated only at intervals determined by its assigned period (e.g., every 2, 4, or 8 time steps). This reduces computational load and encourages learning of multi‑scale temporal patterns.
4.4 Peephole LSTM
Peephole LSTM extends the standard LSTM by allowing the gates to also “peep” at the cell state (rather than only the hidden state). For example, the forget gate receives the cell state as an additional input:
\[ \mathbf{f}_t = \sigma(\mathbf{W}_{xf}\mathbf{x}_t + \mathbf{W}_{hf}\mathbf{h}_{t-1} + \mathbf{W}_{cf}\mathbf{c}_{t-1} + \mathbf{b}_f) \]
Peephole connections can help the gates make more accurate timing decisions, though their practical benefit is sometimes modest.
5 Applications
5.1 Natural Language Processing
5.1.1 Language modeling
Language models predict the next word in a sequence given previous words. RNNs process a sentence word by word and output a probability distribution over the vocabulary. They have been widely used in word‑level and character‑level language modeling, though modern transformers have largely supplanted them for large‑scale tasks.
5.1.2 Machine translation
RNN‑based encoder–decoder models (often with attention) translate a source sentence into a target sentence. The encoder reads the source, and the decoder generates the translation step by step. LSTM and GRU variants are especially popular for handling variable‑length sequences.
5.1.3 Sentiment analysis
Sentiment analysis classifies text (e.g., movie reviews, social media posts) as positive, negative, or neutral. A bidirectional RNN processes the entire text, and the final hidden states are fed into a classifier.
5.2 Time Series Forecasting
5.2.1 Financial data
RNNs can model stock prices, exchange rates, or other financial indicators. Their ability to capture short‑term and long‑term dependencies helps forecast price movements, though the non‑stationary and noisy nature of financial data often limits accuracy.
5.2.2 Weather prediction
Meteorological time series (temperature, humidity, wind speed) are modeled with RNNs to produce short‑term forecasts. Multi‑step forecasting is common, with the network trained to predict several future time points from a sliding window of past observations.
5.3 Speech Recognition
5.3.1 Acoustic modeling
In automatic speech recognition (ASR), RNNs (particularly bidirectional LSTM) process sequences of acoustic features (e.g., Mel‑frequency cepstral coefficients) and output probabilities for phonetic units or characters. Connectionist temporal classification (CTC) is often used as the training criterion.
5.3.2 Language modeling integration
RNN language models are integrated with acoustic models to improve word‑level recognition. The language model scores candidate transcriptions, combining acoustic and linguistic probabilities.
5.4 Handwriting Recognition and Generation
RNNs (often combined with convolutional neural networks) recognize handwritten text from pen‑stroke sequences or images. For generation, an RNN predicts the next pen position, pressure, and other parameters conditioned on a target text, producing realistic handwriting.
6 Limitations and Extensions
6.1 Difficulty modeling long-range dependencies
Despite LSTM and GRU mitigating the vanishing gradient problem, RNNs still struggle with sequences spanning hundreds or thousands of steps. The gating mechanisms help but do not fully eliminate the difficulty, especially when the relevant context is widely separated.
6.2 Computational inefficiency of sequential processing
RNNs process one time step at a time, making them inherently sequential. This prevents parallelization across the time dimension, leading to slower training and inference compared to models that process entire sequences in parallel (e.g., transformers, convolutional networks).
6.3 Relationship with attention mechanisms
Attention mechanisms, originally introduced for machine translation, allow the decoder to directly access encoder hidden states at all time steps. This global context alleviates the need to compress the entire source into a single vector. Attention has been combined with RNNs to improve performance, and later replaced them entirely in transformer architectures.
6.4 Modern alternatives: Transformers and state-space models
Transformers replace recurrence with self‑attention, enabling full parallelization and better handling of long dependencies through positional encodings. State‑space models (e.g., S4, Mamba) use linear time‑invariant dynamics to achieve efficient long‑range modeling while retaining a recurrent structure during inference. These alternatives have become the dominant approaches in many sequence‑processing applications, though RNNs remain useful for low‑resource, small‑scale, or real‑time tasks.