GRU编辑历史

## 1 Introduction

### 1.1 Background of RNNs and vanishing gradients

Recurrent neural networks (RNNs) are a class of artificial neural networks designed for sequential data. They maintain a hidden state that is updated at each time step based on the current input and the previous hidden state, enabling them to capture temporal dependencies. In practice, standard RNNs suffer from the vanishing gradient problem during training via backpropagation through time. Gradients become exponentially small over long sequences, making it difficult for the network to learn long-range dependencies.

### 1.2 Motivation for gated architectures

To overcome vanishing gradients, gated architectures were introduced. These networks incorporate mechanisms that control the flow of information, allowing them to retain relevant past information over many time steps. The most well-known example is the Long Short-Term Memory (LSTM) unit, which uses three gates (input, forget, output). The Gated Recurrent Unit (GRU) was proposed as a simpler alternative, using only two gates while achieving similar performance on many tasks.

## 2 Architecture and Mechanism

### 2.1 Update gate

The update gate \( z_t \) determines how much of the past hidden state should be carried forward to the current hidden state. It is computed from the current input \( x_t \) and the previous hidden state \( h_{t-1} \) via a sigmoid activation:

\[ z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z) \]

A value close to 1 means the previous state is largely retained, while a value close to 0 means it is mostly overwritten.

### 2.2 Reset gate

The reset gate \( r_t \) controls how much of the past hidden state is ignored when computing the candidate hidden state. It is similarly computed:

\[ r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r) \]

When \( r_t \) is near 0, the network effectively resets its memory, forgetting the previous state.

### 2.3 Candidate hidden state and final hidden state

The candidate hidden state \( \tilde{