Overview
ResNet (Residual Network) is a deep convolutional neural network architecture introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun in their 2015 paper "Deep Residual Learning for Image Recognition." It addresses the degradation problem—where increasing network depth leads to higher training error—by employing skip connections (also called residual connections) that allow gradients to flow directly through the network. ResNet won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2015 and has since become a foundational building block in modern computer vision, enabling the training of networks with hundreds or even thousands of layers.
1 Background
1.1 Vanishing and Exploding Gradients
In very deep networks, gradients computed during backpropagation can become extremely small (vanishing) or extremely large (exploding). Vanishing gradients prevent early layers from learning, while exploding gradients cause unstable updates. Traditional solutions included careful weight initialization and gradient clipping, but these methods became insufficient as networks grew deeper.
1.2 The Degradation Problem
Empirical observations showed that beyond a certain depth, adding more layers to a plain network increased training error, even after accounting for overfitting. This degradation problem is distinct from vanishing gradients—it indicates that deep plain networks are inherently harder to optimize, and the added layers fail to learn useful identity mappings.
1.3 Prior Deep Network Efforts
Before ResNet, progress on deep networks included Highway Networks (2015), which used gating mechanisms to control information flow, and Network in Network (2013), which introduced 1×1 convolutions. However, these approaches did not achieve the same level of success as residual learning in scaling to hundreds of layers.
2 Architecture
2.1 Residual Block
A residual block consists of two or more stacked convolutional layers, with a shortcut connection that bypasses the block. The block learns a residual mapping \(\mathcal{F}(x) = \mathcal{H}(x) - x\) instead of the original mapping \(\mathcal{H}(x)\). The output is \(\mathcal{F}(x) + x\), which is easier to optimize when \(\mathcal{F}(x)\) is close to zero.
2.1.1 Skip Connection
The skip connection (or shortcut connection) directly adds the input \(x\) to the output of the block’s convolutional layers. This allows gradients to flow backward through the network unimpeded, mitigating the degradation problem. In the simplest case, \(x\) and \(\mathcal{F}(x)\) must have the same dimensions.
2.1.2 Identity Shortcut vs. Projection Shortcut
When the dimensions of \(x\) and \(\mathcal{F}(x)\) differ (e.g., due to stride or channel changes), two options exist:
- Identity shortcut: Pad the shortcut with zeros or use a 1×1 convolution to match dimensions. The original ResNet paper recommended using a 1×1 convolution with stride 2 for downsampling.
- Projection shortcut: Use a 1×1 convolution to project \(x\) to the required dimensions. This adds extra parameters and is typically used only when dimension mismatch occurs.
2.2 Bottleneck Building Block
For deeper ResNet variants (ResNet-50 and above), a bottleneck block is used to reduce computational cost. It consists of three layers: a 1×1 convolution for reducing channels, a 3×3 convolution, and another 1×1 convolution for restoring channels. The shortcut skips these three layers.
2.2.1 1×1 Convolutions for Dimensionality Reduction
The first 1×1 convolution in the bottleneck block reduces the number of input channels (e.g., from 256 to 64), allowing the 3×3 convolution to operate on a much smaller volume, thus saving parameters and computation.
2.2.2 Computational Efficiency
By using the bottleneck design, ResNet-50 has a comparable computational cost (about 3.8 billion FLOPs) to a shallower plain network but achieves significantly better accuracy. This design enables training of very deep networks on practical hardware.
2.3 Full Network Variants
ResNet comes in several depths, each following a stack of residual blocks with increasing feature map channels (64, 128, 256, 512). The variants differ in the number of blocks per stage.
2.3.1 ResNet-18
Consists of 18 layers using basic residual blocks (two 3×3 convolutions per block). It has 11.7 million parameters and is suitable for resource-constrained settings.
2.3.2 ResNet-34
Extends ResNet-18 with more blocks, totaling 34 layers and 21.8 million parameters. It provides a good balance of accuracy and efficiency.
2.3.3 ResNet-50
Introduces bottleneck blocks and has 50 layers with 25.6 million parameters. It became the most widely used variant for transfer learning.
2.3.4 ResNet-101
A deeper version with 101 layers (44.5 million parameters), offering higher accuracy on large-scale tasks.
2.3.5 ResNet-152
The deepest original variant at 152 layers (60.2 million parameters), achieving state-of-the-art on ImageNet in 2015.
3 Training Methodology
3.1 Batch Normalization
Batch normalization is applied after every convolution and before activation. It normalizes activations to have zero mean and unit variance, accelerating training and allowing higher learning rates. ResNet uses batch normalization consistently in all variants.
3.2 Activation Functions (ReLU)
The Rectified Linear Unit (ReLU) is used as the activation function throughout the network. It introduces non-linearity while mitigating the vanishing gradient problem compared to sigmoid or tanh.
3.3 Weight Initialization
He initialization (also called Kaiming initialization) was introduced in the same paper. It sets initial weights based on the number of input connections to maintain variance across layers, enabling stable training of very deep networks.
3.4 Regularization (Dropout and Data Augmentation)
ResNet relies primarily on data augmentation (random cropping, horizontal flipping, color jitter) rather than dropout. Dropout is used only in the final fully connected layer for some experiments. Weight decay (L2 regularization) is applied to all layers.
4 Applications
4.1 Image Classification (ImageNet)
ResNet’s primary success was on ImageNet classification, where the ResNet-152 model achieved a top-5 error rate of 3.57% in ILSVRC 2015, surpassing human-level performance for the first time on that dataset.
4.2 Object Detection (Faster R-CNN, YOLO)
ResNet serves as a backbone feature extractor in modern object detectors. Faster R-CNN with a ResNet-101 backbone improved detection accuracy, and YOLOv3 adopted a variant of ResNet for feature extraction.
4.3 Semantic Segmentation (DeepLab, PSPNet)
For pixel-level prediction, ResNet backbones are used with atrous convolutions (Dilated ResNet) to maintain spatial resolution. DeepLabv3 and PSPNet both rely on ResNet-101 or ResNet-50 as the base architecture.
4.4 Transfer Learning and Feature Extraction
Pre-trained ResNet models on ImageNet are widely used for transfer learning in tasks with limited data. A common practice is to remove the final classification layer and use the penultimate layer’s output as a feature vector.
5 Extensions and Variants
5.1 Wide ResNet (WRN)
Wide ResNet increases the number of feature map channels (width) rather than depth. It uses a widening factor \(k\) (e.g., 2, 10) on basic blocks.
5.1.1 Increased Width over Depth
Wide ResNet showed that wider but shallower networks can outperform deeper ones with fewer layers, achieving better accuracy and training speed at the cost of more parameters.
5.2 ResNeXt
ResNeXt introduces a new dimension called “cardinality”—the number of parallel, independent paths within a block.
5.2.1 Grouped Convolutions and Cardinality
ResNeXt uses grouped convolutions to split the input into multiple groups, each processed independently, then concatenated. Increasing cardinality improves accuracy without adding many parameters.
5.3 Pre-activation ResNet
Pre-activation ResNet moves batch normalization and ReLU before the convolutional layers in a residual block.
5.3.1 Batch Normalization and ReLU Order
In the original ResNet, batch norm and ReLU follow the convolution (post-activation). Pre-activation places them before the convolution, which improves gradient flow and eases training of very deep networks (e.g., ResNet-1001).
6 Legacy and Influence
6.1 ILSVRC 2015 Victory
ResNet achieved first place in all five tasks of ILSVRC 2015: classification, detection, localization, segmentation, and video. Its success demonstrated the power of residual learning and ignited widespread adoption.
6.2 Impact on Modern Architectures (DenseNet, SENet, EfficientNet)
ResNet’s skip connections inspired subsequent architectures. DenseNet concatenates all previous feature maps; SENet introduces channel-wise attention; EfficientNet uses neural architecture search but still employs residual blocks as a base. The core idea of learning residuals persists.
6.3 Adoption in Industry and Open-Source Libraries (TensorFlow, PyTorch)
ResNet is included as a built-in model in TensorFlow (Keras Applications), PyTorch (torchvision), and other deep learning frameworks. It is used in production systems for image search, autonomous driving, medical imaging, and more.
7 Common Misconceptions and Fun Facts
7.1 "ResNet is Just a Very Long Network"
A common joke in deep learning circles: “ResNet is so deep that its skip connections are actually the network’s main path.” In reality, the identity shortcut is not the primary path; the residual mapping still learns features, and the network’s effective depth matters for performance.
7.2 Skip Connection as a "Gradient Highway"
The term “gradient highway” is a playful metaphor used in memes to describe how gradients bypass layers via skip connections. While not technically accurate (gradients still pass through convolutional layers), the idea captures the essence of ResNet’s relief from vanishing gradients.
7.3 ResNet in Lightweight Settings (ResNet for Mobile)
Some amateur forum posts claim “ResNet-18 is too heavy for mobile,” but lightweight variants like MobileNet and ShuffleNet were designed specifically for mobile. Nevertheless, ResNet-18 is sometimes used as a baseline on smartphone GPUs, and the phrase “ResNet for Mobile” is a humorous misnomer—RealNet (Real-time Network) would be more appropriate.