Vision Transformers (ViT) represent a paradigm shift in computer vision by applying the Transformer architecture, originally designed for natural language processing, directly to image classification. Instead of relying on convolutional layers, ViT divides an image into fixed-size patches, linearly embeds each patch, and processes the resulting sequence of embeddings through a standard Transformer encoder with multi-head self-attention. The model replaces the convolution inductive bias with global attention, achieving competitive or superior performance on large-scale datasets. ViT has since inspired numerous variants and extensions, becoming a foundational architecture in modern visual recognition.
1 Introduction
1.1 Motivation and Background
The Transformer architecture, introduced by Vaswani et al. in 2017 for sequence transduction tasks, demonstrated remarkable success in natural language processing (NLP). Motivated by the hypothesis that a pure attention-based model could also excel in computer vision, researchers explored replacing convolutional operations with self-attention mechanisms. Early attempts combined attention with convolutions or used attention as a replacement for certain layers. The Vision Transformer (ViT) was the first to apply a standard Transformer encoder directly to image patches, eschewing convolutions entirely. This design was inspired by the success of Transformers in processing long-range dependencies and their ability to scale with data.
1.2 Relation to Convolutional Neural Networks (CNNs)
Convolutional neural networks (CNNs) have dominated computer vision for decades, leveraging inductive biases such as locality and translation equivariance through shared convolutional kernels. ViT significantly reduces these built-in biases, relying instead on the Transformer’s global self-attention to learn spatial relationships from data. On small to medium datasets, CNNs generally outperform ViT due to their stronger priors. However, when trained on very large datasets (e.g., ImageNet-21k, JFT-300M), ViT matches or exceeds state-of-the-art CNNs, demonstrating that the inductive bias of convolutions can be replaced by sufficient data and model capacity. ViT also offers advantages in modeling long-range dependencies and can be more efficient in terms of theoretical FLOPs for large patch sizes.
2 Architecture
2.1 Image Patch Embedding
2.1.1 Linear Projection of Patches
The input image is divided into non-overlapping patches of fixed size (e.g., 16×16 pixels). Each patch is flattened into a vector of pixel values and then linearly projected into an embedding space of dimension \(D\) via a trainable matrix. The resulting sequence of patch embeddings serves as the input tokens to the Transformer encoder. The number of tokens is \(N = HW / P^2\) for an image of height \(H\) and width \(W\) with patch size \(P\). This process effectively converts a spatial grid into a 1D sequence, discarding explicit 2D structure.
2.1.2 Positional Encoding (Learnable or Sinusoidal)
Because self-attention is permutation-invariant, positional information must be injected to retain spatial layout. ViT adds a positional embedding to each patch embedding before entering the encoder. Originally, ViT uses learnable 1D positional embeddings, where each position in the sequence (including a special classification token) has a distinct trainable vector. Alternatively, sinusoidal encodings (as in the original Transformer) or 2D-aware variants can be used, but learnable 1D embeddings have been found effective for most vision tasks.
2.2 Transformer Encoder
2.2.1 Multi-Head Self-Attention (MSA)
2.2.1.1 Scaled Dot-Product Attention
The core of self-attention is the scaled dot-product attention mechanism. Given a matrix \(Q\) (queries), \(K\) (keys), and \(V\) (values) derived from the same input sequence, the attention output is computed as \(\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\), where \(d_k\) is the dimension of the key vectors. Scaling by \(\sqrt{d_k}\) prevents the dot products from growing large, stabilizing gradients. In multi-head self-attention, the operation is performed in parallel over \(h\) heads, each with separate learned projections of \(Q\), \(K\), and \(V\), and the outputs are concatenated and linearly projected.
2.2.2 MLP Block and Layer Normalization
After each MSA layer, a feed-forward MLP block is applied, consisting of two linear layers with a GELU activation function in between. The MLP typically expands the dimension by a factor of 4. Layer normalization (LayerNorm) is applied before both the MSA and the MLP blocks (pre-normalization), as opposed to after (post-normalization). This arrangement improves training stability and gradient flow.
2.2.3 Residual Connections
Both the MSA and MLP blocks are wrapped with residual (skip) connections. The output of each sublayer is added to its input: \(x_{l+1} = x_l + \text{Sublayer}(\text{LayerNorm}(x_l))\). Residual connections enable deep Transformer encoders (e.g., 12 or 24 layers) to be trained effectively by alleviating vanishing gradients.
2.3 Classification Head (CLS Token and MLP)
ViT prepends a special learnable [CLS] token to the sequence of patch embeddings. This token passes through the Transformer encoder and its final hidden state is treated as an aggregate representation of the entire image. The [CLS] token is fed into a small MLP head (usually a single linear layer during pre-training) to produce class logits. This design follows the BERT approach in NLP, where the [CLS] token captures sentence-level information. Alternatively, one can use global average pooling over all patch tokens, but the [CLS] token is standard in ViT.
3 Training and Implementation
3.1 Pre-training Strategy
3.1.1 Large-Scale Dataset Requirement (e.g., ImageNet-21k, JFT-300M)
ViT models are typically pre-trained on extremely large labeled image datasets, such as ImageNet-21k (14 million images, 21k classes) or the proprietary JFT-300M (300 million images). The reliance on large-scale data stems from the lack of convolutional inductive biases; without sufficient data, ViT tends to overfit on smaller datasets. Pre-training on these large corpora allows the self-attention mechanism to learn meaningful visual features.
3.1.2 Optimizer and Hyperparameters
ViT is trained using Adam or AdamW optimizer with a learning rate schedule that includes linear warmup (e.g., for 10,000 steps) followed by cosine decay. Weight decay is applied (e.g., 0.1 for large models). Gradient clipping is often used to stabilize training. Batch sizes are large (e.g., 4096) and training runs for hundreds of thousands to millions of steps. Models are trained at resolution 224×224, with patch size 16×16, resulting in 196 patch tokens plus the [CLS] token.
3.2 Fine-tuning on Downstream Tasks
After pre-training, ViT is fine-tuned on smaller target datasets (e.g., ImageNet-1k, CIFAR-100). During fine-tuning, the classification head is replaced with a randomly initialized linear layer for the new number of classes. It is common to fine-tune at higher resolutions (e.g., 384×384) to improve performance. When changing input resolution, the positional embeddings are interpolated (e.g., using bilinear interpolation) to match the new grid size, because the pretrained embeddings correspond to a fixed sequence length. The learning rate for fine-tuning is typically reduced (e.g., 0.001), and a smaller batch size is used.
3.3 Data Augmentation and Regularization
To mitigate overfitting on smaller datasets, ViT training employs extensive data augmentation, including random cropping, horizontal flipping, color jitter, RandAugment, and mixup. Regularization techniques such as stochastic depth (randomly dropping layers during training), label smoothing, and dropout (on the patch embeddings and the MLP output) are standard. These methods are crucial for achieving competitive performance on ImageNet-1k when pre-training on the same dataset.
4 Variants and Improvements
4.1 Data-efficient Image Transformers (DeiT)
4.1.1 Knowledge Distillation with a Teacher CNN
DeiT (Data-efficient Image Transformers) was introduced to make ViT trainable on ImageNet-1k without requiring massive external datasets. The key innovation is a knowledge distillation strategy using a teacher CNN (e.g., RegNetY). A distillation token is added to the sequence, and the model is trained with a combination of hard and soft distillation losses. The student ViT learns from both the true labels and the teacher’s predictions, achieving competitive accuracy to CNNs while maintaining the Transformer architecture.
4.2 Swin Transformer
4.2.1 Shifted Window Multi-Head Self-Attention
Swin Transformer introduces a hierarchical architecture using shifted windows to improve efficiency and enable cross-window connections. Self-attention is computed within non-overlapping local windows (e.g., 7×7 patches). To allow information flow between windows, the window partitioning is shifted in successive layers (shifted window MSA). This design reduces computational complexity from quadratic to linear relative to image size while still capturing global context.
4.2.2 Hierarchical Feature Maps
Unlike ViT’s single-scale sequence, Swin Transformer produces feature maps at multiple resolutions through patch merging layers (similar to pooling in CNNs). This hierarchical structure makes Swin suitable for dense prediction tasks such as object detection and semantic segmentation, as it provides a multi-scale representation compatible with backbone networks like FPN.
4.3 Cross-Attention and Token Mixing Models
4.3.1 Perceiver and CvT
Perceiver decouples the input size from the Transformer capacity by using a cross-attention mechanism that maps a large input (e.g., pixels) to a smaller latent array. It is a general architecture that handles various modalities, including images, audio, and point clouds. Convolutional vision Transformer (CvT) incorporates convolutions into the Transformer design, using convolutional token embedding layers and convolutional projection for Q, K, V. This hybrid approach combines the local modeling of convolutions with the global interaction of attention, improving both performance and efficiency.
5 Applications
5.1 Image Classification
ViT’s primary application is image classification. On ImageNet-1k, a large ViT (ViT-H/14) pretrained on JFT-300M achieved 88.55% top-1 accuracy. Subsequent variants like DeiT, Swin, and CoAtNet have attained even higher scores, often surpassing CNN-based models. ViT-based models serve as backbones for many classification tasks, including fine-grained recognition and medical imaging.
5.2 Object Detection and Instance Segmentation
ViT’s global attention and hierarchical variants (Swin, CvT) have been successfully integrated into object detection frameworks such as Mask R-CNN, DETR, and RetinaNet. Swin Transformer serves as a strong backbone in the Swin-B detection model, achieving state-of-the-art results on COCO. ViT itself, when adapted via feature pyramid networks, also performs well in detection and instance segmentation.
5.3 Semantic Segmentation
For pixel-level segmentation, ViT-based models (e.g., SETR – SEgmentation TRansformer) treat the problem as a sequence-to-sequence task, using a ViT encoder and a decoder with progressive upsampling. Swin Transformer’s hierarchical feature maps naturally integrate with U-Net-like decoders. Swin-L with a UperNet decoder achieves leading results on ADE20K and Cityscapes.
5.4 Video Understanding
ViT can be extended to video by processing spatiotemporal patches. Models like Video Vision Transformer (ViViT) factorize space and time, using separate attention mechanisms or joint spatiotemporal attention. TimeSformer applies divided attention (spatial and temporal in sequence). These architectures achieve competitive accuracy on action recognition benchmarks like Kinetics-400 while maintaining reasonable computational cost.
6 Limitations and Challenges
6.1 Need for Large Datasets
The most significant limitation of ViT is its data hunger. Without pre-training on datasets orders of magnitude larger than ImageNet-1k, ViT underperforms compared to CNNs of similar parameter count. This restricts its use in domains where large labeled datasets are unavailable. DeiT partially mitigates this through distillation, but the requirement for a strong teacher CNN remains a practical constraint.
6.2 Computational Cost and Scalability
Self-attention has quadratic complexity in the number of patches, making ViT computationally expensive for high-resolution images. The full (global) attention used in original ViT becomes prohibitive for large feature maps. Hierarchical models like Swin reduce this cost, but they introduce additional design complexity. Training large ViTs (ViT-L, ViT-H) demands extensive computational resources (e.g., hundreds of TPU-days), limiting accessibility.
6.3 Positional Encoding Resolution Mismatch
When fine-tuning ViT at higher resolutions than pre-training, the positional embeddings must be interpolated, which can cause degradation or require careful tuning. The interpolation often assumes a smooth spatial prior that may not hold for all tasks, leading to performance drops. Some variants (e.g., conditional positional encodings) address this, but the issue is not fully resolved.
7 Future Directions
7.1 Hybrid CNN-Transformer Architectures
Combining the strengths of convolutions (local processing, translation equivariance) with Transformers (global context, dynamic weighting) is a promising direction. Architectures like CoAtNet (Convolution + Attention) and ConvNeXt (modernized CNN with Transformer design choices) have achieved state-of-the-art results. Future work may explore more principled integration, such as attention-augmented convolutions or convolution-mediated token mixing.
7.2 Self-Supervised and Few-Shot Learning
Reducing ViT’s dependence on labeled data through self-supervised pre-training (e.g., masked image modeling, contrastive learning) is an active area. Models like MAE (Masked Autoencoder) and DINO have shown that ViT can learn strong visual representations without labels. Few-shot learning applications leveraging ViT’s global attention are also being explored, potentially enabling rapid adaptation with minimal examples.
7.3 Efficient Attention Mechanisms
To address computational cost, research is focused on more efficient attention variants, such as linear attention (e.g., Performer), sparse attention (e.g., Swin), and axial attention. Additionally, reducing the number of tokens through token merging or pruning (e.g., ToMe – Token Merging) can accelerate ViT inference without significant accuracy loss. These innovations will be crucial for deploying ViT on resource-constrained devices.