A transformer is a deep learning architecture introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al. It relies entirely on self-attention mechanisms to process sequential data, abandoning recurrence and convolution. Transformers have become the foundation of modern natural language processing (NLP) and have extended to computer vision, reinforcement learning, and other domains, enabling models like BERT, GPT, and vision transformers.
1 Architecture
The transformer follows an encoder–decoder structure composed of stacked identical layers, each relying on self-attention and feed-forward networks. The architecture processes sequences in parallel, enabling efficient training on modern hardware.
1.1 Encoder–Decoder Structure
The original transformer comprises an encoder that maps an input sequence to a continuous representation and a decoder that generates an output sequence from that representation. Both consist of multiple identical blocks.
1.1.1 Encoder Block
Each encoder block contains two sub-layers: a multi-head self-attention mechanism followed by a position-wise feed-forward network. Residual connections and layer normalization are applied around each sub-layer. The encoder processes the entire input sequence simultaneously.
1.1.2 Decoder Block
Each decoder block has three sub-layers: masked multi-head self-attention (with masking to prevent attending to future positions), cross-attention (attending to the encoder's output), and a feed-forward network. Residual connections and layer normalization are applied as in the encoder.
1.2 Self-Attention Mechanism
Self-attention computes a weighted sum of all positions in a sequence, where the weights are learned based on pairwise similarity. This allows the model to capture long-range dependencies without the sequential bottleneck of recurrence.
1.2.1 Scaled Dot-Product Attention
The core operation computes attention scores as the dot product of queries and keys, scaled by the inverse square root of the key dimension. These scores are passed through a softmax to obtain weights, which are then multiplied by values. The scaling prevents vanishing gradients for large dimensions.
1.2.2 Multi-Head Attention
Instead of performing a single attention function, the model projects queries, keys, and values linearly h times into lower-dimensional spaces, computes attention in parallel, concatenates the results, and projects them again. This allows the model to attend to information from different representation subspaces.
1.3 Positional Encoding
Since transformers process all positions simultaneously and have no inherent notion of order, positional encodings are added to the input embeddings to inject sequence position information.
1.3.1 Sinusoidal Encoding
The original paper uses fixed sinusoidal functions with different frequencies for each dimension. These encodings allow the model to extrapolate to sequence lengths unseen during training.
1.3.2 Learned Positional Embeddings
Alternatively, positional embeddings can be learned as trainable parameters, similar to word embeddings. This approach is simpler but may not generalize to lengths beyond those seen during training.
1.4 Feed-Forward Networks
Each encoder and decoder block contains a position-wise feed-forward network consisting of two linear transformations with a ReLU activation in between. This network is applied identically to each position, increasing the model's capacity for nonlinear transformations.
1.5 Layer Normalization and Residual Connections
Layer normalization stabilizes training by normalizing activations across the feature dimension. Residual connections (skip connections) allow gradients to flow directly through the network, enabling the training of very deep models. In the original transformer, layer normalization is applied after each sub-layer addition (post-norm), though later variants often use pre-norm.
2 Training and Optimization
Training transformers involves supervised learning with large text corpora, using specialized loss functions, optimizers, and regularization techniques tailored to the architecture.
2.1 Loss Functions
2.1.1 Cross-Entropy Loss
For language modeling and sequence generation tasks, the standard loss is categorical cross-entropy between the predicted token probabilities and the ground-truth tokens. It is minimized to improve next-token prediction accuracy.
2.1.2 Label Smoothing
Label smoothing replaces hard one-hot targets with a soft distribution that assigns a small uniform probability to all tokens. This reduces overconfidence and improves generalization and calibration.
2.2 Optimizer and Learning Rate Schedule
2.2.1 Adam with Warm-Up
The Adam optimizer is used with a custom learning rate schedule: a linear warm-up period followed by a decay proportional to the inverse square root of the step number. This prevents early instability and allows smooth convergence.
2.3 Regularization Techniques
2.3.1 Dropout
Dropout is applied to the output of each sub-layer, to the embeddings, and to the attention weights during training. It helps prevent overfitting, especially in large models.
2.3.2 Weight Decay
L2 weight decay (also known as regularization) penalizes large parameter magnitudes and is commonly used in combination with AdamW, a variant that decouples weight decay from the adaptive learning rate.
3 Variants and Extensions
Since the original transformer, numerous variants have been proposed, adapting the architecture for different tasks, efficiency needs, and modalities.
3.1 BERT Family
3.1.1 BERT (Bidirectional Encoder Representations from Transformers)
BERT uses only the encoder stack and is pretrained with masked language modeling and next-sentence prediction. It captures bidirectional context and achieves state-of-the-art results on many NLP benchmarks.
3.1.2 RoBERTa, ALBERT, DistilBERT
RoBERTa improves BERT by removing the next-sentence prediction objective and training longer on more data. ALBERT reduces parameter count through factorized embeddings and cross-layer sharing. DistilBERT is a smaller, faster version trained via knowledge distillation.
3.2 GPT Family
3.2.1 GPT-1, GPT-2, GPT-3, GPT-4
The Generative Pre-trained Transformer (GPT) series uses only the decoder stack with masked self-attention. GPT-1 established the viability of unsupervised pretraining. GPT-2 demonstrated scale benefits. GPT-3 introduced in-context learning with 175 billion parameters. GPT-4 further improved reasoning, multimodal abilities, and alignment.
3.3 Encoder-Decoder Variants
3.3.1 T5 (Text-to-Text Transfer Transformer)
T5 frames every NLP task as a text-to-text problem, using a standard encoder–decoder transformer. It is pretrained with a denoising objective (span corruption) on the C4 dataset.
3.3.2 BART
BART combines a bidirectional encoder (like BERT) with an autoregressive decoder (like GPT). It is pretrained by corrupting text (e.g., token masking, sentence permutation) and learning to reconstruct the original.
3.4 Sparse and Efficient Transformers
3.4.1 Reformer
Reformer uses locality-sensitive hashing to approximate attention and reversible layers to reduce memory footprint, enabling longer sequences with limited resources.
3.4.2 Longformer
Longformer replaces full self-attention with a combination of sliding window attention and global attention, allowing efficient processing of documents up to thousands of tokens.
3.5 Vision Transformers
3.5.1 ViT (Vision Transformer)
ViT applies a transformer directly to image patches treated as a sequence of tokens, achieving competitive performance on image classification when pretrained on large datasets.
3.5.2 DeiT, Swin Transformer
DeiT (Data-efficient Image Transformers) uses distillation to train ViT with less data. Swin Transformer introduces hierarchical feature maps and shifted windows, achieving state-of-the-art in object detection and segmentation.
4 Applications
Transformers have been applied across diverse domains, often achieving or surpassing prior state-of-the-art results.
4.1 Natural Language Processing
4.1.1 Machine Translation
The original transformer was designed for machine translation, outperforming recurrent and convolutional models on WMT benchmarks. Modern systems still use transformer-based architectures.
4.1.2 Text Summarization and Generation
Transformers power abstractive summarization models (e.g., PEGASUS) and generative models like GPT that can produce coherent long-form text.
4.1.3 Question Answering and Sentiment Analysis
BERT-based models achieve high scores on SQuAD (question answering) and GLUE/SuperGLUE (sentiment, entailment, etc.). Fine-tuned transformers are widely deployed in industry.
4.2 Computer Vision
4.2.1 Image Classification
ViT and its variants match or exceed convolutional neural networks (CNNs) on ImageNet when trained on sufficient data. They also transfer well to smaller datasets.
4.2.2 Object Detection and Segmentation
Vision transformers like DETR (Detection Transformer) and Swin Transformers perform end-to-end object detection and semantic segmentation without task-specific components.
4.3 Multimodal and Other Domains
4.3.1 Speech Processing (Speech Transformers)
Transformers are used for speech recognition (e.g., wav2vec 2.0), text-to-speech, and spoken language understanding, often combined with convolutional front-ends.
4.3.2 Reinforcement Learning (Decision Transformer)
The Decision Transformer reformulates reinforcement learning as a sequence modeling problem, predicting actions conditioned on past states and returns.
5 Impact and Influence
The transformer has reshaped the landscape of artificial intelligence, establishing new paradigms and driving rapid progress.
5.1 Pre-Training and Fine-Tuning Paradigm
Transformers enabled the two-stage approach: large-scale unsupervised pretraining on diverse text followed by supervised fine-tuning on specific tasks. This reduces the need for labeled data and has become standard in NLP.
5.2 Scaling Laws and Large Language Models
Research on scaling transformers revealed predictable relationships between model size, data size, and performance. This led to the development of large language models (LLMs) with hundreds of billions of parameters, such as GPT-3 and PaLM, which exhibit emergent abilities like in-context learning and reasoning.
5.3 Computational and Ethical Considerations
Training large transformers requires massive computational resources, raising concerns about energy consumption and carbon emissions. The deployment of LLMs also introduces ethical issues, including bias, misinformation, and potential misuse. Research continues on efficient architectures, distillation, and alignment techniques to address these challenges.