BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained natural language processing (NLP) model developed by Google in 2018. It is based on the Transformer architecture and uses a bidirectional approach to learn deep contextual representations of words by jointly conditioning on both left and right contexts in all layers. BERT has achieved state-of-the-art results on a wide range of NLP tasks, including question answering, sentiment analysis, and named entity recognition, by fine-tuning the pre-trained model on task-specific data.
1 Architecture
1.1 Transformer Encoder
BERT employs the encoder component of the Transformer architecture, originally introduced by Vaswani et al. in 2017. The encoder consists of a stack of identical layers, each containing two core sub-layers: a multi-head self-attention mechanism and a position-wise feed-forward network. Both sub-layers are augmented with layer normalization and residual connections.
1.1.1 Multi-Head Self-Attention
The multi-head self-attention mechanism allows the model to attend to different positions in the input sequence simultaneously. It computes attention scores between every pair of tokens, enabling the model to capture relationships between words regardless of their distance. Each attention head operates on a projected version of the input, and the outputs of all heads are concatenated and linearly transformed.
1.1.2 Position-wise Feed-Forward Networks
Each position in the sequence is processed independently by a fully connected feed-forward network. This network consists of two linear transformations with a non-linear activation function (typically GELU) in between. It introduces additional capacity and non-linearity to the model.
1.1.3 Layer Normalization and Residual Connections
Layer normalization stabilizes training by normalizing the activations across the feature dimension. Residual connections add the input of a sub-layer to its output, which helps mitigate the vanishing gradient problem and enables training of deep networks. Both are applied after each sub-layer in the Transformer encoder.
1.2 Bidirectional Context
A key innovation of BERT is its bidirectional context modeling. Unlike earlier models that processed text left-to-right or right-to-left, BERT conditions each token on both its left and right contexts simultaneously. This is achieved through the Masked Language Model pre-training objective, which forces the model to consider the full context.
1.2.1 Comparison with Unidirectional Models (e.g., GPT)
Unidirectional models, such as GPT (Generative Pre-trained Transformer), process text in a single direction (left-to-right) and predict the next token based only on previous tokens. BERT's bidirectional approach captures richer contextual information because it can leverage both preceding and following words. However, bidirectional models cannot easily generate text autoregressively, whereas unidirectional models are naturally suited for generation tasks.
1.2.2 Masked Language Model (MLM) Objective
The MLM objective randomly masks a percentage of input tokens and trains the model to predict the original vocabulary ID based on the surrounding context. This denoising autoencoding-like task enables bidirectional context to be used because the model must infer the masked token from both left and right.
1.3 Input Representations
BERT uses a combination of three types of embeddings to represent each input token: token embeddings, segment embeddings, and position embeddings. These are summed to form the final input representation.
1.3.1 Token Embeddings
Token embeddings map each token (subword unit) in the vocabulary to a dense vector. BERT uses a WordPiece tokenization scheme with a vocabulary of approximately 30,000 tokens.
1.3.2 Segment Embeddings
Segment embeddings indicate which sentence a token belongs to. This is particularly useful for tasks involving pairs of sentences, such as question answering or natural language inference. Segment embeddings are usually of two types: one for sentence A and one for sentence B.
1.3.3 Position Embeddings
Position embeddings encode the position of each token in the input sequence. Unlike the original Transformer, which uses sinusoidal positional encodings, BERT uses learnable positional embeddings.
1.3.4 Special Tokens ([CLS], [SEP])
Two special tokens are added to every input sequence: a [CLS] token at the beginning, whose final hidden state is used for classification tasks, and a [SEP] token that separates and marks the end of sentences. For tasks involving sentence pairs, each sentence ends with [SEP].
2 Training
2.1 Pre-training Tasks
BERT is pre-trained on two unsupervised tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). These tasks allow the model to learn deep language representations from large text corpora.
2.1.1 Masked Language Model (MLM)
In MLM, a random subset of tokens in the input sequence is selected for prediction. The model is trained to predict the original token given its context. This forces the model to develop a strong understanding of word semantics and syntax.
2.1.1.1 Masking Strategy (80-10-10)
When a token is selected for masking, the following strategy is applied with a probability of 15% across all tokens:
- 80% of the time: the token is replaced with the [MASK] token.
- 10% of the time: the token is replaced with a random token.
- 10% of the time: the token is left unchanged.
This strategy prevents the model from relying too heavily on the [MASK] token and encourages it to use full context.
2.1.2 Next Sentence Prediction (NSP)
NSP is a binary classification task where the model receives two sentences and must determine whether the second sentence follows the first in the original text. This task helps BERT learn relationships between sentences, which benefits downstream tasks like question answering and natural language inference.
2.2 Training Data
BERT is pre-trained on a large corpus comprising the BooksCorpus (800 million words) and English Wikipedia (2.5 billion words). The Wikipedia data is preprocessed by extracting only text passages (ignoring lists, tables, and headers).
2.2.1 BooksCorpus and English Wikipedia
The BooksCorpus consists of over 11,000 unpublished books spanning various genres. English Wikipedia provides a diverse set of articles covering a wide range of topics. Together, they offer rich linguistic patterns and domain coverage.
2.3 Pre-training Hyperparameters
BERT Base uses a configuration of 12 Transformer layers, 768 hidden units, and 12 attention heads (110 million parameters). BERT Large uses 24 layers, 1024 hidden units, and 16 attention heads (340 million parameters).
2.3.1 Batch Size, Learning Rate, and Optimizer
Pre-training typically uses a batch size of 256 sequences (each of length 512 tokens) for BERT Base. The learning rate is set to 1e-4 with a warmup phase over the first 10,000 steps, followed by linear decay. The Adam optimizer is used with β1=0.9, β2=0.999, and a weight decay of 0.01. Training runs for 1,000,000 steps (approximately 40 epochs over the corpus).
3 Variants and Extensions
3.1 BERT Base and BERT Large
The two primary versions differ in model size and performance:
- BERT Base: 12 layers, 768 hidden, 12 heads, 110M parameters. Comparable in size to OpenAI's GPT.
- BERT Large: 24 layers, 1024 hidden, 16 heads, 340M parameters. Achieves higher accuracy on most tasks at the cost of more computational resources.
3.2 Domain-Specific BERTs
Several specialized variants of BERT have been developed to perform better on domain-specific text.
3.2.1 BioBERT (Biomedical)
BioBERT is pre-trained on biomedical text from PubMed abstracts and PMC full-text articles. It consistently outperforms BERT on biomedical NLP tasks such as named entity recognition, relation extraction, and question answering.
3.2.2 SciBERT (Scientific Text)
SciBERT is trained on a large corpus of scientific publications from the Semantic Scholar database, covering multiple disciplines. It uses its own vocabulary (SciVocab) and achieves strong results on scientific text classification and entity recognition.
3.2.3 ClinicalBERT (Clinical Notes)
ClinicalBERT is pre-trained on de-identified electronic health records from the MIMIC-III database. It is designed for clinical NLP tasks like patient readmission prediction and medical entity extraction.
3.3 Efficiency Improvements
Several variants aim to reduce the computational and memory costs of BERT while maintaining or improving performance.
3.3.1 DistilBERT
DistilBERT uses knowledge distillation during pre-training to compress BERT Base by 40% while retaining 97% of its language understanding capabilities. It has 66M parameters and runs 60% faster.
3.3.2 ALBERT (A Lite BERT)
ALBERT introduces factorized embedding parameterization and cross-layer parameter sharing to reduce memory usage. It also replaces NSP with sentence order prediction (SOP). Despite fewer parameters, ALBERT achieves competitive or better performance.
3.3.3 RoBERTa (Robustly Optimized BERT)
RoBERTa builds on BERT by using more training data, larger batches, longer training, and removing the NSP objective. It also uses dynamic masking instead of static masking. RoBERTa often outperforms BERT on benchmarks.
4 Fine-tuning and Applications
4.1 Fine-tuning Process
Fine-tuning adapts the pre-trained BERT model to a specific downstream task by training it on a labeled dataset. The pre-trained weights are updated via backpropagation, typically with a lower learning rate to prevent catastrophic forgetting.
4.1.1 Task-Specific Head Addition
For each downstream task, a simple classification layer is added on top of BERT's encoder. For example:
- For text classification, a linear layer takes the [CLS] token representation.
- For question answering, a span prediction layer predicts the start and end positions of the answer.
- For token-level tasks like NER, a linear layer takes each token's representation.
4.1.2 Fine-tuning Hyperparameters
Common hyperparameters include a batch size of 16–32, learning rate of 2e-5 to 5e-5 (with Adam), and training for 2–4 epochs. Dropout (0.1) is applied in the classification head. Hyperparameter tuning is task-specific.
4.2 Downstream Tasks
BERT has been successfully applied to a wide range of NLP tasks.
4.2.1 Text Classification (e.g., Sentiment Analysis)
In sentiment analysis, BERT takes a sentence (e.g., a movie review) and predicts its sentiment (positive/negative). The [CLS] representation is fed to a softmax classifier. BERT captures subtle contextual cues that improve accuracy over bag-of-words or LSTM-based models.
4.2.2 Question Answering (e.g., SQuAD)
For the Stanford Question Answering Dataset (SQuAD), BERT receives a question and a passage, then outputs a span (start and end indices) from the passage as the answer. BERT achieved human-level performance on SQuAD 1.1 and near-human on SQuAD 2.0.
4.2.3 Named Entity Recognition (NER)
In NER, BERT labels each token in a sequence with an entity type (person, location, organization, etc.). The token representations are passed through a linear layer and a conditional random field (CRF) to enforce label dependencies.
4.2.4 Natural Language Inference (NLI)
In NLI, BERT evaluates the logical relationship between a premise and a hypothesis (entailment, contradiction, or neutral). The [CLS] representation is used for classification. BERT models like RoBERTa have topped the GLUE NLI tasks.
5 Evaluation and Benchmarks
5.1 GLUE Benchmark
The General Language Understanding Evaluation (GLUE) benchmark is a collection of nine NLP tasks, including sentiment analysis (SST-2), linguistic acceptability (CoLA), and NLI (MNLI). BERT Base achieved an average score of 86.7, surpassing previous state-of-the-art models. BERT Large achieved 87.4.
5.2 SQuAD Leaderboard
On the SQuAD 1.1 leaderboard, BERT Large achieved an F1 score of 93.2, closely matching human performance (91.2 exact match, 96.0 F1). On the more challenging SQuAD 2.0 (which includes unanswerable questions), BERT Large scored 83.6 exact match and 88.7 F1.
5.3 Impact and Limitations
5.3.1 Computational Cost
Pre-training BERT requires significant computational resources. Training BERT Base on 4 TPUs took about 4 days. BERT Large required 16 TPUs and took about 4 days as well. Fine-tuning is less expensive but still requires a GPU for practical use. This cost has motivated the development of more efficient variants and distillation techniques.
5.3.2 Lack of Generative Capability
Because BERT uses bidirectional context via MLM, it cannot generate text in a left-to-right manner without modifications. It is primarily an encoder-based model suited for understanding tasks. For text generation, models like GPT or T5 are more appropriate. Some extensions (e.g., BART, T5) combine BERT-like encoding with autoregressive decoding to overcome this limitation.