contextual embeddings

1 Introduction to contextual embeddings

1.1 Definition and core concept

Contextual embeddings are dense vector representations of words or subword tokens in natural language processing (NLP) that dynamically adjust based on the surrounding linguistic context. Unlike static word vectors that assign a single, fixed representation to each word type, contextual embeddings produce a unique vector for every occurrence of a token, influenced by the entire sentence or passage in which it appears. These embeddings are generated by deep neural networks that process sequences of text, capturing syntactic roles, semantic meanings, and discourse-level information. The core idea is that the meaning of a word is determined by its usage; thus, the same word in different contexts should receive different embeddings to reflect polysemy, homonymy, and pragmatic nuances.

1.2 Comparison with static embeddings

1.2.1 Polysemy handling

Static embeddings such as Word2Vec and GloVe represent each word with a single vector, conflating all its senses into one averaged representation. For example, the word "bank" (financial institution) and "bank" (river side) share the same vector, which can degrade performance on tasks requiring sense disambiguation. Contextual embeddings resolve this by generating context-specific vectors: "bank" in "He deposited money at the bank" receives a different embedding than in "He sat on the river bank." This capability arises from the model's attention to surrounding words, allowing it to capture sense-specific features.

1.2.2 Fixed vs. dynamic representations

Static embeddings are fixed after training: the same word always maps to the same vector, regardless of context. They are generated by counting or predicting co-occurrence patterns over a corpus and stored in a lookup table. Contextual embeddings are dynamic: they are computed on the fly as the model processes an input sentence. This dynamism enables the model to modulate representations based on syntactic structure (e.g., subject vs. object), semantic roles, and even stylistic variation. The shift from fixed to dynamic representations is one of the most significant advances in NLP, enabling models to generalize better across diverse linguistic phenomena.

1.3 Historical development

1.3.1 Pre-transformers era (ELMo, CoVe)

Before the transformer architecture became dominant, contextual embeddings were pioneered by recurrent neural network (RNN) based models. The Contextual Vectors (CoVe) model (McCann et al., 2017) used a deep LSTM encoder trained on machine translation to produce context-aware representations. Shortly after, ELMo (Embeddings from Language Models) (Peters et al., 2018) introduced a bidirectional LSTM trained on a large text corpus using a language modeling objective. ELMo extracted contextual embeddings from the internal states of the bidirectional LSTM, combining representations from all layers via task-specific weights. ELMo achieved state-of-the-art results on several NLP benchmarks at the time, demonstrating the power of deep contextualized word representations.

1.3.2 Transformer revolution (BERT, GPT)

The introduction of the transformer architecture (Vaswani et al., 2017) marked a paradigm shift. BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) employed a deep bidirectional transformer encoder trained with masked language modeling and next sentence prediction. BERT's contextual embeddings captured rich contextual information from both left and right contexts simultaneously, outperforming ELMo on a wide range of tasks. Meanwhile, GPT (Generative Pre-trained Transformer) (Radford et al., 2018) used a unidirectional (causal) transformer decoder, generating left-to-right contextual embeddings suitable for autoregressive generation. These models—and their successors like RoBERTa, GPT-2, and GPT-3—established contextual embeddings as the de facto standard in NLP, replacing static embeddings in most applications.

2 Technical foundations

2.1 Architectural approaches

2.1.1 Recurrent neural network (RNN) based

Early contextual embedding models, such as ELMo and CoVe, relied on RNNs, specifically long short-term memory (LSTM) networks. ELMo used a two-layer bidirectional LSTM: one LSTM processes the sequence left-to-right and another right-to-left. The hidden states from both directions at each token position are concatenated to form a single contextual vector. While RNN-based models can capture sequential dependencies, they suffer from vanishing gradients and limited parallelization, making it difficult to model long-range dependencies efficiently.

2.1.2 Transformer-based

Transformers replaced recurrence with self-attention mechanisms, enabling parallel processing and better capture of long-range dependencies. Transformer-based contextual embeddings are computed by stacking multiple self-attention and feed-forward layers.

2.1.2.1 Encoder-only models (BERT)

Encoder-only models, such as BERT, consist of a stack of transformer encoder layers. Each layer applies multi-head self-attention, allowing each token to attend to all other tokens in the input sequence. This bidirectional attention produces embeddings that incorporate context from both directions. BERT's output is a sequence of contextual vectors, one per input token, which can be used directly for classification or fine-tuned on downstream tasks. Variants like RoBERTa, ALBERT, and DistilBERT follow the same encoder-only paradigm.

2.1.2.2 Decoder-only models (GPT)

Decoder-only models, such as GPT, use transformer decoder layers with causal (masked) self-attention, where each token can only attend to previous tokens in the sequence. This left-to-right, autoregressive architecture is well suited for language generation tasks. The contextual embeddings produced by decoder-only models are conditioned solely on the left context, making them unidirectional. Despite this limitation, they excel at tasks like text completion and dialogue generation. Larger decoder-only models (GPT-3, GPT-4) have demonstrated remarkable few-shot and zero-shot capabilities.

2.1.3 Hybrid and masked language models

Some models combine elements of both encoder and decoder architectures. For example, T5 (Text-to-Text Transfer Transformer) uses an encoder-decoder structure, where the encoder produces bidirectional contextual embeddings, and the decoder generates output autoregressively. Another hybrid approach is XLM (Cross-lingual Language Model), which uses masked language modeling across multiple languages. Additionally, models like XLNet incorporate permutation language modeling to capture bidirectional context while retaining the autoregressive property. These hybrid designs offer flexibility for both understanding and generation tasks.

2.2 Training objectives

2.2.1 Masked language modeling (MLM)

Masked language modeling, used in BERT and its variants, randomly masks a percentage of input tokens (typically 15%) and trains the model to predict the original masked words based on their surrounding context. The objective is to minimize the cross-entropy loss between predicted and actual tokens. MLM forces the model to learn bidirectional context, as the prediction uses both left and right information. This objective is particularly effective for learning contextual embeddings that capture nuanced semantic and syntactic relationships.

2.2.2 Causal language modeling (CLM)

Causal language modeling, also known as autoregressive language modeling, is used in decoder-only models like GPT. The model predicts the next token in a sequence given all previous tokens. The training objective is to maximize the probability of the observed text under a left-to-right factorization. CLM does not require masking; it naturally generates embeddings that are causal (only left context). This objective is well suited for generation tasks where output is produced sequentially.

2.2.3 Next sentence prediction (NSP)

Next sentence prediction is an auxiliary objective used in BERT's pre-training. Given two sentences A and B, the model must predict whether B follows A in the original text. The goal is to learn relationships between sentences, such as coherence and discourse structure. NSP has been shown to improve performance on tasks like question answering and natural language inference. However, later studies (e.g., RoBERTa) found that NSP may be less critical than MLM, and some models remove it in favor of larger batch sizes and longer training.

2.3 Input representation

2.3.1 Tokenization (subword, BPE, WordPiece)

Contextual embedding models typically use subword tokenization to handle out-of-vocabulary words and morphological variations. Byte-Pair Encoding (BPE) and WordPiece are two common algorithms. BPE iteratively merges the most frequent pairs of characters or byte sequences to form a fixed-size vocabulary of subword units. WordPiece, used in BERT, is similar but uses a likelihood-based criterion for merging. Subword tokenization ensures that any word can be represented as a sequence of known tokens, reducing the vocabulary size while preserving linguistic structure.

2.3.2 Positional encodings

Since transformers are permutation-invariant (self-attention has no inherent notion of order), positional encodings are added to token embeddings to inject sequence position information. Two main types exist: absolute positional encodings (used in BERT), which add a fixed or learned vector for each position index, and relative positional encodings (used in GPT-2, T5), which encode the distance between tokens. Some models, like RoBERTa, use learned absolute positional encodings, while others, like ALBERT, share them across layers.

2.3.3 Segment and token type embeddings

For tasks involving multiple sentences (e.g., question answering, NLI), models like BERT use segment embeddings to distinguish between input segments. Typically, two segment embeddings (A and B) are added to token embeddings to mark which sentence a token belongs to. Additionally, special tokens like [CLS] (classification) and [SEP] (separator) are inserted. Token type embeddings encode whether the token belongs to the first or second segment, enabling the model to process paired inputs effectively.

3 Extracting and using contextual embeddings

3.1 Layer selection strategies

3.1.1 Last layer vs. weighted summation

When using a pre-trained model as a feature extractor (without fine-tuning), one must decide which internal layer(s) to use as embeddings. The simplest approach is to take the hidden states from the last transformer layer. However, for some tasks, embeddings from earlier layers may capture more syntactic information, while later layers are more semantic. Weighted summation, as popularized by ELMo, computes a weighted average of all layers' hidden states, where the weights are learned during task-specific training. This provides flexibility and often yields better performance than using only the last layer.

3.1.2 Task-specific fine-tuning vs. feature extraction

Two primary paradigms exist for using contextual embeddings. In feature extraction, the pre-trained model's parameters are frozen, and its output embeddings are fed into a separate classifier (e.g., a linear layer or an LSTM). This approach is computationally cheaper and works well when labeled data is scarce. In fine-tuning, the entire pre-trained model is loaded and further trained on the downstream task, allowing all layers to adapt to the target domain. Fine-tuning typically yields higher performance but requires more computational resources and careful training to avoid overfitting. Many modern frameworks default to fine-tuning, especially with transformer models.

3.2 Transfer learning pipeline

3.2.1 Pre-training phase

Pre-training is the initial step where a large-scale model is trained on a massive text corpus (e.g., Wikipedia, Common Crawl) using self-supervised objectives like MLM or CLM. This phase is computationally expensive and typically performed only once by research labs or large organizations. The resulting pre-trained model weights capture general linguistic knowledge, including syntax, semantics, and world knowledge. The model is then made publicly available through model hubs.

3.2.2 Fine-tuning for downstream tasks

Fine-tuning adapts the pre-trained model to a specific task by continuing training on a labeled dataset. This phase usually involves adding a task-specific output head (e.g., a classification layer for sentiment analysis) and updating the model's parameters via supervised learning. Fine-tuning is relatively quick and data-efficient, often requiring only a few thousand examples to achieve strong performance. The contextual embeddings produced after fine-tuning are tailored to the target task while retaining the benefits of pre-trained knowledge.

3.3 Common frameworks and libraries

3.3.1 Hugging Face Transformers

The Hugging Face Transformers library is the most widely used toolkit for working with contextual embeddings. It provides a unified API to load, fine-tune, and deploy thousands of pre-trained models, including BERT, GPT, T5, and many others. The library supports both PyTorch and TensorFlow, offers easy access to tokenizers, and includes pipelines for common tasks (e.g., text classification, question answering). It also hosts the Model Hub, a repository where users can share and download fine-tuned models.

3.3.2 TensorFlow Hub

TensorFlow Hub provides a collection of pre-trained model components, including contextual embedding modules such as BERT and ELMo. These modules can be integrated into TensorFlow pipelines as reusable layers. However, TensorFlow Hub has become less prominent since the rise of Hugging Face, but it remains useful for projects already invested in TensorFlow 2.x.

3.3.3 PyTorch and Flair

PyTorch is a popular deep learning framework that offers native support for contextual embeddings through libraries like "transformers" (Hugging Face) and "Flair." Flair is a dedicated NLP library that simplifies the extraction of contextual embeddings from various models (e.g., BERT, ELMo, Flair's own character-level LSTM). It provides a high-level interface for embedding words, sentences, and documents, making it easy to incorporate contextual representations into custom architectures.

4 Applications in natural language processing

4.1 Text classification and sentiment analysis

Contextual embeddings have become the default representation for text classification tasks. By feeding the [CLS] token's embedding (or a pooled representation) into a classifier, models can achieve state-of-the-art performance on sentiment analysis, topic categorization, spam detection, and more. The context-sensitive nature of the embeddings allows the model to disambiguate sentiment based on phrasing (e.g., "not bad" vs. "bad") and capture subtle nuances such as irony or sarcasm.

4.2 Named entity recognition (NER)

In NER, contextual embeddings help identify entities like persons, locations, and organizations. Because the same word may be an entity in one context and a common noun in another (e.g., "Apple" as a company vs. fruit), context-sensitive vectors are crucial. Models typically augment contextual embeddings with a sequence labeling head (e.g., a conditional random field) to predict entity spans. BERT-based NER models have achieved F1 scores exceeding 95% on standard benchmarks like CoNLL-2003.

4.3 Question answering and reading comprehension

For extractive question answering (e.g., SQuAD), a question and a passage are concatenated and fed into a transformer encoder. Contextual embeddings from the model are used to predict start and end positions of the answer span within the passage. The bidirectional attention in encoder-only models allows the model to relate the question to relevant parts of the passage, leading to highly accurate answers. Fine-tuned BERT models have matched or exceeded human performance on SQuAD 2.0.

4.4 Machine translation and summarization

Encoder-decoder models (e.g., T5, BART) leverage contextual embeddings for sequence-to-sequence tasks. In machine translation, the encoder produces contextual representations of the source text, which are then decoded into the target language. In summarization, the encoder processes the input document, and the decoder generates a condensed version. Contextual embeddings enable the model to capture discourse structure and important information, improving fluency and faithfulness.

4.5 Semantic textual similarity and retrieval

Contextual embeddings can be used to compute similarity between sentences or documents by comparing their pooled embeddings (e.g., via cosine similarity). This is the basis for semantic search, paraphrase detection, and clustering applications. However, raw embeddings from large models may not be optimal for similarity tasks; sentence embedding models like Sentence-BERT are fine-tuned specifically to produce high-quality, discriminative contextual embeddings for comparison.

5 Challenges and limitations

5.1 Computational cost and memory footprint

Generating contextual embeddings requires running a deep neural network, which is computationally expensive compared to static embeddings that are simply lookup operations. Large transformer models with hundreds of millions or billions of parameters demand powerful GPUs or TPUs, and memory usage can be prohibitive for long documents or real-time applications. This poses a barrier for smaller organizations and resource-constrained environments. Techniques such as knowledge distillation, quantization, and pruning attempt to reduce this cost.

5.2 Context length and positional bias

Most transformer models have a fixed maximum context length (e.g., 512 tokens for BERT, 1024 for GPT-2). Texts exceeding this length must be truncated or split into segments, which can lose long-range dependencies. Even within the context window, positional encodings may bias the model toward local patterns, making it harder to capture very long-range relationships. Recent models like Longformer and Reformer address this by using sparse attention mechanisms to handle longer sequences.

5.3 Domain adaptation and out-of-distribution performance

Pre-trained models are typically trained on general-domain text (e.g., Wikipedia, news). When applied to specialized domains such as medicine, law, or scientific literature, their contextual embeddings may underperform due to domain-specific vocabulary, jargon, and different stylistic conventions. Fine-tuning on in-domain data can mitigate this, but supervised data may be scarce. Domain adaptation techniques, including continued pre-training on domain corpora, are active areas of research.

5.4 Interpretability and bias in contextual embeddings

Contextual embeddings are opaque: it is difficult to understand which aspects of the input contribute to a particular embedding. This lack of interpretability hinders trust in high-stakes applications. Moreover, pre-trained models can learn and amplify societal biases present in training data (e.g., gender, racial, or socioeconomic stereotypes). Biases may be encoded in contextual embeddings, leading to unfair or discriminatory outcomes. Mitigation strategies include debiasing post-processing, adversarial training, and careful curation of training data.

6 Evaluation and benchmarks

6.1 Intrinsic evaluation tasks

6.1.1 Word similarity and relatedness

Intrinsic evaluation measures the quality of contextual embeddings without reference to downstream tasks. Word similarity tasks, such as WordSim-353 and SimLex-999, compare the cosine similarity between embeddings of word pairs against human judgments of semantic similarity. However, because contextual embeddings vary per context, these evaluations often use averaged representations over many contexts or focus on specific senses. This makes intrinsic evaluation less straightforward than for static embeddings.

6.1.2 Word sense disambiguation

Word sense disambiguation (WSD) tasks assess whether contextual embeddings can correctly distinguish between different senses of a word. Datasets like SemCor and Senseval provide annotated examples with sense labels. A contextual embedding model is evaluated by clustering or classifying embeddings of ambiguous words and comparing to the gold sense labels. High performance on WSD indicates that the embeddings effectively capture contextual variation.

6.2 Extrinsic evaluation tasks (GLUE, SuperGLUE)

Extrinsic evaluation measures the usefulness of contextual embeddings on downstream tasks. The General Language Understanding Evaluation (GLUE) benchmark, and its harder successor SuperGLUE, consist of multiple tasks covering sentiment analysis, natural language inference, paraphrasing, and more. Models are fine-tuned on each task and scored according to task-specific metrics. Contextual embedding models such as BERT and RoBERTa have set state-of-the-art scores on these benchmarks. Performance on GLUE/SuperGLUE is a standard proxy for overall language understanding capability.

6.3 Cross-lingual and multilingual benchmarks

Multilingual benchmarks assess contextual embeddings across languages. XNLI (Cross-lingual NLI) extends the MultiNLI dataset to 15 languages. Other benchmarks include MLQA (Multilingual Question Answering) and XTREME (Cross-lingual TRansfer Evaluation of Multilingual Encoders). These tasks measure how well contextual embeddings learned from one language can transfer to others (zero-shot cross-lingual transfer) or how well multilingual models (e.g., mBERT, XLM-R) perform across diverse languages.

7 Future directions

7.1 Efficient contextual embeddings (distillation, quantization)

To address computational costs, researchers are developing lighter variants of large models. Knowledge distillation trains a smaller "student" model to mimic the output of a larger "teacher" model, producing compact yet effective contextual embeddings (e.g., DistilBERT, TinyBERT). Quantization reduces the precision of model weights (e.g., from 32-bit to 8-bit), lowering memory and accelerating inference with minimal accuracy loss. These approaches make contextual embeddings more accessible for edge devices and real-time applications.

7.2 Long-context modeling (e.g., Longformer, Reformer)

Scaling transformers to longer sequences remains an active area. Models like Longformer and BigBird use sparse attention patterns (e.g., sliding window, global attention) to reduce the quadratic complexity of full self-attention, enabling context lengths up to 8,192 tokens or more. Reformer introduces locality-sensitive hashing to approximate attention, further improving efficiency. Future work may enable context windows on the order of millions of tokens, allowing modeling of entire books or long documents.

7.3 Multimodal contextual embeddings

Combining text with other modalities (images, audio, video) to create multimodal contextual embeddings is a growing trend. Models like CLIP (Contrastive Language-Image Pre-training) align text and image embeddings in a shared space. Others, such as LLaVA, integrate language models with vision encoders. Multimodal embeddings enable tasks like image captioning, visual question answering, and cross-modal retrieval. The goal is to build holistic representations that understand the world across different sensory inputs.

7.4 Continual learning and adaptation

Pre-trained models are typically static once released. Continual learning aims to update contextual embeddings incrementally as new data arrives, without catastrophic forgetting of previous knowledge. This is important for models deployed in dynamic environments, such as news or social media, where language evolves. Techniques like elastic weight consolidation, replay buffers, and progressive neural networks are being explored. Successful continual learning would allow contextual embeddings to remain current and adaptive over time.