1.1 Definition and Rationale

Subword information refers to the processing and representation of text at a granularity between individual characters and whole words. In natural language processing (NLP), subword units are obtained by splitting words into smaller, frequently occurring segments—such as byte-pair encodings, WordPiece tokens, or unigram segments—that capture morphological and orthographic regularities. The primary rationale for using subword tokenization is to address the open-vocabulary problem: no fixed word-level vocabulary can cover all possible words, especially in morphologically rich languages, domain-specific terminology, or user-generated content with spelling variations. Subword tokenization offers a compact vocabulary that can represent any input string while preserving meaningful linguistic substructure.

1.2 Relationship to Character‑level and Word‑level Representations

1.2.1 Trade‑offs in Granularity

Character-level representations offer full coverage and robustness to misspellings but suffer from long sequence lengths and a lack of lexical semantics. Word-level representations capture semantics and are computationally efficient (short sequences) but cannot handle unseen words and require large vocabularies. Subword tokenization strikes a balance: it reduces sequence length compared to characters while enabling open-vocabulary coverage with a moderate vocabulary size (typically 10k–100k tokens). The choice of granularity affects model capacity, memory usage, and downstream performance.

1.2.2 Handling Out‑of‑Vocabulary Tokens

Out-of-vocabulary (OOV) tokens are a critical challenge for word-level models. Subword tokenization eliminates OOV tokens by decomposing any unknown word into a sequence of known subword units (e.g., “unhappiness” might become “un##”, “happiness”). This decomposition is deterministic for algorithms like Byte-Pair Encoding (BPE) and WordPiece, allowing models to process arbitrary input without special “unknown” tokens.

1.3 Historical Context and Development

1.3.1 Early Morphological Segmentation

Before the widespread adoption of neural networks, linguistic approaches such as stemming and lemmatization attempted to reduce words to root forms. Rule-based morphological analyzers (e.g., for Finnish or Turkish) were developed to handle agglutinative morphology. However, these systems required language-specific resources and manual rules, and they struggled with irregular forms and unknown words.

1.3.2 Rise of Neural Network‑based Subword Models

The introduction of neural machine translation (NMT) in the mid-2010s accelerated the need for subword methods. In 2016, Sennrich et al. proposed Byte-Pair Encoding (BPE) for NMT, demonstrating that splitting words into frequent character sequences improved translation quality for rare and unknown words. Shortly thereafter, Wu et al. (2016) introduced WordPiece for Google’s Neural Machine Translation system, and Kudo (2018) proposed Unigram language model tokenization. These methods became standard in transformer architectures such as BERT, GPT, and T5.

2.1 Byte‑Pair Encoding (BPE)

2.1.1 Basic Algorithm and Frequency Merging

Byte-Pair Encoding (BPE) is a data compression algorithm adapted for subword tokenization. It starts with a character-level vocabulary (e.g., all Unicode bytes) and iteratively merges the most frequent adjacent pair of symbols. Each merge creates a new symbol representing the pair. The process continues until a predefined vocabulary size is reached. To tokenize new text, the learned merge rules are applied greedily to the input string. BPE is simple, deterministic, and widely used.

2.1.2 Variants: SentencePiece and GPT‑style BPE

SentencePiece (Kudo & Richardson, 2018) extends BPE by treating the input as a raw byte stream, eliminating the need for pre-tokenization (e.g., splitting on whitespace). It includes direct support for byte-level BPE and unigram LM tokenization. The GPT family (GPT-2, GPT-3, GPT-4) uses a similar byte-level BPE (often called “Byte-Level BPE” or “BBPE”), which operates on bytes rather than Unicode characters, ensuring uniform handling of all characters and reducing the risk of unicode normalization issues.

2.2 WordPiece

2.2.1 Likelihood‑based Merging Criterion

WordPiece (Schuster & Nakajima, 2012) shares the iterative merging process of BPE but uses a different criterion: at each step, it selects the pair that maximizes the likelihood of the training data when merged. This likelihood-based objective is derived from a language model of subword units. The resulting vocabulary tends to capture linguistically meaningful subunits more effectively than pure frequency-based merging.

2.2.2 Use in BERT and Other Transformers

WordPiece gained prominence through its use in BERT (Devlin et al., 2019). BERT’s tokenizer (often called “BERT WordPiece”) pre-tokenizes on whitespace and punctuation, then applies WordPiece merging. It uses a special prefix “##” to indicate that a token is a continuation of a word (e.g., “un##”, “##able”). This notation helps preserve word boundaries during reconstruction. WordPiece is also employed in models such as ALBERT, DistilBERT, and ELECTRA.

2.3 Unigram Language Model Tokenization

2.3.1 Probabilistic Segmentation and Regularization

The Unigram tokenizer (Kudo, 2018) models segmentation as a probabilistic process. It defines a vocabulary of subword units and assigns each unit a probability, typically derived from a unigram language model. During training, it alternates between estimating unit probabilities via the Expectation-Maximization (EM) algorithm and pruning low-probability units. The final vocabulary is selected to minimize the expected loss (e.g., negative log-likelihood) over the training data. This approach allows multiple candidate segmentations; during inference, the tokenizer returns the most probable segmentation.

2.3.2 Training Procedure and Vocabulary Selection

Training begins with a seed vocabulary (e.g., all characters plus high-frequency character pairs). The EM algorithm estimates unit probabilities, and the units with the smallest marginal contribution to the overall likelihood are removed. This pruning is repeated until the desired vocabulary size is reached. Unigram tokenization adapts well to different languages and can be combined with SentencePiece for language-agnostic tokenization.

2.4 Hybrid and Adaptive Approaches

2.4.1 Morpheme‑aware Segmentation

Some approaches integrate linguistic knowledge by using morphological analyzers to guide subword segmentation. For example, BPE could be constrained to avoid splitting morphemes that are roots or affixes. Systems like Morfessor (Creutz & Lagus, 2005) provide unsupervised morphological segmentation that can serve as initial subword units. Hybrid methods often improve performance on morphologically rich languages at the cost of additional computational complexity.

2.4.2 Dynamic Vocabulary Methods

Rather than using a fixed vocabulary, dynamic methods adjust the subword vocabulary during training or inference. Language models can learn to compose subword units on the fly, or use reinforcement learning to select segmentations that minimize downstream loss. However, these methods are not yet widely adopted due to increased training instability and slower inference.

3.1 Subword Embeddings

3.1.1 Initialization and Positional Encoding

Subword tokens are mapped to dense vector embeddings, typically initialized randomly and learned during training. The embedding matrix has dimensions (vocabulary size × embedding dimension). Because subword sequences are often longer than word-level sequences, the positional encoding (e.g., sinusoidal or learned) must be applied at the subword level. Transformers treat each subword token as a separate position, preserving the order of the original text.

3.1.2 Handling Ambiguity and Multiple Segmentations

A word can have multiple valid subword segmentations (especially with Unigram tokenizers). Models must be robust to this ambiguity. Common practices include using the most probable segmentation during inference or averaging embeddings over multiple segmentations at training time. Some models (e.g., BERT) use a deterministic tokenizer to avoid ambiguity, while others (e.g., some multilingual models) employ a fallback strategy for rare characters.

3.2 Subword‑level Attention Mechanisms

3.2.1 Full Attention Span Across Subword Tokens

In transformer-based architectures, the self-attention mechanism operates over the full sequence of subword tokens. Attending to every subword token allows the model to capture relationships between any two subword positions. This is computationally expensive for long sequences (O(n²)) but enables powerful context modeling. Subword attention has been shown to learn syntactic and morphological patterns even without explicit word boundaries.

3.2.2 Hierarchical or Chunked Attention

To reduce computational cost, some models employ hierarchical attention: first attend within word-level chunks, then across chunks. Reformer (Kitaev et al., 2020) used locality-sensitive hashing to approximate full attention, and Longformer (Beltagy et al., 2020) combined local sliding-window attention with global attention for selected subword positions. These methods can handle sequences of tens of thousands of subword tokens, which is beneficial for processing entire documents.

3.3 Subword and Char‑level Hybrid Models

3.3.1 Character‑CNN Over Subword Units

Some architectures, such as ELMo (Peters et al., 2018), use a Character Convolutional Neural Network (Char-CNN) to produce character-level embeddings that are then combined with subword information. The Char-CNN provides robust handling of out-of-vocabulary tokens and can capture orthographic similarities (e.g., “run”, “runs”, “running” share character n-grams). These hybrid models often outperform pure subword models on tasks requiring sensitivity to spelling.

3.3.2 End‑to‑End Subword Transformer Pipelines

Most modern language models (e.g., GPT, BERT, T5) are end-to-end pipelines: raw text → tokenizer → embedding + transformer → output. The same subword vocabulary is used throughout pretraining and fine-tuning. In these pipelines, the tokenizer is a fixed component, and the transformer learns to model dependencies between subword tokens without explicit character-level features. Fine-tuning on specific tasks may require task-specific tokenization adjustments, but the underlying subword representation remains unchanged.

4.1 Machine Translation

4.1.1 Reducing Vocabulary Explosion Across Languages

Machine translation systems must handle vocabularies covering multiple languages. Subword tokenization drastically reduces the combined vocabulary size: instead of storing millions of word forms, a shared subword vocabulary of 32k–64k tokens can represent all languages. This enables multilingual translation models (e.g., Google’s Multilingual Neural Machine Translation) to share parameters across languages.

4.1.2 Handling Compound and Agglutinative Languages

Languages like German (compounds) and Turkish (agglutination) have very long words that are rare or unseen at the word level. Subword tokenization naturally decomposes such words (e.g., “Donaudampfschifffahrtsgesellschaft” into “Donau”, “dampf”, “schiff”, …). This leads to more reliable translation and better generalization to novel compounds.

4.2 Text Generation and Language Modeling

4.2.1 Open‑ended Generation with Subword Vocabularies

Language models like GPT‑3 generate text token by token using a subword vocabulary. The vocabulary is large enough to produce coherent human-like text while remaining manageable for training. Subword tokens enable the model to generate rare words, acronyms, and novel combinations by concatenating common subword pieces.

4.2.2 Controlling Tokenization for Style and Fluency

Some generation tasks benefit from controlling the segmentation granularity. For example, using a smaller subword vocabulary (more tokens per word) may improve coverage of rare spellings at the cost of longer sequences. Conversely, a larger vocabulary (fewer tokens per word) can speed up generation but may produce unnatural splits (e.g., “artificial” becoming “art#i#ficial”). Researchers have explored meta-learning to adapt tokenization to the desired output style.

4.3.1 Subword Indexing for Morphologically Rich Languages

Search engines traditionally use stemming or lemmatization to handle morphological variants. Subword indexing replaces these with a unified approach: documents and queries are tokenized into subword units, allowing matches on partial word forms. For example, a query for “runs” might match “run##s” in an indexed document. This improves recall for morphologically rich languages like Arabic or Finnish.

4.3.2 Approximate Matching and Query Expansion

Subword tokenization facilitates approximate string matching by allowing common substrings to act as triggers. In query expansion, the system can add subword variants (e.g., “–ing” forms) to the query without requiring a thesaurus. Additionally, subword-based embeddings can be used to compute similarity between words with spelling differences (e.g., “color” vs. “colour”).

4.4 Sentiment Analysis and Text Classification

4.4.1 Subword Features for Domain Adaptation

Sentiment models often trained on product reviews may encounter domain-specific jargon (e.g., “uninstalled” in software reviews). Subword tokenization helps model generalize because it can decompose the unknown word into “un##” and “##installed” (where “installed” is likely in the vocabulary). This allows the model to adapt to new domains without retraining the tokenizer.

4.4.2 Robustness to Spelling Variations

Social media text and user comments frequently contain misspellings, abbreviations, and emoji. Subword tokenization, especially byte-level BPE, can handle typos (e.g., “happpy” → “happ##py”) by breaking unknown character sequences into known substrings. This robustness improves classification accuracy on noisy text.

5.1 Metrics for Tokenization Quality

5.1.1 Compression Ratio and Vocabulary Efficiency

Compression ratio is the average number of subword tokens per word (or per character) in a corpus. A lower ratio indicates a more efficient tokenization (fewer tokens to process). Vocabulary efficiency measures how many tokens are needed to cover a given percentage of the corpus (e.g., coverage of the top 10k tokens). Good tokenization balances high coverage with a compact vocabulary, minimizing both tokens per sentence and total vocabulary size.

5.1.2 Downstream Task Performance

The ultimate evaluation of a tokenizer is its impact on downstream task metrics such as translation BLEU score, language model perplexity, or classification F1. Controlling for model architecture and training data, tokenization variants can be compared. Several studies have reported that BPE and Unigram achieve comparable results, while WordPiece often yields slightly better results on tasks requiring linguistic precision.

5.2 Common Benchmarks and Datasets

5.2.1 Cross‑lingual and Monolingual Corpora

Subword tokenizers are typically trained on large monolingual corpora (e.g., English Wikipedia, Common Crawl) or multilingual corpora (e.g., CCAligned, OPUS). Cross-lingual benchmarks like XNLI, PAWS-X, and FLORES‑200 evaluate translation and classification quality across languages. Subword tokenization is a crucial preprocessing step in these benchmarks.

5.2.2 Subword‑specific Stress Tests (e.g., Rare and Misspelled Words)

Some benchmarks are designed to probe tokenization weaknesses. For example, the Subword Robustness Evaluation (SRE) dataset includes rare words, morphologically complex forms, and common misspellings. The ability of a tokenizer to preserve meaning in these cases is measured by downstream accuracy. Other stress tests include character-level noise (random insertions/deletions) and adversarial misspellings.

5.3 Challenges and Limitations

5.3.1 Subword Boundary Ambiguity

For certain algorithms (like Unigram), multiple segmentations are possible. Even deterministic tokenizers like BPE may produce counterintuitive splits (e.g., “bedroom” → “be##droom”) that do not align with linguistic morphemes. This ambiguity can lead to inconsistent representations across different contexts or tokenizers, affecting model reproducibility.

5.3.2 Computational Overhead in Inference

Subword tokenization adds an extra preprocessing step (tokenization) to the inference pipeline. For very large vocabularies (e.g., >100k tokens) or long sequences, the tokenizer can become a bottleneck. Some approaches cache tokenization results, but dynamic tokenization (like SentencePiece) may be slower in software compared to simple word-splitting. Additionally, GPUs process fixed-length sequences, so variable-length subword sequences require padding or bucketing.

6.1 Adaptive and Context‑sensitive Subword Segmentation

Current tokenizers apply a fixed segmentation rule regardless of context. Future work may develop models that learn to segment differently depending on the surrounding text—for example, treating “bank” as a single token in finance but splitting “bank” + “er” in a morphological context. End-to-end segmenters that jointly learn tokenization and the downstream model could reduce information loss and allow the model to choose the optimal granularity.

6.2 Integration with Multimodal and Code‑switched Data

As models incorporate images, audio, and mixed-language text, subword tokenization must adapt. For code-switching (e.g., English-Hindi mixed sentences), a shared subword vocabulary across scripts is beneficial. Multimodal tokenizers might combine visual tokens with subword tokens, or use a unified byte-level vocabulary to handle any data type. Research on “tokenization-free” models (e.g., ByT5, which operates on raw bytes) suggests an alternative direction.

6.3 Ethical and Practical Considerations

6.3.1 Subword Bias in Low‑resource Languages

Most subword tokenizers are trained on large, high-resource language corpora (e.g., English, Chinese). When applied to low-resource languages, the vocabulary often contains insufficient coverage, leading to many unknown subword splits and degraded model performance. Efforts such as multilingual tokenization (e.g., XLM‑R’s SentencePiece) and language-specific boosting (oversampling low-resource data during tokenizer training) aim to mitigate this bias, but disparities remain.

6.3.2 Transparency and Reproducibility

Tokenization settings (vocabulary size, merge rules, normalization) significantly affect model behavior. However, many models do not report the exact tokenization configuration, making it difficult to reproduce results. The field is moving toward standardizing tokenization libraries (e.g., Hugging Face Tokenizers, Google’s SentencePiece) and encouraging researchers to release tokenizer checkpoints alongside model weights. Reproducibility also requires careful documentation of subword preprocessing steps, especially in multilingual and low-resource contexts.