FastText

1 Core Concepts

FastText is built on the principle that words can be decomposed into smaller subword units, enabling the model to capture morphological and orthographic patterns. It trains word embeddings using two classic architectures—skip‑gram and continuous bag of words—both adapted to incorporate character n‑gram information. This design allows FastText to handle rare and unseen words by composing their representations from learned subword vectors.

1.1 Subword (Character n‑gram) Representation

Instead of treating each word as an atomic token, FastText represents a word as a bag of character n‑grams. For example, the word "apple" is broken into all contiguous substrings of length 3 to 6 (e.g., "app", "ppl", "ple", "appl", "pple", "apple"), plus the whole word itself. Special boundary symbols < and > are added to distinguish prefixes and suffixes. The embedding for a word is the sum or average of the embeddings of its constituent n‑grams. This approach captures shared substrings across related words (e.g., "run", "running", "runner") and enables the model to generate vectors for out‑of‑vocabulary words by summing the n‑gram embeddings that exist in the model’s vocabulary.

1.2 Skip‑gram with Negative Sampling

FastText implements the skip‑gram model, where the goal is to predict a word’s context given the target word. In the subword variant, the target word’s representation is built from its character n‑grams, and the context words are treated as whole tokens (or also as n‑gram aggregates). Training uses negative sampling: for a correct context word, the model updates weights to increase the dot product between the target and context vectors; for a set of randomly selected negative words, it decreases the dot product. This is computationally efficient and scales to large vocabularies.

1.3 Continuous Bag of Words (CBOW)

The CBOW architecture predicts the target word given its surrounding context words. In its FastText implementation, the context is represented as the sum of the n‑gram vectors of each context word. The model then computes a softmax (or hierarchical softmax) over the vocabulary. CBOW trains faster than skip‑gram but generally gives lower performance on rare words and analogy tasks; FastText offers both options to suit different precision‑speed trade‑offs.

2 Model Architecture

FastText provides two main operating modes: one for learning unsupervised word vectors and one for supervised text classification. Both share the subword‑based representation.

2.1 Word Vector Training

In the unsupervised mode, the user provides a raw text corpus. FastText processes it to build a vocabulary and then trains either skip‑gram or CBOW with negative sampling or hierarchical softmax. The training algorithm iterates over the corpus, updating n‑gram embeddings to maximize the likelihood of observed word‑context pairs. The resulting word vectors can be used directly for similarity and analogy tasks.

2.2 Text Classifier (Supervised Learning)

FastText’s supervised mode treats text classification as a word‑embedding problem: each document is represented as the average of its word vectors (which are built from n‑grams). This averaged vector is fed into a linear classifier that outputs a probability distribution over labels. The model is trained to minimize the negative log‑likelihood of the true label.

2.2.1 Hierarchical Softmax for Large Label Sets

When the number of output classes is large (e.g., thousands or more), computing a full softmax over all labels becomes expensive. FastText uses a hierarchical softmax based on a Huffman tree built from label frequencies. Each leaf corresponds to one label; the probability of a label is the product of probabilities along the path from the root to that leaf. This reduces the per‑training‑step complexity from O(K) to O(log K), where K is the number of labels.

2.2.2 Loss Functions

FastText supports several loss functions for classification: softmax (with or without hierarchy) and negative sampling. The negative sampling loss approximates the softmax by treating each training example as a binary classification over a small set of sampled negative labels. This is useful when the label set is very large and fast training is desired. Additionally, the user can specify a margin‑based ranking loss, which tries to keep the true label’s score higher than that of randomly sampled negative labels by a margin.

3 Implementation Details

FastText is written in C++ for speed and memory efficiency. It provides flexible interfaces to suit different users and environments.

3.1 C++ Library and Python Bindings

The core library is implemented in C++ with minimal dependencies. It exposes an API for training, loading, and querying models. Python bindings are available via the fasttext package on PyPI, allowing users to call the same functions from Python. The Python wrapper mirrors the C++ functionality, supporting both the command‑line options and programmatic access through a FastText class.

3.2 Command‑line Interface

FastText includes a command‑line tool (fasttext) that can be used directly for common tasks: fasttext skipgram and fasttext cbow for unsupervised training, fasttext supervised for classifier training, and fasttext predict for inference. Arguments such as -minn, -maxn (minimum and maximum n‑gram length), -dim (vector dimension), -epoch, -lr (learning rate), and -epoch are set via command flags. This interface makes FastText easy to integrate into scripts and pipelines.

3.3 Pre‑trained Models

The project provides several pre‑trained models that can be downloaded and used directly for various tasks, saving users the time and resources of training from scratch.

3.3.1 Wikipedia & Common Crawl Vectors

Pre‑trained word vectors are available for multiple languages, trained on Wikipedia dumps and Common Crawl corpora. These vectors embed 300 dimensions and cover vocabularies of up to 2 million words. They include subword n‑gram information, so embeddings for out‑of‑vocabulary words can be computed on the fly. A separate set of vectors trained on Common Crawl (with 600B tokens) provides even broader coverage.

3.3.2 Language Identification Model

FastText ships a pre‑trained language identification model that can detect 176 languages. It is a supervised classifier trained on data from Wikipedia, Tatoeba, and SETimes. The model is compact (around 1 MB) and can classify short text snippets (e.g., a single sentence) with high accuracy. The fasttext predict command can be used to obtain the predicted language and a confidence score.

4 Applications

FastText’s combination of speed, subword sensitivity, and built‑in classification makes it suitable for a wide range of natural language processing tasks.

4.1 Text Classification

FastText is often used for document or sentence classification, including sentiment analysis, topic labeling, and spam detection. The supervised classifier trains quickly on large datasets (millions of documents) and achieves accuracy competitive with more complex deep learning models while being orders of magnitude faster. It is particularly effective when the input text contains many rare or misspelled words, thanks to its n‑gram representation.

4.2 Semantic Similarity & Analogy

The unsupervised word vectors capture semantic and syntactic relationships. Users can compute cosine similarity between word vectors or solve word analogies (e.g., “king – man + woman ≈ queen”). Because FastText embeddings incorporate subword information, they perform well on morphological analogies (e.g., “run : running : : walk : walking”) and can handle typos and infrequent forms.

4.3 Language Identification

The pre‑trained language identification model enables rapid detection of the language of a text snippet. This is used in preprocessing pipelines, content filtering, and multilingual search. The model’s small size allows it to be deployed on mobile devices and in low‑latency servers.

4.4 Named Entity Recognition (NER)

Although FastText is not designed as a sequence‑labeling model, its word vectors can serve as features for NER systems. By providing embeddings for words and their character n‑grams, FastText helps recognize entities that appear in different forms (e.g., “Microsoft,” “Microsoft’s,” “microsoft”) or that are rare in the training data. Many NLP pipelines incorporate FastText embeddings as input to a CRF or BiLSTM layer.

5 Performance and Benchmarks

FastText is benchmarked on several standard tasks, comparing its accuracy and training speed against other popular word‑embedding methods.

5.1 Comparison with Word2vec and GloVe

Word2vec (skip‑gram and CBOW) and GloVe are the two most common alternatives. FastText differs primarily by its subword representation.

5.1.1 Accuracy on Analogies

On word analogy datasets (e.g., the Google analogy test set), FastText often outperforms word2vec and GloVe on morphological analogies (e.g., verb tenses, plural forms) because the subword model captures shared affixes. On semantic analogies (e.g., country‑capital), performance is comparable; FastText may be slightly lower on purely semantic relations where subword cues are irrelevant. Overall, the gap narrows with larger training corpora.

5.1.2 Training Speed

FastText’s training is highly optimized. On a typical multicore CPU, it can process billions of tokens per hour. Compared to word2vec, FastText adds overhead due to n‑gram hash lookups, but this is mitigated by efficient hash tables and multithreading. For the classification mode, FastText trains up to several hundred times faster than deep learning classifiers such as CNNs or LSTM‑based models while achieving similar accuracy on many benchmarks.

6 Extensions and Variants

The subword idea has inspired several related models and expansions.

6.1 StarSpace (Related Framework)

StarSpace is another FAIR library that generalizes FastText’s embedding approach to a wider range of tasks, including learning embeddings for entities, documents, and graph nodes. It uses a unified similarity‑based loss and supports multi‑modal inputs. While FastText is focused on text, StarSpace can be applied to collaborative filtering, link prediction, and other non‑text domains.

6.2 Multilingual FastText

FastText supports training multilingual word vectors when given aligned or comparable corpora. Several research projects have extended the subword model to learn cross‑lingual embeddings by sharing n‑gram information between languages that share scripts or by using bilingual dictionaries for alignment. The output vectors can then be used for cross‑lingual transfer tasks such as zero‑shot classification.

7 Limitations

Despite its strengths, FastText has certain drawbacks that users should consider.

7.1 Context Insensitivity

FastText uses a bag‑of‑n‑grams representation and a fixed context window during unsupervised training. It does not capture word order beyond the local window, nor does it model long‑range dependencies or polysemy. A word like "bank" will have a single embedding that conflates its financial and river meanings. This limitation is shared with most simple word‑embedding models and is partially addressed by contextual models such as BERT and ELMo.

7.2 Memory Footprint for Large Vocabularies

Storing embeddings for all character n‑grams can be memory‑intensive. For a large vocabulary (e.g., 2 million words) with n‑grams of length 3–6, the number of n‑gram vectors may reach tens of millions. FastText uses a hash table to map n‑grams to a fixed number of buckets (e.g., 2 million), which saves memory but introduces collisions that can degrade embedding quality. Users with very large datasets or limited RAM may need to tune the -bucket parameter or choose a smaller model.