Overview
Skip-gram is a neural network-based word embedding model introduced by Mikolov et al. in 2013 as part of the Word2Vec framework. It learns dense vector representations (embeddings) of words by predicting the surrounding context words given a target word in a sliding window over text. Unlike its counterpart continuous bag-of-words (CBOW), Skip-gram performs well on rare words and is widely used in natural language processing tasks such as semantic similarity, analogy reasoning, and as input features for downstream models.
1 Definition and motivation
1.1 Purpose of word embeddings
Word embeddings transform discrete words into continuous, low-dimensional vectors that capture semantic and syntactic relationships. Traditional one-hot encoding treats each word as an independent symbol, failing to capture similarity between words. Word embeddings address this by placing semantically similar words close together in the vector space, enabling generalization across contexts. The Skip-gram model was motivated by the need for efficient, high-quality embeddings that could be trained on large corpora.
1.2 Distinction from CBOW
The continuous bag-of-words (CBOW) model predicts a target word given its surrounding context words, averaging the context vectors. Skip-gram inverts this task: it predicts the context words given a single target word. This inversion makes Skip-gram more computationally expensive per training example but empirically yields better representations for rare words, as each rare target word is used to predict multiple contexts, providing more training signal.
2 Model architecture
2.1 Input and output layers
The input layer receives a one-hot encoded vector representing the target word of vocabulary size \(V\). The output layer produces a probability distribution over \(V\) words, representing the likelihood of each word appearing in the context. In practice, the output layer can be implemented as a softmax layer or approximated using hierarchical softmax or negative sampling.
2.2 Hidden layer and embedding matrix
The hidden layer is a linear projection without activation. It consists of \(N\) neurons, where \(N\) is the embedding dimension. The weight matrix between the input and hidden layer, of size \(V \times N\), serves as the embedding matrix: each row corresponds to the dense vector of a word. The output weight matrix (size \(N \times V\)) maps hidden representations to output scores.
2.3 Objective function
Given a target word \(w_t\) and a context word \(w_{t+j}\) within a window of size \(m\), the objective is to maximize the log probability of the context given the target:
\[
| \frac{1}{T} \sum_{t=1}^{T} \sum_{-m \le j \le m, j \neq 0} \log p(w_{t+j} | w_t) |
|---|
\]
| where \(T\) is the total number of words in the corpus. The probability \(p(w_O | w_I)\) is modeled by the softmax function over all vocabulary words. |
|---|
3 Training algorithms
3.1 Negative sampling
Negative sampling (NEG) approximates the softmax by treating the problem as a binary classification task. For each positive context–target pair, the model samples \(k\) negative words (non-context words) from a noise distribution and updates the weights to distinguish the true context from negative samples.
3.1.1 Noise distribution
The noise distribution \(P_n(w)\) is typically the unigram distribution raised to the power of 3/4, i.e., \(P_n(w) \propto freq(w)^{3/4}\). This choice gives slightly more weight to rare words compared to the raw frequency, improving the quality of embeddings.
3.1.2 Loss function with negative sampling
The loss for one target–context pair \((w_I, w_O)\) with \(k\) negative samples \(w_{neg}\) is:
\[ L = - \log \sigma(v'_{w_O}^\top v_{w_I}) - \sum_{i=1}^k \log \sigma(- v'_{w_{neg,i}}^\top v_{w_I}) \]
where \(v_{w_I}\) is the input embedding, \(v'_w\) is the output embedding, and \(\sigma\) is the sigmoid function. This loss maximizes the probability of the positive pair while minimizing the probability of negative pairs.
3.2 Hierarchical softmax
| Hierarchical softmax (HS) replaces the flat softmax with a binary tree, reducing the computational complexity from \(O( | V | )\) to \(O(\log | V | )\). Each word corresponds to a leaf node in the tree, and the probability of a word given the context is computed as the product of probabilities along the path from root to leaf. |
|---|
3.2.1 Huffman tree construction
A Huffman tree is built based on word frequencies, with more frequent words having shorter paths. This reduces the average number of computations during training, as common words are reached with fewer node evaluations.
3.2.2 Binary tree probability computation
At each internal node \(n\), the probability of taking the left or right branch is given by a sigmoid over the dot product of the context vector and a node-specific parameter. The probability of a word \(w\) is:
\[
| p(w | w_I) = \prod_{j=2}^{L(w)} \sigma([\text{attr}] \cdot v'_{\text{path}(w,j)}^\top v_{w_I}) |
|---|
\]
where \([\text{attr}]\) is 1 if the next node is the left child and -1 otherwise, and \(L(w)\) is the length of the path.
3.3 Subsampling of frequent words
To balance the influence of very frequent words (e.g., "the", "and"), each word \(w_i\) is discarded during training with probability:
\[ P(\text{discard}) = 1 - \sqrt{\frac{t}{f(w_i)}} \]
where \(f(w_i)\) is the word frequency and \(t\) is a threshold (typically \(10^{-5}\)). This reduces training time and improves the quality of embeddings by focusing on more discriminative word pairs.
4 Mathematical formulation
4.1 Softmax probability for context prediction
The probability that a context word \(w_O\) appears given a target word \(w_I\) is:
\[
| p(w_O | w_I) = \frac{\exp(v'_{w_O}^\top v_{w_I})}{\sum_{w=1}^{V} \exp(v'_w^\top v_{w_I})} |
|---|
\]
where \(v_{w_I}\) is the input embedding (from the hidden layer) and \(v'_w\) is the output embedding.
4.2 Gradient derivation
The gradient of the log-likelihood with respect to the output vector \(v'_w\) for a correct context \(w_O\) is:
\[
| \frac{\partial \log p(w_O | w_I)}{\partial v'_w} = |
|---|
\begin{cases}
| v_{w_I} (1 - p(w_O | w_I)) & \text{if } w = w_O \\ |
|---|---|
| -v_{w_I} \, p(w | w_I) & \text{otherwise} |
\end{cases} \]
For the input vector \(v_{w_I}\), the gradient is the sum over all output words of similar terms. These gradients drive the stochastic gradient descent updates.
4.3 Stochastic gradient descent updates
Model parameters are updated after each training pair using stochastic gradient descent (SGD). For a given learning rate \(\eta\):
\[ v'_{w}^{(new)} = v'_{w}^{(old)} + \eta \cdot \text{gradient} \]
\[ v_{w_I}^{(new)} = v_{w_I}^{(old)} + \eta \cdot \sum_{w} \text{gradient}_{w} \]
In practice, updates are performed asynchronously across multiple threads to expedite training on large corpora.
5 Implementation details
5.1 Window size and dynamic context
The context window size \(m\) is typically set to 5–10 words on each side. To vary the influence of distant words, the window size is randomly sampled from 1 to \(m\) for each training example, giving closer words more weight. The dynamic context also acts as a regularizer.
5.2 Learning rate schedule
Training begins with an initial learning rate (e.g., 0.025) that linearly decreases to zero over the course of training. This schedule helps the model converge by taking large steps early and fine-tuning later.
5.3 Multi-threading and asynchronous updates
Word2Vec implementations use multi-threading with shared parameter storage. Each thread processes a portion of the corpus independently, and parameter updates are applied asynchronously without locking. Despite potential race conditions, this approach yields high efficiency and good convergence in practice.
6 Evaluation and benchmarks
6.1 Word similarity tasks (e.g., WordSim-353)
Skip-gram embeddings are evaluated by computing cosine similarity between word vectors and comparing the ranking to human judgments. WordSim-353 contains 353 word pairs with similarity scores. The Spearman correlation between model outputs and human ratings serves as a metric; Skip-gram typically achieves correlations around 0.5–0.7.
6.2 Analogy tasks (e.g., Google Analogy Test Set)
The Google Analogy Test Set contains 19,544 analogies of the form “a is to b as c is to ?” (e.g., “king : queen :: man : ?”). The model solves analogies by vector arithmetic: \(v_b - v_a + v_c\) and finding the nearest neighbor. Skip-gram with negative sampling achieves accuracy of around 70–75% on this test set.
6.3 Comparison with other embedding methods
Compared to CBOW, Skip-gram yields higher accuracy on analogy and rare word tasks but is slower to train. Compared to GloVe, which uses global matrix factorization, Skip-gram often performs similarly but is more flexible for incremental updates. Subword-enhanced variants like FastText improve performance on morphologically rich languages.
7 Variants and extensions
7.1 Skip-gram with subword information (FastText)
FastText extends Skip-gram by representing each word as a bag of character n-grams. For example, the word “where” contains n-grams “wh”, “whe”, “her”, “ere”, “re”. The final word embedding is the sum of its n-gram vectors. This allows the model to handle out-of-vocabulary words and capture morphological similarities.
7.2 Skip-gram with negative sampling (SGNS)
Skip-gram with negative sampling (SGNS) is the most popular variant of the original model. It combines the Skip-gram architecture with the negative sampling loss, offering a good balance between efficiency and embedding quality. SGNS is often synonymous with “Word2Vec Skip-gram”.
7.3 Dependency-based Skip-gram
Instead of using a linear context window, dependency-based Skip-gram defines context words based on syntactic relations (e.g., subject, object). For each target word, the context consists of words linked via dependency parse edges. This produces embeddings that are more sensitive to syntactic similarity than to topical similarity.
8 Limitations
8.1 Lack of polysemy handling
Skip-gram learns a single static vector per word, which cannot distinguish multiple meanings (e.g., “bank” as financial institution vs. river bank). Contextualized models like BERT address this limitation by generating dynamic embeddings conditioned on the surrounding text.
8.2 Out-of-vocabulary words
The standard Skip-gram model assigns no vector to words not seen during training. This is problematic for domain-specific terms or spelling variations. Subword approaches (e.g., FastText) mitigate this issue.
8.3 Bias in learned representations
Word embeddings trained on large corpora often inherit societal biases present in the training text, such as gender or racial stereotypes. For example, analogy “doctor : male :: nurse : female” reflects a biased association. These biases can propagate to downstream applications and require debiasing techniques.
9 Applications
9.1 Text classification
Pre-trained Skip-gram embeddings serve as input features for classifiers such as logistic regression, SVMs, or recurrent neural networks. They improve performance over bag-of-words representations by capturing semantic similarity, especially when labeled data is limited.
9.2 Machine translation
In early neural machine translation systems, Skip-gram embeddings were used to initialize the source and target language embeddings. This provided a warm-start for the encoder–decoder models, reducing training time and improving translation quality for low-resource language pairs.
9.3 Information retrieval
Word embeddings enable semantic search where queries are matched to documents based on vector similarity, rather than exact keyword matches. Skip-gram embeddings help retrieve conceptually related documents even when the vocabulary does not overlap.
10 History and influence
10.1 Origin in Word2Vec
Skip-gram was introduced in the paper “Efficient Estimation of Word Representations in Vector Space” by Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean (2013). It was released alongside the CBOW model in the Word2Vec toolkit, which provided highly optimized C code for training on billions of words. The release of pre-trained vectors trained on Google News (100 billion words) popularized the method.
10.2 Impact on deep learning for NLP
Skip-gram and Word2Vec revitalized the field of distributional semantics and became a foundational technique in deep learning for NLP. They inspired subsequent work on paragraph vectors (Doc2Vec), subword embeddings (FastText), and contextualized representations (ELMo, BERT). The concept of predicting context from target (or vice versa) also influenced later models in graph embedding and recommendation systems.