Word2Vec编辑历史

Word2Vec is a group of shallow, two-layer neural network models developed by a team led by Tomas Mikolov at Google in 2013 to produce word embeddings—dense vector representations of words that capture semantic and syntactic relationships from large text corpora. By learning from local word co-occurrence patterns, Word2Vec enables tasks such as analogy reasoning, clustering, and feature engineering in natural language processing (NLP). Its two main architectures, Continuous Bag-of-Words (CBOW) and Skip-gram, along with efficient training techniques like negative sampling and hierarchical softmax, have made it a foundational tool in modern NLP.

## 1 Background

### 1.1 Neural language models

Before Word2Vec, neural language models (e.g., those by Bengio et al., 2003) used feedforward networks with a hidden layer to predict the next word given a fixed history. These models learned distributed representations for words as a byproduct, but training was computationally expensive because the output layer required a softmax over the entire vocabulary. Word2Vec distilled the embedding learning task into a simpler, highly scalable objective.

### 1.2 Distributional semantics and distributed representations

The theoretical foundation of Word2Vec is the distributional hypothesis—words that appear in similar contexts have similar meanings. Distributed representations (dense vectors) encode this similarity in a low-dimensional continuous space, in contrast to sparse one‑hot vectors. Word2Vec operationalizes this hypothesis by training on local co‑occurrence windows, producing embeddings that reflect both semantic and syntactic patterns.

## 2 Architectures

### 2.1 Continuous Bag-of-Words (CBOW)

CBOW predicts a target word given its surrounding context words within a fixed window. The context words are averaged (or summed) to form a single vector, which is then projected through a weight matrix to produce a probability distribution over the vocabulary. CBOW is faster