A co‑occurrence window (also called a context window or span) is a fixed number of tokens around a target word within a text, defining the immediate linguistic environment used to compute co‑occurrence statistics. These statistics are foundational in distributional semantics, collocation analysis, and word‑embedding models. The window’s size—commonly symmetric (e.g., ±5 words) or asymmetric (e.g., only preceding words)—strongly affects whether the captured relationships are syntactic, semantic, or topical. The concept is central to techniques such as latent semantic analysis, skip‑gram models, and pointwise mutual information calculations.
A co‑occurrence window specifies a contiguous sequence of tokens that are considered to co‑occur with a given target token. For each occurrence of the target in a corpus, the tokens inside the window are recorded as its context. The collection of these contexts across all occurrences forms the basis for building co‑occurrence matrices or training predictive models.
1.1 Token-level context vs. sentence-level context
Token‑level context operates at the granularity of individual words or punctuation, whereas sentence‑level context treats the entire sentence as the boundary (and sometimes as a single unit). Token‑level windows provide finer control over distance and direction, while sentence‑level windows are simpler but collapse all items within a sentence into one bag without regard to distance.
1.2 Symmetric and asymmetric windows
A symmetric window includes an equal number of tokens before and after the target, e.g., ±2 means two tokens left and two tokens right. An asymmetric window includes tokens only from one side—left‑only or right‑only—or uses unequal lengths (e.g., 3 left and 5 right). Asymmetric windows are used when directionality matters, such as in language modeling where only preceding tokens are available.
1.3 Window size (span length)
The span length is the total number of tokens included in the window. It is a critical hyperparameter that determines the type of information extracted.
1.3.1 Narrow windows (1–3 tokens)
Narrow windows capture tight syntactic relationships, such as adjective‑noun, verb‑object, or determiner‑noun combinations. They are effective for extracting collocations and for modeling morphological or syntactic behavior because nearby words often have strong grammatical links.
1.3.2 Wide windows (5–10 tokens or more)
Wide windows include more distant context, which tends to reflect topical or thematic associations (e.g., “climate” with “change”, “global”, “temperature”). Such windows are common in topic modeling and distributional semantic models that aim to capture broad semantic similarity.
1.3.3 Adaptive windows
Adaptive windows adjust their size dynamically based on properties of the text, such as sentence length, punctuation, or part‑of‑speech. For instance, a window might expand until a clause boundary is reached or shrink near rare words to avoid noise. Adaptive strategies aim to balance syntactic and semantic coverage without a fixed size.
2.1 Distributional hypothesis
The distributional hypothesis, famously attributed to Firth (1957), states that words that occur in similar contexts tend to have similar meanings. The co‑occurrence window operationalizes “context” into a measurable span, directly linking observed distributions to semantic similarity.
2.2 Co‑occurrence matrix and term‑document analogy
A co‑occurrence matrix records how often each word appears within the window of every other word. This structure is analogous to a term‑document matrix: instead of documents, the columns (or rows) represent context words, and each cell counts co‑occurrences. The matrix is the raw material for count‑based vector space models.
2.3 Role in vector space models
2.3.1 Count‑based models (e.g., bag‑of‑words, TF‑IDF)
In count‑based models, the window defines which words are counted as “co‑occurring” for each target. The resulting vectors are used directly (e.g., in bag‑of‑words) or weighted (e.g., TF‑IDF). The choice of window size determines the sparsity and semantic granularity of these vectors.
2.3.2 Prediction‑based models (e.g., word2vec, GloVe)
Prediction‑based models learn dense embeddings by predicting context words from a target (skip‑gram) or a target from context words (CBOW). The window size here sets the number of context samples per occurrence; smaller windows emphasize syntactic relations, larger windows capture topical relations. GloVe explicitly uses a weighted co‑occurrence matrix built from the window.
3.1 Directionality
3.1.1 Left‑only, right‑only, and bidirectional windows
Left‑only windows (e.g., for language model training) restrict context to preceding tokens, while right‑only windows consider only following tokens. Bidirectional windows combine both sides and are typical for symmetric semantics. The choice reflects the task: left‑only is natural for text generation, bidirectional for synonym detection.
3.1.2 Positional weighting (distance decay)
Many models assign lower weight to tokens far from the target. Common weighting schemes include linear decay, Gaussian decay, or reciprocal distance. This reduces the influence of distant, less relevant words and improves the quality of embeddings for local patterns.
3.2 Boundary considerations
3.2.1 Sentence boundaries
Often, windows are truncated at sentence boundaries to avoid mixing unrelated clauses. Crossing sentence boundaries can introduce noise because words from different propositions are not semantically related. Some models allow windows to extend across sentences only when the target is near the start or end of a sentence.
3.2.2 Punctuation and stop‑word filtering
Punctuation characters are sometimes excluded from the window because they are not content words; however, they can carry syntactic information (e.g., commas signaling apposition). Stop‑word removal is common in wide windows to avoid over‑frequent tokens that dilute signal. These decisions affect both the count statistics and the resulting semantic space.
3.3 Impact on collocation and association measures
3.3.1 Pointwise mutual information (PMI)
PMI measures the strength of association between two words: PMI(w1, w2) = log( P(w1,w2) / (P(w1)P(w2)) ). The co‑occurrence probability P(w1,w2) is computed from windows. Narrow windows yield higher PMI for tightly bound collocations (e.g., “strong tea”), while wide windows produce lower PMI for accidental co‑occurrence.
3.3.2 Log‑likelihood and chi‑square tests
These statistical tests compare observed co‑occurrence counts within the window against expected counts under independence. Both depend on the window size: larger windows increase total counts but may dilute the signal for genuine associations. The window effectively defines the observation unit for contingency tables.
4.1 Collocation extraction and phraseology
Collocation extraction relies on co‑occurrence windows to identify non‑compositional phrases (e.g., “kick the bucket”). Statistical measures (PMI, log‑likelihood) are computed over windows of typical size 2–5 to capture multi‑word units.
4.1.1 Semantic prosody
Semantic prosody refers to the habitual collocates of a word that convey a positive, negative, or neutral connotation (e.g., “cause” co‑occurs with negative words like “problem”). Wide windows help reveal such prosodic patterns because they capture the broader evaluative context.
4.1.2 Automatic term recognition
For extracting domain‑specific terms, windows of size 2–4 are used to find noun‑noun compounds and adjective‑noun pairs that recur frequently in a corpus. The window ensures that only adjacent or near‑adjacent words are considered, filtering out accidental co‑occurrence.
4.2 Word sense induction and disambiguation
Different senses of a polysemous word often appear with distinct sets of collocates. By using multiple window sizes or adaptive windows, sense‑specific contexts can be clustered. For example, “bank” near “river” vs. “bank” near “money” is distinguished by different local contexts captured by a small window.
4.3 Discourse and text genre analysis
4.3.1 Variation across registers
Different registers (e.g., news vs. fiction) exhibit different co‑occurrence patterns. Wide windows are used to study topic‑driven collocations, while narrow windows reveal grammatical preferences. Comparing window‑based statistics across registers helps quantify stylistic differences.
4.3.2 Author‑specific lexical patterns
An author’s distinctive collocational style can be captured by comparing co‑occurrence windows across texts. Narrow windows highlight unique syntactic choices (e.g., preferred verb‑object pairs), contributing to authorship attribution studies.
5.1 Selection of optimal window size
There is no universal optimal window size. The choice depends on the linguistic level of interest (syntax vs. topic) and the sparsity of the corpus. A single fixed size often fails to capture both local and global patterns simultaneously, and grid search may be prohibitively expensive for large vocabularies.
5.2 Sparse data and rare words
Rare words have few occurrences, so even a moderately wide window can lead to zero co‑occurrence counts for many pairs. This sparseness degrades the reliability of association measures and embeddings. Smoothing techniques or back‑off strategies are often required.
5.3 Sensitivity to corpus composition
The composition of the corpus (genre, topic distribution, size) heavily influences co‑occurrence statistics. A window optimized for a balanced corpus may perform poorly on a domain‑specific one. This sensitivity means that results often do not transfer well across different text collections.
6.1 N‑gram and skip‑gram
An n‑gram is a contiguous sequence of n tokens; it can be viewed as a fixed‑size window where the target is not distinguished. A skip‑gram extends the n‑gram by allowing gaps (skips) within the window, providing a way to capture non‑contiguous co‑occurrence patterns. The co‑occurrence window defines which tokens are allowed as skips.
6.2 Context bag and bag‑of‑words
A context bag is a set of tokens from a window (or sentence) without order or distance information. The bag‑of‑words model treats a document as a bag of tokens; similarly, a context bag treats a window as a bag. Both discard positional information, a simplification that is sometimes desirable for topic models.
6.3 Neighborhood (graph‑based co‑occurrence)
In graph‑based models, words are nodes, and edges are weighted by co‑occurrence counts within a window. The neighborhood of a node is the set of words that appear within a certain number of steps in the graph, often derived from window‑based edges. This perspective connects window‑based co‑occurrence to network analysis and random‑walk embeddings like DeepWalk.