TF‑IDF (Term Frequency–Inverse Document Frequency) is a numerical statistic used in information retrieval and text mining to evaluate the importance of a word in a document relative to a collection of documents (corpus). It combines two components: term frequency (TF), which measures how often a term appears in a specific document, and inverse document frequency (IDF), which scores how rare or common the term is across the entire corpus. The product of TF and IDF yields a weight that increases with the term’s frequency in the document but is offset by its frequency across the corpus, thus filtering out common words while highlighting terms that are more distinctive.
1 Introduction
1.1 Historical background
The concept underlying TF‑IDF emerged in the early 1970s as part of the development of automated information retrieval systems. The term “inverse document frequency” was first formally introduced by Karen Spärck Jones in a 1972 paper. Earlier work by Hans Peter Luhn in the 1950s had proposed the use of term frequency for weighting, but it was Spärck Jones who recognized that terms appearing in many documents should be downweighted because they are less discriminating. The first combined TF‑IDF weighting scheme appeared in the 1970s within the SMART retrieval system developed at Cornell University.
1.2 Motivation in information retrieval
In text retrieval, the goal is to rank documents in response to a user query. Simple term frequency alone would give high scores to common words (e.g., “the”, “and”) that appear in almost every document. TF‑IDF addresses this by penalizing terms that are frequent throughout the corpus, so that matches on rare but meaningful terms contribute more. This allows the system to return documents that are both relevant (containing the query terms) and distinctive (containing terms that are relatively unique to them).
2 Mathematical formulation
2.1 Term frequency (TF)
Term frequency measures how many times a term \( t \) occurs in a document \( d \). Several variants exist.
2.1.1 Raw count
The simplest definition is the raw count: \[ \text{tf}(t, d) = f_{t,d} \] where \( f_{t,d} \) is the number of occurrences of term \( t \) in document \( d \).
2.1.2 Logarithmically scaled frequency
To reduce the impact of very frequent terms, a logarithmic scale is often used: \[ \text{tf}(t, d) = \begin{cases} 1 + \log_{10}(f_{t,d}) & \text{if } f_{t,d} > 0 \\ 0 & \text{otherwise} \end{cases} \]
2.1.3 Augmented frequency (normalized)
Augmented frequency normalizes by the maximum term frequency in the document: \[ \text{tf}(t, d) = 0.5 + 0.5 \cdot \frac{f_{t,d}}{\max \{ f_{w,d} : w \in d \}} \] This ensures the value ranges between 0.5 and 1 for terms that appear.
2.2 Inverse document frequency (IDF)
IDF measures how rare a term is across the corpus. Let \( N \) be the total number of documents, and \( \text{df}(t) \) be the number of documents containing term \( t \).
2.2.1 Standard IDF formula
The classic IDF is: \[ \text{idf}(t) = \log \frac{N}{\text{df}(t)} \] This gives a high value for terms that appear in few documents and a low (or zero) value for terms that appear in many.
2.2.2 Smoothed IDF variants
To avoid division by zero when a term is not in the corpus, a smoothing factor is added: \[ \text{idf}(t) = \log \left( \frac{N}{1 + \text{df}(t)} \right) + 1 \] or equivalently \( \log \frac{1 + N}{1 + \text{df}(t)} \). Another common variant adds 1 to the denominator: \( \log \frac{N}{1 + \text{df}(t)} \).
2.3 TF‑IDF weight calculation
The TF‑IDF weight for term \( t \) in document \( d \) is the product: \[ \text{tfidf}(t, d) = \text{tf}(t, d) \times \text{idf}(t) \] Different combinations of TF and IDF definitions yield different weighting schemes.
3 Variants and normalizations
3.1 TF weighting schemes
3.1.1 Binary TF
Binary TF assigns a value of 1 if the term appears in the document (regardless of frequency) and 0 otherwise. It is used when occurrence alone matters, not the count.
3.1.2 Sublinear TF
Sublinear TF uses a logarithm or square root to compress the range of raw frequencies. The logarithmic scaling (Section 2.1.2) is a typical example.
3.2 IDF weighting schemes
3.2.1 Probabilistic IDF
Probabilistic IDF derives from a probabilistic model of relevance: \[ \text{idf}(t) = \log \frac{N - \text{df}(t)}{\text{df}(t)} \] This is used in some retrieval models such as the Robertson–Spärck Jones weighting.
3.2.2 Smooth IDF
Smooth IDF adds a constant to the denominator or numerator to avoid extreme values when \( \text{df}(t) \) is very small. The smoothed forms in Section 2.2.2 are common.
3.3 Normalized TF‑IDF
Normalization adjusts the weights across a document so that longer documents do not automatically get higher scores.
3.3.1 Euclidean length normalization
Each TF‑IDF vector (over all terms in the document) is divided by its Euclidean norm: \[ \mathbf{v}_d = \frac{(\text{tfidf}(t_1,d), \ldots, \text{tfidf}(t_n,d))}{\sqrt{\sum_{i=1}^n \text{tfidf}(t_i,d)^2}} \] This ensures all document vectors have length 1.
3.3.2 Maximum frequency normalization
The TF‑IDF weights are divided by the maximum weight of any term in the document: \[ \text{tfidf}_{\text{norm}}(t,d) = \frac{\text{tfidf}(t,d)}{\max_{w \in d} \text{tfidf}(w,d)} \] This scales values to the range [0,1].
4 Applications
4.1 Information retrieval
4.1.1 Document ranking
In a search engine, TF‑IDF is used to score documents given a query. For each query term, its TF‑IDF weight in the document is summed (or averaged), and documents are ranked by this score.
4.1.2 Query matching
TF‑IDF can also represent the query itself as a vector of weights, and the similarity between query and document vectors is computed (e.g., via cosine similarity). This is the basis of the vector space model.
4.2 Text mining
4.2.1 Keyword extraction
By computing TF‑IDF for each term in a single document relative to a general corpus, the highest-weighted terms are often good keywords that characterize the document’s content.
4.2.2 Text summarization
TF‑IDF weights can help identify the most important sentences in a document. Sentences containing high-weight terms are considered more informative and are candidates for inclusion in an extractive summary.
5 Limitations
5.1 Lack of semantic context
TF‑IDF treats words as independent tokens. It does not capture word order, synonyms, or polysemy. For example, “bank” (financial institution) and “bank” (river bank) have the same weight, and synonyms like “car” and “automobile” are treated as entirely different terms.
5.2 Sensitivity to corpus composition
IDF depends on the choice of corpus. If the corpus is small or biased, the IDF values may not reflect true rarity. Additionally, terms that are rare in one corpus but common in general language can be overvalued.
6 Extensions and alternatives
6.1 BM25 (Okapi BM25)
BM25 is a probabilistic retrieval function that extends TF‑IDF by incorporating document length normalization and saturation of term frequency. It is widely used in modern search engines and has been shown to outperform classic TF‑IDF on many benchmarks.
6.2 TF‑IDF with word embeddings
Recent work combines TF‑IDF with dense vector representations (e.g., Word2Vec, GloVe). Instead of using raw term weights, the weighted average of word embeddings (where the weight is the TF‑IDF value) produces a document embedding that captures some semantic similarity.
6.3 Pointwise Mutual Information (PMI)
PMI measures the association between two terms based on their co-occurrence in a corpus. For feature selection, PMI can be used in place of IDF to weight terms by how strongly they correlate with a particular category, though it is less common for general retrieval than TF‑IDF.