1 Definition and role in information retrieval

Ground truth relevance relevance judgments used to evaluate information retrieval (IR) systems. In practice, it is the set of labeled query–document pairs (or graded scores) that an authoritative process produces so that system outputs can be compared against an agreed reference. These judgments are treated as the closest available approximation to the user’s intended notion of relevance for the queries in an evaluation task.

1.1 What “ground truth” means in IR evaluation

In IR evaluation, “ground truth” does not imply perfect truth about real-world usefulness. Instead, it denotes the reference labels that evaluators designate as correct for the specific test collection and labeling instructions. The term emphasizes that metrics are computed relative to a consistent set of judgments, allowing different retrieval methods to be compared on the same benchmark.

1.2 Relevance judgments vs. system scores

Retrieval systems typically produce ranked lists with internal scores (e.g., similarity, relevance estimates, or learned ranking scores). These numbers are not identical to relevance labels. Ground truth relevance provides an external evaluation criterion: whether a document should be considered relevant to a query according to the assessment process. Evaluation metrics combine the ordering produced by system scores with the correctness indicated by the ground truth labels.

1.3 Binary vs. graded relevance labels

Ground truth relevance can be encoded as binary labels (relevant vs. not relevant) or as graded levels (for example, not relevant, somewhat relevant, and highly relevant). Graded labels are common because relevance can have degrees. When graded labels are available, evaluation metrics such as nDCG can reward systems for placing more useful results higher in the ranking rather than treating all relevant documents as equivalent.

2 Creating ground truth relevance labels

Creating ground truth relevance labels involves selecting queries and documents, defining what relevance means, and performing human judgments (often with quality controls). The resulting label set is then used for evaluation and, in some workflows, for training machine learning models.

2.1 Assessor selection and training

Assessors are typically recruited for their ability to interpret documents in the context of a query and follow the relevance guidelines consistently.

2.1.1 Relevance judgment guidelines

Guidelines specify the criteria for deciding relevance at the intended granularity. They may include how to treat partial matches, topical overlap without direct usefulness, duplicates, navigation targets, or documents that answer only indirectly. For graded relevance, instructions often define the difference between adjacent grades so assessors can apply the scale consistently.

2.1.2 Calibration and inter-assessor consistency

Because judgment quality depends on consistency, organizers often include calibration steps. These may involve pilot labeling, feedback sessions, and joint review of disagreements. Statistical checks can also be used to identify assessors whose decisions diverge from the consensus, followed by additional training or re-labeling of affected items.

2.2 Query–document pooling strategies

Since labeling all possible documents for every query is usually infeasible, evaluation collections commonly use pooling. Pooling collects a candidate set by combining outputs from multiple retrieval systems, then labels documents from that combined set.

2.2.1 Constructing candidate sets for labeling

Candidate sets are formed by running several retrieval methods over the query set and taking the top results per method, often with a depth cutoff. The rationale is that relevant documents are likely to appear in at least one system’s top results, making them more likely to be included in the pool for labeling.

2.2.2 Bias reduction via diversified pools

Pooling can bias ground truth toward documents that current systems already retrieve well. Using diverse methods (e.g., different indexing, retrieval models, or query expansion strategies) reduces this bias by increasing coverage of different lexical and semantic patterns. Some workflows also vary pool depth or include additional retrieval methods to improve the chance that important documents enter the labeled candidate set.

2.3 Judgment granularity and documentation

How relevance is operationalized affects both metric behavior and the interpretability of results. Clear documentation ensures that future users understand what the labels do and do not represent.

2.3.1 Document-level relevance criteria

Document-level criteria specify the expected usefulness of an entire document for satisfying the query’s information need. This includes how assessors interpret relevance when a document contains relevant passages but not necessarily in a way that fully addresses the query, as well as how to treat near-duplicates or documents that share topical overlap while differing in actual utility.

2.3.2 Handling ambiguous or borderline cases

Ambiguity arises when relevance is subjective, when the query intent is underspecified, or when documents partially match. Guidelines may provide escalation procedures, additional reading context, or explicit rules for borderline cases. Some labeling efforts also allow for uncertainty categories, though these are less common when metrics assume a fixed set of relevance grades.

3 Evaluation metrics based on ground truth

Once ground truth labels exist, evaluation metrics measure how well a system’s ranked output aligns with those judgments. Different metrics emphasize different aspects such as early precision, ranking quality under graded relevance, or robustness across queries.

3.1 Rank-based effectiveness measures

Rank-based metrics evaluate performance as a function of the order produced by the system.

3.1.1 Precision and recall at k

Precision at k (P@k) counts how many of the top k retrieved documents are relevant, using ground truth labels. Recall at k (R@k) gauges how many relevant documents are captured within the top k portion relative to the total relevant set. These measures are simple and interpretable but may not fully reflect ranking quality beyond the cutoff.

3.1.2 Average Precision (AP) and MAP

Average Precision (AP) aggregates precision values computed at each point where a relevant document occurs within the ranked list. Mean Average Precision (MAP) averages AP across queries. AP and MAP reward systems that place relevant documents earlier, while still accounting for how many relevant items appear throughout the ranking.

3.1.3 nDCG and graded relevance use

Normalized discounted cumulative gain (nDCG) uses graded relevance to compute gain values and applies a discount factor that reduces the contribution of items retrieved lower in the ranking. Normalization compares a system’s nDCG against an ideal ranking for the same query, yielding a score in a comparable range across different queries. When only binary labels are available, nDCG can still be used, but graded judgments typically provide richer discrimination.

3.2 Set-based evaluation considerations

Set-based views treat retrieval results more like unordered sets rather than ordered rankings. Although classic IR metrics are often rank-sensitive, set-based aggregation can be useful for certain reporting practices.

3.2.1 Macro vs. micro aggregation across queries

When aggregating across a query set, macro-averaging gives equal weight to each query’s score, while micro-averaging pools counts across queries before computing a final value. Macro aggregation emphasizes fairness across different query difficulties; micro aggregation can be influenced by queries with larger numbers of relevant documents or larger label counts.

3.3 Ties, missing judgments, and normalization

Real evaluation data rarely matches ideal assumptions perfectly. Metrics must handle ranking ties, incomplete labeling, and the normalization choices that make scores comparable.

3.3.1 Treatment of unjudged documents

Ground truth is usually incomplete: only a subset of documents receives labels. Common evaluation practices treat unjudged documents as neither confirmed relevant nor confirmed non-relevant, or they apply specific heuristics such as considering unjudged items as non-relevant. Different policies affect metric estimates, especially when systems retrieve many items outside the judged pool.

3.3.2 Sensitivity to judgment incompleteness

If relevant documents are missing from the labeled set, metrics can underestimate a system’s true effectiveness. This is particularly problematic for recall-oriented measures and for systems that retrieve novel documents not covered by the pool. Sensitivity depends on pooling depth, the diversity of systems used to build the pool, and the distribution of relevance in the full document space.

4 Reliability and limitations

Ground truth relevance labels support evaluation, but they are subject to systematic and random errors. Understanding these limitations helps interpret results responsibly.

4.1 Incompleteness and unjudged documents

Pooling and labeling constraints imply that the absence of a label does not necessarily indicate non-relevance. As a result, evaluation scores reflect performance relative to the labeled subset, not necessarily relative to all documents in a collection. Incomplete coverage can lead to unfair comparisons when one system retrieves more of the unlabeled region.

4.2 Label noise and human subjectivity

Even with guidelines, assessors may interpret queries differently, perceive relevance differently, or vary in strictness. Label noise can degrade the reliability of evaluation outcomes and can mislead model training when labels are used as supervised targets.

4.2.1 Measuring inter-rater agreement

Inter-rater agreement statistics quantify how similarly assessors label the same items. These measures help detect inconsistent labeling practices and can guide targeted retraining. Agreement levels are commonly interpreted alongside the difficulty of the query set and the clarity of the guidelines.

4.2.2 Common sources of judgment variance

Variance can originate from unclear query intent, document ambiguity, differences in assessor background knowledge, or inconsistent handling of partial relevance. Systems can also influence variance indirectly because pools may overrepresent documents of a certain type, skewing what assessors encounter during labeling.

4.3 Domain transfer and context dependence

Ground truth relevance is context-specific: labels depend on the document collection, the query set, and the labeling policy. When the evaluation context changes—such as using a different corpus, updating query formulations, or changing the topical scope—the same relevance notions may not hold, reducing the generality of conclusions drawn from benchmark scores.

5 Practical workflow for using ground truth

Organizations and research groups use ground truth in repeatable workflows: building datasets, running systems, mapping outputs to labels, and reporting results with appropriate caveats.

5.1 Building an evaluation dataset

An evaluation dataset packages queries, relevance labels, and metadata needed for reproducible evaluation.

5.1.1 Query set selection and intent coverage

Query selection aims to cover different intents and difficulty levels so that evaluation reflects a range of user needs. Coverage can include informational, navigational-like, or exploratory intents depending on the benchmark goals. Well-designed query sets also include enough diversity in vocabulary and subject matter to avoid overfitting to a narrow topic distribution.

5.2 Running retrieval and mapping results to judgments

Systems are executed over the same index or collection configuration used when building the benchmark. Their ranked outputs are matched to labeled query–document pairs, and the metric computation uses those labels to evaluate effectiveness.

5.3 Reproducibility and versioning of labels

Because ground truth labels can be updated or re-labeled, datasets typically include version identifiers and documentation about changes. Reproducibility requires recording the label release used, pooling depth and construction details when available, and any processing steps that affect metric calculation.

5.4 Reporting results with uncertainty-aware practices

When labels are incomplete or noisy, reported scores can be treated as estimates rather than exact truths. Some workflows report variance across query subsets, include confidence intervals through resampling, or describe limitations related to unjudged documents. Clear reporting helps readers avoid overinterpreting small differences between systems.

6 Advanced topics

Beyond standard offline metrics, ground truth relevance can support more nuanced learning and evaluation strategies.

6.1 Preference data and pairwise relevance signals

Instead of labeling each document absolutely, some frameworks collect preference judgments between pairs of documents for a given query (e.g., which one is more relevant). Preference data can be converted into training signals for ranking models or used directly in specialized evaluation settings, though it requires additional labeling design.

6.2 Active learning for reducing labeling cost

Active learning uses model feedback to choose which query–document pairs to label next. The goal is to reduce labeling expense while improving performance. Ground truth still provides the supervision, but it is acquired selectively rather than exhaustively.

6.3 Continuous evaluation and online relevance updates

As information needs and corpora evolve, relevance may shift over time. Continuous evaluation approaches periodically refresh labels or update assessments for selected items, aiming to maintain alignment between the benchmark’s relevance notion and the current state of the content domain.

6.4 Learning-to-rank training targets derived from ground truth

Learning-to-rank models often require labeled relevance grades. Ground truth can be transformed into objectives such as pointwise, pairwise, or listwise loss functions. The reliability of these training targets depends on label quality and the degree to which the training query–document distribution matches the evaluation and deployment scenarios.

Ground truth relevance is closely tied to other evaluation artifacts and signals. Distinguishing these concepts clarifies how IR systems are assessed.

7.1 Gold labels, relevance assessments, and test collections

“Gold labels” refers to the authoritative judgments designated for evaluation. Relevance assessments describe the labeling process and outputs, while a test collection bundles queries, documents, and relevance labels into a reusable benchmark package.

7.2 Ideal ranking and upper-bound comparisons

An ideal ranking is the ordering that would maximize a chosen metric given the ground truth labels. Upper-bound comparisons use the ideal or best-possible scenario to quantify how far a system is from theoretical maximum, helping diagnose whether metric ceiling effects or label limitations constrain observed performance.

7.3 Click-based signals vs. ground truth relevance

Click data and other interaction logs provide implicit feedback about what users engage with, but clicks can reflect presentation bias, position effects, or exploratory behavior rather than explicit relevance. Ground truth relevance is typically treated as more direct for evaluation, while click-based signals are often used for training, weak supervision, or supplementary analysis.