1 Introduction to Multiple Testing and Discovery

1.1 Why multiple hypothesis testing inflates false positives

When many statistical tests are performed at once, using a single significance threshold for each test produces an inflated chance of reporting at least one incorrect result. The inflation arises because random variability alone can generate seemingly “significant” outcomes in some subset of tests, even when no real effect is present for those hypotheses. As the number of comparisons grows, the probability of at least one false positive increases sharply, even if each individual test is calibrated to have a small error rate.

1.2 From error rates to discovery-oriented error control

1.2.1 Family-wise error rate (FWER) as a comparison point

A classical approach is to control the family-wise error rate: the probability of making one or more false rejections across the entire family of hypotheses. This is a strict notion of error control and directly targets the “any false discovery” scenario. While useful in some settings, FWER control can become highly conservative when hypotheses are numerous, because it must guard against false rejections occurring anywhere in the family.

1.2.2 Per-comparison error rate and its limitations

Control of the per-comparison error rate (often linked to testing each hypothesis at level α) ensures that each test individually has a limited probability of being wrong. However, it does not constrain the overall behavior across many tests. Consequently, analysts can end up with a large number of false positives among a long list of discoveries, even when every test meets its own marginal error target.

1.3 The concept of “discoveries” in statistical testing

In discovery-oriented workflows, the emphasis is not on whether any error occurs, but on how many incorrect findings are expected among the findings declared noteworthy. This perspective leads naturally to metrics tied to the proportion or fraction of false positives among all rejected null hypotheses. Under this framing, procedures aim to deliver a controlled level of false discoveries while allowing more true effects to be detected than is typically possible under stricter criteria.

2 Definition and Interpretation of FDR

2.1 Mathematical definition of false discovery rate

The false discovery rate (FDR) is defined as the expected value of the ratio \[ \mathrm{FDR}=\mathbb{E}\left[\frac{V}{R}\right], \] where \(V\) denotes the number of false discoveries (rejected true null hypotheses) and \(R\) is the total number of discoveries (rejections). A convention sets the fraction to zero when \(R=0\), so the expression remains well-defined even in the absence of any rejections.

2.2.1 False discovery proportion (FDP)

For a given realized dataset, the false discovery proportion (FDP) is \[ \mathrm{FDP}=\frac{V}{R}, \] again taken as zero when no hypotheses are rejected. FDP is a random variable that represents the realized fraction of false positives among all reported discoveries.

2.2.2 Expected FDP and interpretation as FDR

FDR is the expectation of the FDP over repeated realizations of the random experiment (under the statistical model). Thus, an FDR level \(q\) is interpreted as an average target: across repeated studies, the mean fraction of incorrect discoveries among reported ones is controlled to be at most \(q\) under the stated assumptions. This interpretation is often more aligned with practical reporting, where researchers compile lists of candidate discoveries.

2.3 Edge cases and conventions (e.g., zero discoveries)

The convention \(\mathrm{FDP}=0\) when \(R=0\) ensures that datasets with no rejections do not inflate the rate through division by zero. It also clarifies that the error control is distributional: even if some realizations yield many rejections with a higher FDP, the overall expected ratio remains bounded under the procedure’s guarantees.

2.4 How to read FDR in practice

In practice, an analyst specifies a target FDR level \(q\) and then uses an FDR-controlling procedure to select a rejection set. The outcome is a set of discoveries along with an adjusted decision rule derived from p-values (or test statistics). The guarantee is not that each individual discovery is correct with probability \(1-q\); rather, it is that the expected fraction of false discoveries among all selected findings is controlled.

3 FDR Control Procedures

3.1 Benjamini–Hochberg (BH) procedure

3.1.1 Step-up testing mechanism

The Benjamini–Hochberg (BH) procedure is a step-up method. Given p-values \(p_1,\dots,p_m\), the p-values are sorted in increasing order: \(p_{(1)}\le \cdots \le p_{(m)}\). BH finds the largest index \(k\) such that \[ p_{(k)} \le \frac{k}{m}q, \] and then rejects all hypotheses corresponding to p-values \(p_{(1)},\dots,p_{(k)}\). If no such \(k\) exists, no hypotheses are rejected.

3.1.2 Assumptions and when BH controls FDR

BH’s classic guarantee holds under independence of p-values for true nulls, or under certain types of positive dependence conditions among p-values. In many practical scenarios, dependence exists (e.g., genes with correlated expression, neighboring pixels in imaging). BH may still perform well under weaker assumptions, but rigorous FDR control depends on the dependence structure satisfying the conditions used in the theoretical results.

3.2 Storey’s procedure and the role of π0

3.2.1 Estimating the proportion of true nulls

Storey’s approach introduces the idea that if not all hypotheses are null, then the effective multiplicity is smaller than \(m\). Let \(\pi_0\) denote the proportion of true null hypotheses. Storey’s procedure estimates \(\pi_0\) from the data, typically using the distribution of p-values near 1, where true null p-values are expected to accumulate under standard assumptions.

3.2.2 Adaptive thresholds for increased power

With an estimate \(\widehat{\pi}_0\), the rejection thresholds become less stringent than the BH thresholds when \(\widehat{\pi}_0 < 1\). This adaptation often increases the number of true discoveries while maintaining an FDR guarantee in the sense of the underlying theoretical framework (which depends on how \(\widehat{\pi}_0\) is estimated and on the dependence assumptions).

3.3 Step-down and step-up variants

3.3.1 Strong vs. weak control perspectives

Variants of BH include step-down methods, which start from more stringent thresholds and relax only after certain conditions are met. In the theoretical literature, “strong control” refers to guarantees under all configurations of true and false null hypotheses, whereas “weak control” focuses on particular configurations (often the global null case). Whether a given variant provides strong or weak control determines how broadly its guarantees apply.

3.3.2 Extensions for structured testing problems

Structured extensions address situations where hypotheses are grouped, ordered, or linked by known scientific structure. For example, hypotheses might be arranged in hierarchies, or one might test effects across pathways rather than individual markers. These extensions aim to use additional structure to improve power while preserving target error control.

3.4 Procedures for dependencies among tests

3.4.1 Positive dependence conditions (high level)

Some results for FDR control under dependence rely on positive dependence properties—informally, situations where p-values tend to behave in a coordinated manner that does not create excessive false rejections. When such conditions are satisfied, BH-type procedures can retain FDR control without requiring full independence.

3.4.2 General dependence and robustness considerations

General dependence is more challenging because p-values may move in ways that can defeat simplified threshold logic. Robust approaches exist, including procedures designed to retain control under broader dependence assumptions, sometimes at the cost of reduced power. In applied settings, analysts often pair theoretical guidance with diagnostics and sensitivity checks.

4 Conditions, Theoretical Guarantees, and Control Modes

4.1 FDR vs. mFDR and other variants

Beyond the basic FDR definition, related criteria include modified or alternative error measures designed to handle situations where the number of hypotheses tested or the distributional assumptions differ. One example is mFDR, which conditions on the event of at least one true discovery or on other structural aspects of the testing framework. These variants are useful when exact FDR definitions interact inconveniently with model features, but the core aim remains to control an expected proportion of false findings.

4.2 Strong control versus weak control

4.2.1 Meaning of “all configurations” control

Weak control ensures an error bound under a specific global scenario, often when all null hypotheses are true. Strong control requires the bound to hold for every possible arrangement of true and false nulls. Strong control is particularly relevant in practice because, realistically, some hypotheses are likely to be true effects while others are not.

4.2.2 Practical implications for analysts

If only weak control is guaranteed, applying the procedure to a setting where many alternatives exist may result in deviation from the targeted error expectation. Strong control provides more reassurance that the nominal FDR level remains meaningful across the kinds of mixed truth configurations encountered in real studies.

4.3 Typical assumptions in proofs (overview level)

Theoretical guarantees typically rely on assumptions about p-value validity under true nulls (e.g., stochastic dominance relative to the uniform distribution) and assumptions about dependence among p-values for true null hypotheses. Additional conditions may involve the monotonicity or shape of certain rejection thresholds, ensuring that stepwise procedures behave consistently as p-values vary.

4.4 Conservativeness and calibration issues

Even when theoretical assumptions approximately hold, finite-sample calibration can be imperfect. Conservative behavior may occur when dependence is stronger than assumed, when p-values are discrete or otherwise non-ideal, or when estimators used in adaptive procedures are noisy. Conversely, in some mismatched scenarios, procedures may be anti-conservative. These issues motivate careful use of appropriate p-value computation, modern software implementations, and validation via simulation.

5 Estimation, Calibration, and Practical Implementation

5.1 Choosing the target level q

5.1.1 Interpreting q as an error expectation

The parameter \(q\) sets the intended upper bound on the expected FDP, not on the probability that any particular discovery is wrong. Choosing \(q\) is therefore a trade-off: larger values typically yield more rejections (higher discovery counts) but allow a higher expected fraction of false ones.

5.1.2 Sensitivity to q in real workflows

In many analyses, conclusions may change as \(q\) varies. For stable reporting, practitioners may examine how the set of discoveries changes across a plausible range of \(q\) values, or use domain criteria to pick a value that balances follow-up burden and tolerance for incorrect leads.

5.2 Handling p-values and test statistics inputs

FDR procedures are typically driven by p-values. Ensuring that p-values meet their nominal validity under true nulls is critical. If test statistics are not calibrated or p-values are computed with inappropriate distributional assumptions, FDR guarantees may no longer hold. In discrete settings, specialized methods or conservative adjustments may be needed to reflect that p-values can take limited values.

5.3 Computing adjusted thresholds efficiently

Implementations require sorting p-values and then identifying the cutoff index. For large \(m\), computational efficiency matters, but BH-style procedures are generally straightforward: the dominant cost is sorting, followed by a linear scan to find the maximal valid index. Adaptive procedures add computation for estimating \(\pi_0\), but these are also feasible with standard algorithms.

5.4 Common software pipelines (conceptual overview)

5.4.1 Reproducibility and reporting requirements

Reliable workflows include recording the exact procedure version, the chosen \(q\), how p-values were obtained, and any options affecting dependence handling or \(\pi_0\) estimation. Reproducibility also benefits from reporting key intermediate quantities, such as the number of rejections and the p-value cutoff used.

6 Comparing FDR with Other Error Controls

6.1 FDR vs. FWER

6.1.1 When FWER may be overly conservative

FWER control is designed to prevent any false rejection with high probability. With a large number of tests, this constraint can push thresholds so low that true effects are missed frequently. In high-dimensional settings, a procedure that aims to minimize the chance of even one false positive can sacrifice substantial power.

6.1.2 Trade-offs in power and strictness

FDR control typically permits some false rejections, as long as their expected proportion stays controlled. This often increases the number of true discoveries compared with FWER-based methods. The best choice depends on whether the scientific goal prioritizes eliminating all false positives or tolerates a limited fraction to gain sensitivity.

6.2 FDR vs. family-wise and per-test perspectives

Unlike family-wise approaches that focus on the existence of any error, FDR explicitly targets the composition of the rejection set. Compared with per-test error control, FDR accounts for how errors distribute among many findings. These differences lead to distinct operational meanings: an investigator seeking a ranked list of candidates often aligns better with FDR, while one focusing on confirmatory claims in a small number of tests may align more with stricter controls.

6.3 Practical decision criteria for method selection

6.3.1 Error tolerance and study goals

If downstream work can validate candidates selectively, a moderate FDR target can be appropriate. If follow-up resources are limited or incorrect claims are particularly costly, stricter error criteria may be favored.

6.3.2 Expected number of discoveries

Method selection can also depend on expected yield. FDR-controlling procedures often produce more discoveries, which may be advantageous when the cost per follow-up is low relative to the benefit of identifying true effects. However, analysts should consider how a larger candidate list impacts practical review and verification.

7 Performance Assessment and Diagnostics

7.1 Simulation studies for evaluating FDR procedures

Performance assessment commonly uses simulations that mimic the structure of the data: effect sizes, prevalence of true signals, correlation patterns, and p-value computation methods. Simulations can estimate empirical FDR and compare it to the nominal \(q\), as well as evaluate whether power gains persist under the assumed dependence model.

7.2 Power and detection efficiency under FDR control

Power under FDR constraints reflects the ability to identify true signals while keeping false discoveries controlled. Efficient FDR procedures aim to improve power relative to more stringent error measures. Analysts often measure detection probability, number of true positives among the rejections, or recall/precision-like summaries in contexts where ground truth can be defined.

7.3 Empirical verification of error behavior

Beyond simulations, analysts may use resampling approaches or partial validation strategies to check whether the observed error behavior aligns with expectations. While such checks cannot replace formal guarantees, they provide useful evidence when theoretical assumptions are uncertain.

7.4 Visual diagnostics (e.g., null vs. alternative signals)

7.4.1 Interpreting outcome plots and summaries

Visual tools can include histograms or density plots of p-values, empirical cumulative distribution functions, and plots of adjusted thresholds versus ordered p-values. In typical scenarios, p-values under alternatives tend to concentrate near zero, whereas null p-values are more uniformly spread. These patterns help assess whether the data exhibit signals consistent with the assumptions underlying adaptive or weighted procedures.

8 Special Topics and Extensions

8.1 Weighted FDR and importance weighting (overview)

Weighted FDR methods allow different hypotheses to carry different importance or prior relevance. The procedure uses weights to alter effective thresholds, which can increase power for scientifically important tests while still controlling an appropriate weighted error criterion. Choosing weights typically requires careful justification, as inappropriate weighting can undermine error guarantees.

8.2 Grouped or hierarchical testing frameworks

8.2.1 Controlling error across families of hypotheses

In grouped settings, hypotheses are organized into families (e.g., by pathway, brain region, or experimental batch). Hierarchical procedures control error across multiple levels, such as allocating an overall tolerance and distributing it across groups. This structure can help balance discoveries between major categories and refine which signals are pursued.

8.3 Online and sequential FDR control (overview)

Sequential FDR control addresses scenarios where hypotheses arrive over time and decisions may need updating as new tests are performed. Online methods provide rules for controlling FDR under streaming or adaptive sampling conditions, often by designing thresholds that maintain validity under the evolving nature of the dataset.

8.4 Post-selection inference considerations (high level)

FDR-controlled selection changes the set of hypotheses that proceed to subsequent analysis. Post-selection inference examines the distribution of estimates or test statistics after selection, accounting for the data-driven nature of the selected set. These considerations matter when the goal extends beyond discovery lists to accurate uncertainty quantification for selected effects.

9 Reporting and Best Practices

9.1 What to report for FDR-based results

Reporting typically includes the number of hypotheses tested, the FDR level \(q\), the procedure used (e.g., BH, Storey), and the criteria for rejection (such as the p-value cutoff or rejection rule). It is also useful to summarize the number of discoveries and the range of p-values among rejected hypotheses.

9.2 Communicating “expected proportion of false discoveries”

Clear communication emphasizes that FDR bounds the expected fraction of incorrect rejections, averaged over hypothetical repetitions. Researchers should avoid statements implying that any given discovery has a fixed probability of being true or false based solely on the chosen \(q\).

9.3 Avoiding common pitfalls in interpretation

9.3.1 Confusing FDR with FWER or with FDR=0 outcomes

A frequent mistake is equating FDR control with the guarantee that no false positives occur. FDR control does not eliminate false discoveries; it limits how many are expected as a fraction. Additionally, observing an outcome with zero false rejections in one dataset does not imply that FDR is zero in a broader probabilistic sense.

9.4 Reproducible analysis checklist (conceptual)

Best practice includes documenting p-value computation details, dependence-relevant preprocessing, the exact settings used in software, random seeds where applicable, and the versioned outputs that determine the rejection set. When adaptive methods are used, reporting how \(\pi_0\) was estimated and any tuning parameters supports transparency.

10 Summary and Key Takeaways

10.1 Core definition and intuition

FDR formalizes discovery-oriented error control by targeting the expected fraction of false discoveries among all reported rejections. This shifts attention from the existence of any error to the composition of the selected set.

10.2 Major procedures and when to use them

The BH procedure provides a foundational step-up method with widely understood conditions for FDR control. Storey’s procedure extends BH by adapting thresholds using an estimate of the proportion of true nulls, often improving power. Variants and extensions address dependency, weighting, grouping, and sequential testing.

10.3 Practical guidance for achieving FDR control

Successful application depends on valid p-values, appropriate procedure choice relative to dependence structure and study goals, and careful selection of the target \(q\). When assumptions may be uncertain, simulation-based checks and transparent reporting help ensure that empirical behavior aligns with the intended error control.