1 Overview of multiple testing and error rates

1.1 The need for multiplicity adjustments

When many hypotheses are tested at once, using a single testing threshold (such as a fixed 5% significance level) often leads to an inflated rate of spurious findings. The core issue is that chance alone can produce small p-values even when null hypotheses are true. Multiplicity adjustments modify decision rules so that overall error behavior remains controlled as the number of tests grows.

1.2 Family-wise error rate (FWER) versus false discovery rate (FDR)

Multiple-comparison error metrics emphasize different notions of “overall correctness.” The family-wise error rate (FWER) concerns the probability of at least one false rejection among all tests. By contrast, the false discovery rate (FDR) targets the expected proportion of false rejections among the rejected set. This difference allows FDR procedures to be less conservative than FWER controls, especially when there are many tests and only a fraction correspond to true signals.

Several closely related quantities are used in practice. The FDR is defined as the expected value of the proportion of false discoveries conditional on the event of at least one rejection, and it is commonly interpreted as an “average” contamination level among discoveries. Variants also include the positive false discovery rate (pFDR), which conditions differently, and the tail-focused form called the false discovery exceedance. The choice among these metrics depends on the scientific question and on how analysts handle cases with few or no discoveries.

2 The Benjamini–Hochberg procedure (BH)

2.1 Basic setup: hypotheses, p-values, and rank ordering

Consider m null hypotheses H1, …, Hm with corresponding p-values p1, …, pm. The BH procedure sorts the p-values from smallest to largest, producing an ordered sequence p(1) ≤ p(2) ≤ … ≤ p(m). Each ordered p-value is associated with a specific hypothesis, and the ordering determines which hypotheses fall below the rejection threshold.

2.2 BH step-up rule and rejection cutoff

BH uses a step-up rule: for a target FDR level q, it finds the largest index k such that p(k) ≤ (k/m) · q. All hypotheses with p-values at ranks 1 through k are rejected; those with p-values above p(k) are not rejected. This “largest admissible k” construction yields a monotone rejection set as thresholds increase.

2.3 Choice of the nominal FDR level (q)

The parameter q is the nominal FDR control level. Selecting q involves a trade-off: larger q permits more rejections but increases the expected fraction of false discoveries, while smaller q is stricter and typically yields fewer discoveries. In applied settings, q is often chosen to reflect the tolerance for occasional false positives among reported findings.

2.4 Worked-out outline of the algorithm

  1. Compute p-values for all m hypotheses.
  2. Sort p-values to obtain p(1), …, p(m).
  3. For each rank i, compute the BH threshold (i/m) · q.
  4. Identify k as the maximum i where p(i) ≤ (i/m) · q.
  5. Reject hypotheses corresponding to p(1) through p(k).
  6. If no p(i) satisfies the inequality, set k = 0 and reject none.

2.5 Edge cases and practical considerations

If all p-values are large relative to the increasing thresholds, BH returns no rejections. If many p-values are extremely small, BH may reject a substantial portion of hypotheses, making the dependence structure and calibration especially relevant in finite samples. Another practical point is that BH relies on p-values being valid under their null hypotheses; malformed p-values (e.g., from mis-specified models) can undermine error control.

3 Statistical properties and assumptions

3.1 Dependence structures: when BH guarantees hold

BH’s classical theoretical guarantee holds under conditions stronger than mere validity of p-values. In the original formulation, guarantees are provided under independence among p-values corresponding to true null hypotheses. More generally, results extend to specific dependence patterns that preserve certain positive regression properties for indicators of small p-values.

3.2 Independence versus positive dependence

Under independence of p-values (or related assumptions), BH controls FDR at the target level q. Under positive dependence—often formalized through conditions such as positive regression dependence on subsets—the procedure remains valid. These frameworks capture many realistic scenarios where test statistics are correlated but not in a way that systematically drives too many p-values downward simultaneously.

3.3 Conservativeness and behavior in finite samples

Theoretical statements typically consider asymptotic or exact properties under idealized conditions. In finite samples, BH can be conservative or slightly liberal depending on how dependence and distributional assumptions line up with those required by the proofs. Conservativeness is common when p-values are not too discrete and dependence is mild, while liberal behavior can occur if key assumptions fail.

3.4 Relation to power and calibration

FDR control is often discussed alongside statistical power, the probability of rejecting false null hypotheses. Because BH is calibrated to control an average proportion of false rejections rather than the probability of any false rejection, it can achieve higher power than FWER methods for the same nominal stringency. Calibration of q is therefore central: the nominal level should be interpreted as a target for expected false discovery proportion under the assumed conditions.

4.1 Benjamini–Yekutieli (BY) procedure

The Benjamini–Yekutieli (BY) procedure extends FDR control to broader dependence settings. It uses a more conservative threshold involving a harmonic-series correction factor. As a result, BY typically yields fewer rejections than BH when dependence is complex but offers stronger validity under minimal assumptions.

4.2 Step-down versus step-up approaches

BH is a step-up procedure. Related methods use step-down logic, which can alter rejection behavior when p-values are ordered. Step-down procedures often provide improved control properties under some configurations or can be more stable in certain dependence patterns. The choice between step-up and step-down versions is usually guided by theoretical guarantees and by practical performance considerations.

4.3 Adaptive BH procedures

Adaptive procedures attempt to improve power without sacrificing control by estimating quantities related to the effective number of true nulls. A common strategy is to replace m in the BH threshold with an estimate of the number of hypotheses that are likely null. Adaptation can yield more discoveries when the signal fraction is substantial, but it requires careful construction to maintain error control.

4.4 Storey’s q-value and estimation-based variants

Storey’s q-value framework estimates the proportion of true null hypotheses, often denoted π0. From this estimate, one can form adjusted rejection thresholds that aim to maintain the desired FDR level while leveraging the observed distribution of p-values. The resulting quantities are widely used because they provide an interpretable mapping from p-values to estimated “q-values,” though implementation details and the choice of estimation tuning parameters can affect outcomes.

5 Implementation details

5.1 Sorting and threshold computation strategies

A straightforward BH implementation sorts the p-values and iteratively checks the inequality p(i) ≤ (i/m)·q. For efficiency, threshold values (i/m)·q can be precomputed as an increasing sequence, and the maximal valid index can be found by scanning from largest to smallest rank or by using vectorized comparisons in software environments.

5.2 Handling ties in p-values

In theory, ties are not problematic, but in practice p-values may be discrete and therefore repeat. Standard BH implementations typically apply the rule using the ordered p-values as given, which implicitly determines k based on the tied values. Analysts should use software that implements a well-defined convention for ties and should check that their pipeline is consistent across reproducibility runs.

5.3 Computational complexity for large numbers of tests

The dominant cost in BH is sorting, which generally scales as O(m log m). For very large m, this remains feasible in many settings, but memory usage and data handling become practical constraints. Efficient data structures and streaming pre-processing (when possible) can help, while maintaining correct sorting and threshold evaluation.

5.4 Software and reproducible workflows

BH is available in many statistical libraries and command-line tools. For reproducibility, workflows often include fixed seeds for any resampling steps used to generate p-values, versioned software, and saved intermediate outputs such as the sorted p-values and the selected cutoff index k. Transparent recording of q and any adaptation or estimation steps is also important.

6 Interpretation in applied studies

6.1 Reporting results with FDR control

In applied research, it is common to report which hypotheses are rejected under BH at level q, alongside summary statistics such as the number of discoveries and the estimated or targeted FDR. Because BH is designed for expected proportions, reporting may emphasize that some false positives are allowed within the rejected set, provided the overall proportion remains near the target under the stated assumptions.

6.2 Visualization of selected hypotheses (e.g., threshold plots)

A common diagnostic is a threshold plot: ordered p-values are displayed against their BH thresholds (i/m)·q. The rejection cutoff appears where the p-value curve crosses below the threshold curve. Such plots help readers see how strongly the rejected hypotheses differ from non-rejected ones and whether the cutoff is driven by a cluster of borderline p-values or by clearly separated signals.

6.3 Translating rejections into scientific conclusions

Translating statistical rejections into scientific claims typically requires domain knowledge. An FDR-controlled rejection means the hypothesis is deemed unlikely to be null given the model used to compute p-values, and the set of such decisions is expected to contain only a limited fraction of false discoveries. Nevertheless, follow-up validation—such as replication studies, effect size assessment, and robustness checks—remains central to turning statistical findings into durable conclusions.

7 Extensions beyond standard BH

Extensions consider scenarios where hypotheses are grouped, filtered, or tested conditionally. Some procedures control FDR within subsets of hypotheses, such as controlling the false discovery proportion among a specific category. This is useful when different scientific priorities attach to different hypothesis families or when the testing strategy is hierarchical.

7.2 BH-like methods in structured testing settings

Structured testing arises in settings where hypotheses are not exchangeable, or where test statistics have known organization (e.g., spatial, temporal, or network structures). BH-like strategies adapt rejection rules to reflect structure while attempting to maintain FDR-type guarantees. These methods often rely on reweighting or resampling to manage correlation patterns induced by the structure.

7.3 Hierarchical and group-wise generalizations

Hierarchical frameworks may incorporate multiple stages of testing, such as screening followed by confirmatory testing. Group-wise generalizations evaluate hypotheses aggregated into clusters and then drill down. In these contexts, the challenge is preserving error control while allowing richer decision rules that mirror how discoveries are interpreted in the application.

8 Example scenarios and simulation perspectives

8.1 Synthetic example: ranking and cutoff selection

Suppose m = 10 tests and q = 0.1. After computing p-values and sorting them, the analyst checks each ordered p(i) against i/10·0.1. If, for example, p(1) and p(2) satisfy the inequalities but p(3) does not, then k = 2 and only the first two ranked hypotheses are rejected. This illustrates how the BH cutoff depends on the largest rank that remains below the corresponding threshold.

8.2 Comparative behavior versus FWER methods

FWER procedures, such as Bonferroni-style adjustments, aim to keep the probability of any false rejection below a target. In large-scale problems with many tests, such methods can be overly stringent, often reducing sensitivity. By controlling the expected fraction of false discoveries instead, BH commonly yields more rejections for the same nominal tolerance, though the exact gain depends on signal strength and dependence among tests.

8.3 Typical simulation design for assessing FDR control

Simulation studies often generate test statistics under both null and alternative hypotheses, then compute p-values and apply BH at various q values. Performance is assessed by measuring the empirical FDR, defined as the average realized proportion of false rejections among rejected hypotheses. To probe robustness, simulations vary dependence structures, effect sizes, and proportions of true nulls.

8.4 Sensitivity analysis for dependence and signal strength

Sensitivity analyses explore how BH’s behavior changes when dependence assumptions are weakened or when signals are weak. Analysts may compare scenarios such as independent nulls versus correlated null statistics, or sparse alternatives versus dense alternatives. These studies reveal whether observed FDR tracks the nominal q and how power changes as the fraction of true signals or their magnitudes vary.