1 Concept and Problem Setup

1.1 Why multiple testing is needed

Multiple testing arises when researchers examine many hypotheses using the same dataset, such as testing whether each gene is associated with an outcome or whether each region shows differential activity. When tests are performed in bulk, random noise can create extreme statistics for some hypotheses even if all null conditions are true. Error rates that are calibrated for a single test—like the probability of at least one false positive—no longer describe the overall risk across the entire set. Multiple testing methods provide principled ways to correct this mismatch by controlling how often incorrect conclusions occur among all reported results.

1.2 Families of hypotheses and test indexing

A key modeling step is to decide what constitutes a “family” of hypotheses. The family determines which errors should be controlled together (e.g., all genes tested in an experiment, all edges in a network scan, or all comparisons within a study). Hypotheses are often indexed as \(H_1,\dots,H_m\), with corresponding p-values \(p_1,\dots,p_m\) or test statistics \(T_1,\dots,T_m\). Procedures then use the collection of p-values jointly to decide which nulls are rejected.

1.3 Common error metrics

1.3.1 Family-wise error rate (FWER)

The family-wise error rate is the probability of making at least one false rejection within the chosen family. Formally, if \(V\) is the number of true null hypotheses incorrectly rejected, then \( \text{FWER} = P(V \ge 1)\). FWER control is often desirable in high-stakes contexts because it limits the chance of any false positive, but it can be conservative when many tests are performed.

1.3.2 False discovery rate (FDR)

The false discovery rate is the expected proportion of false rejections among all rejected hypotheses. If \(R\) denotes the number of rejections and \(V\) the number of false rejections, then \( \text{FDR} = E\!\left[\frac{V}{\max(R,1)}\right]\). This criterion is less stringent than FWER, typically allowing more discoveries while still bounding the average rate of false ones.

Complementary criteria quantify errors among non-rejections. The false nondiscovery rate concerns the expected fraction of truly false null hypotheses that are not rejected. Related measures include the false omission rate and variations that target control of missed signals, often used in settings where failing to detect true effects is costly. Such criteria complement FDR-based approaches rather than replacing them, because different scientific goals emphasize different types of mistakes.

1.4 Null and alternative hypotheses in bulk testing

In bulk testing, each hypothesis has its own null and alternative, but the collection exhibits structure: many hypotheses may share a common null (e.g., “no association”), while alternatives may differ in magnitude, direction, or covariate dependence. Practical modeling often assumes a mixture perspective in which some fraction of hypotheses are truly null and the rest are truly non-null. This view connects multiple testing procedures to ideas about estimating the number or proportion of signals, which can enable more adaptive and less conservative corrections.

2 Classical Error Control Methods

2.1 Bonferroni-type adjustments

2.1.1 Single-step Bonferroni

The Bonferroni method controls FWER by bounding each individual test at a reduced significance level. If the desired FWER is \(\alpha\), each hypothesis is rejected when \(p_i \le \alpha/m\). This approach is broadly applicable and easy to implement, relying only on the union bound. Its main drawback is conservatism: when \(m\) is large, the threshold can become so stringent that power drops markedly.

2.1.2 Holm step-down procedure

The Holm procedure improves on Bonferroni while maintaining FWER control. P-values are ordered \(p_{(1)} \le \cdots \le p_{(m)}\). Starting from the smallest p-value, the method compares \(p_{(k)}\) to \(\alpha/(m-k+1)\), proceeding sequentially until a comparison fails; all remaining hypotheses are not rejected. Step-down logic preserves strong error control while typically yielding more rejections than single-step Bonferroni.

2.2 Hochberg and step-up family-wise methods

Step-up procedures reject hypotheses starting from the largest p-values in a way that can be less conservative than Holm under certain dependence patterns. One classic example is Hochberg’s procedure, which can control FWER when test statistics satisfy specific positive dependence conditions. Like other stepwise methods, it uses ordered p-values and iterates thresholds, balancing error control with increased sensitivity.

2.3 Šidák correction and assumptions

The Šidák correction provides an alternative FWER control strategy under assumptions of independent test statistics. For independent tests and desired FWER \(\alpha\), each test uses a threshold \(1-(1-\alpha)^{1/m}\), derived from the probability that none are rejected under the global null. Because independence is often unrealistic, Šidák is primarily used when its assumptions are plausible or as a benchmark for conservative adjustments.

2.4 Comparing stringency across FWER methods

FWER-controlling methods differ in how conservative they are in practice. Bonferroni is generally the most conservative, while Holm is often closer to optimal among broadly applicable procedures. Hochberg can be less stringent under favorable dependence structures, and Šidák can be quite efficient under independence. The relative behavior depends on the joint distribution of p-values and the proportion of true signals; as dependence grows or when assumptions fail, the nominal error control properties determine which method remains reliable.

3 False Discovery Rate (FDR) Approaches

3.1 Benjamini–Hochberg (BH) procedure

3.1.1 Standard BH thresholding

The BH procedure controls FDR at level \(\alpha\) under independence or certain positive dependence conditions. P-values are ordered \(p_{(1)} \le \cdots \le p_{(m)}\). One finds the largest index \(k\) such that \[ p_{(k)} \le \frac{k}{m}\alpha, \] then rejects all hypotheses with p-values \(p_{(i)} \le p_{(k)}\). This step-up rule yields more discoveries than FWER methods, especially when the number of true nulls is large.

3.1.2 Behavior under independence

Under independence, BH’s step-up threshold aligns well with the mixture nature of signals: as true effects increase, the proportion of small p-values grows, allowing more hypotheses to pass the adaptive cutoff. Under independence, the theoretical guarantee that the expected false discovery proportion is bounded is often met closely in finite samples, though deviations can still occur due to estimation effects or imperfect modeling of the test statistics.

3.2 Benjamini–Yekutieli (BY) procedure

3.2.1 Control under arbitrary dependence

BY extends FDR control to settings with arbitrary dependence among tests. It modifies the BH critical values by incorporating a factor that depends on the harmonic series, making the procedure more conservative. While this factor can reduce power, it provides a safety net when dependence is complex and positive-dependence assumptions for BH are uncertain.

3.3 Storey’s approach and estimating π0

3.3.1 Adaptive FDR with π0 estimation

Storey’s framework targets a more adaptive notion of FDR control by estimating \(\pi_0\), the proportion of true null hypotheses. When fewer nulls are present (smaller \(\pi_0\)), the procedure can use less conservative thresholds and increase power. Typically, \(\pi_0\) is estimated from the distribution of p-values in regions near 1, reflecting the idea that true null p-values are approximately uniform. This adaptivity introduces a new source of uncertainty, so careful choices of tuning parameters and robustness checks are often important.

3.4 Variants for adaptive and weighted testing

Weighted and adaptive FDR methods address situations where hypotheses have different prior plausibility or where test statistics have different variances. In weighted approaches, p-values may be adjusted using weights that emphasize some hypotheses over others, with corresponding theoretical modifications required to preserve FDR control. Variants also exist for two-sided testing, grouped testing, and procedures tailored to covariate-informed or stratified structures.

4 Resampling and Permutation-Based Techniques

4.1 Permutation tests in multiple testing

Permutation methods generate an empirical null distribution by shuffling labels or otherwise breaking the relationship between predictors and outcomes. In multiple testing, one can create permutation-based p-values for each hypothesis and then apply an error-control procedure using these p-values. This strategy can be particularly valuable when analytic null distributions are hard to derive or when dependence among tests is complicated.

4.2 Westfall–Young procedures

4.2.1 Step-down minP and maxT ideas

Westfall–Young methods use resampling to account for the joint behavior of test statistics under the null. Two common families are minP and maxT approaches, which rely on tracking the distribution of the minimum p-value or the maximum test statistic across hypotheses during permutations. Step-down versions can yield improved power while maintaining strong control of FWER or related criteria, largely because the calibration reflects the dependence structure present in the data.

4.3 Bootstrap-based multiple testing concepts

Bootstrap resampling can approximate uncertainty in situations where permutation is not straightforward, such as when resampling must respect clustering or dependence in time series or complex survey data. Bootstrap-based multiple testing typically requires careful design of the resampling scheme to ensure the null distribution is approximated adequately. Because resampling can distort dependence, theoretical and empirical validation is often necessary.

4.4 Computational considerations

Resampling procedures can be computationally intensive, especially with large \(m\) and many permutations. Practical workflows often balance the number of resamples against desired resolution in tail probabilities. Efficient implementations use parallel computing, caching of intermediate computations, and careful selection of the error criterion to limit unnecessary resampling overhead.

5 Local and Graph-Based Multiple Testing

5.1 Local false discovery rates

Local fdr replaces the global, hypothesis-set-level error rate with a per-hypothesis quantity that depends on the observed test statistic. Conceptually, it estimates the posterior probability that a given hypothesis is null given its statistic value. This allows ranking hypotheses by their likelihood of being null, which can be useful for decision-making and for producing adaptive thresholds based on covariate-dependent signal strength.

5.2 Empirical Bayes perspective

5.2.1 Estimating distributions of test statistics

Empirical Bayes methods treat the collection of test statistics as arising from a mixture model: a null component plus one or more alternative components. Parameters of this mixture are estimated from the data, enabling calculation of posterior probabilities and locally optimal rejection rules under certain loss functions. This perspective connects multiple testing to density estimation and to shrinkage ideas familiar from Bayesian inference, though the final procedure remains data-driven rather than fully Bayesian.

5.3 Multiple testing with hierarchical structures

Hierarchical testing recognizes that hypotheses may be nested, grouped, or organized across resolutions (e.g., wavelet coefficients, multi-level biomarker panels, or topic hierarchies). Procedures may allocate error control across layers, such as testing at a coarse level and refining within significant groups. The goal is to avoid treating all hypotheses as independent stand-alone items when scientific structure suggests otherwise.

5.4 Graphical dependencies and neighborhood smoothing

When tests are related through a graph—such as spatial adjacency, network neighborhoods, or gene regulatory pathways—dependencies can be exploited using smoothing or Markov random field ideas. Graph-based multiple testing often aims to borrow strength across neighboring nodes, producing more stable local decisions. Methods may incorporate penalties or modeling of the joint distribution to enhance power without abandoning rigorous error control.

6 Dependence and Correlation Between Tests

6.1 Independence vs. dependence assumptions

Many classical results distinguish between independence and dependence among p-values or test statistics. Independence simplifies derivations and supports sharper guarantees for procedures like BH. In real datasets, correlations often exist due to shared covariates, batch effects, linkage disequilibrium, or overlapping features in model construction. Multiple testing theory addresses this by specifying dependence classes under which error control holds or by designing procedures that remain valid under weaker assumptions.

6.2 Positive dependence and PRDS

Positive dependence conditions, such as PRDS (positive regression dependence on a subset), provide a middle ground between full independence and arbitrary dependence. Under PRDS-like assumptions, certain monotonicity properties hold that preserve the validity of BH-style procedures. These conditions are technical but practically important: when dependence is largely positive, FDR control may remain accurate without resorting to highly conservative corrections.

6.3 Measuring dependence in practice

Dependence is often assessed indirectly through diagnostics: correlation of summary statistics, behavior of permutation-based distributions, or stability of rejection sets under resampling. However, dependence can be nonlinear and hypothesis-specific, so simple linear correlation may not capture it fully. In applied work, researchers commonly combine theoretical guidance with empirical checks, such as verifying that error rates approximate nominal levels under resampling schemes.

6.4 Implications for FWER/FDR procedures

The choice between FWER and FDR procedures is closely tied to dependence structure. FWER methods that rely on union bounds are more robust but conservative, while FDR methods may be efficient when dependence is positive but require caution when dependence is arbitrary. When dependence is complex, permutation-based calibration or dependence-robust FDR procedures can provide safer error guarantees.

7 Power, Effect Size, and Practical Trade-offs

7.1 The role of signal strength

Power in multiple testing depends on how strongly alternatives differ from the null. As effects become weaker, p-values drift upward, reducing the number of thresholds crossed by stepwise procedures. Because multiple testing typically uses more stringent cutoffs than single-test analyses, a given effect size can yield substantially lower discovery rates unless the study design provides enough data or effect magnitude.

7.2 Sensitivity vs. specificity trade-offs

Error control methods balance two competing goals: identifying true signals (sensitivity) while limiting false reports (specificity). FWER control emphasizes specificity by preventing any false positive, often at the cost of sensitivity. FDR control tolerates some false discoveries in expectation to improve sensitivity. Local and Bayesian approaches add a refined trade-off by weighting decisions by estimated probability of being null.

7.3 Choosing target error rates

The target level \(\alpha\) determines how strict the thresholding becomes. Lower \(\alpha\) reduces expected error but may decrease the number of rejections and the interpretability of results when power is limited. Selection can depend on study goals, downstream costs of false positives, and the tolerance for follow-up experimentation. In exploratory contexts, some analyses aim for moderate FDR levels, while confirmatory studies often prefer stronger control criteria.

7.4 Power analysis for multiple testing

Power analysis extends beyond single-test calculations because the probability of rejecting at least one hypothesis or rejecting a particular fraction depends on the multiple-testing rule. Approaches include simulation studies under plausible effect sizes and dependence structures, approximate analytical results for certain models, and asymptotic approximations for large \(m\). In practice, researchers often perform scenario-based simulations to compare competing procedures under expected sparsity and correlation.

8 P-Values, Statistics, and Thresholding

8.1 From p-values to adjusted q-values

In FDR contexts, adjusted measures are often reported as q-values: p-value-like summaries that correspond to the minimal FDR level at which a hypothesis would be rejected by a given procedure. Under BH, q-values can be derived from the ordered p-values and the step-up thresholds. Reporting q-values helps communicate results in terms of controlled false discovery expectations rather than raw p-value significance.

8.2 Step-up vs. step-down algorithms

Step-up methods tend to reject all hypotheses below a data-dependent cutoff corresponding to a larger index in the ordered list, while step-down methods reject until the first failure of a threshold and then stop. The distinction matters because the direction of the algorithm affects conservatism and finite-sample behavior, particularly under dependence. Many guarantees rely on monotonicity properties that fit naturally with one algorithmic direction.

8.3 Interpreting adjusted significance outputs

Adjusted outputs change interpretation: a “significant” result under FDR control is not equivalent to a small probability of being a false discovery for each individual hypothesis in a strict sense, but it relates to the expected proportion of false rejections among all declared discoveries. Good interpretation also acknowledges that the procedure targets the chosen error metric and family definition, and that assumptions about dependence or model validity influence accuracy.

8.4 Handling discrete or tied p-values

When tests are discrete, such as those derived from counts or small-sample exact tests, p-values may take only a limited set of values and ties are common. Stepwise procedures can behave differently in discrete settings: thresholds may be less granular, and nominal error control can become conservative or inaccurate depending on the method. Adjustments may involve modified critical values, conservative bounds, or randomization strategies to restore calibration.

9 Procedures for Different Study Designs

9.1 Exploratory screening vs. confirmation

Exploratory screening aims to find candidate signals among many possibilities, typically prioritizing discovery. Confirmatory analysis aims to validate specific findings while maintaining strict error properties. Multiple testing methods can serve both goals, but exploratory stages often use FDR-oriented approaches and confirmatory stages may use more stringent corrections or distinct data splits to avoid inflating error through repeated use of the same information.

9.2 Two-stage and multi-stage testing

Two-stage testing uses an initial screen to reduce the hypothesis set and then applies more targeted procedures within the reduced family. Multi-stage designs extend this idea across multiple rounds, potentially using hierarchical selection or adaptive thresholds. Proper control requires accounting for the selection step; naive application of standard corrections after screening can lead to error inflation.

9.3 Nested hypotheses and selective inference basics

Nested hypotheses occur when rejection in one model constrains or implies structure in another. Selective inference addresses the fact that selection rules—such as choosing features based on data—affect the distribution of subsequent test statistics. Theoretical tools aim to provide valid p-values or error control after selection, ensuring that reported significance reflects the entire workflow rather than only the last step.

9.4 Replication and cross-validation workflows

Replication tests the reproducibility of discovered signals in independent samples. Cross-validation is often used to estimate predictive performance rather than hypothesis error rates, but it can influence which models are examined, thereby affecting inference validity. Multiple testing workflows increasingly use data splitting: discovering candidates in one portion of the data and validating in another, reducing the risk of double-dipping and making error control more aligned with the analysis plan.

10 Software, Workflows, and Reporting

10.1 Common implementation patterns

Implementation usually follows one of several patterns: compute test statistics, obtain p-values, then apply a stepwise or weighted adjustment procedure. For FDR methods, many tools allow BH or BY variants, while for dependence-robust approaches users may choose resampling-based corrections. In hierarchical or graph-based settings, algorithms often involve additional model fitting steps to estimate mixture parameters or local posterior quantities.

10.2 Reproducible analysis pipelines

Reproducibility is supported by documenting random seeds, resampling counts, and software versions, especially for permutation or bootstrap methods. Pipelines typically store intermediate outputs such as p-value vectors, ordering indices, and estimated nuisance parameters. Clear pipeline structure helps ensure that the chosen family of hypotheses and the corresponding adjustment method remain consistent throughout the workflow.

10.3 Reporting adjusted results and assumptions

Reporting generally includes the error metric targeted (FWER or FDR), the procedure used (e.g., BH, Holm, permutation-based Westfall–Young), and the family definition. Because guarantees may depend on dependence or calibration assumptions, authors often report how p-values were obtained, whether resampling was used, and whether any data-driven grouping or filtering occurred prior to adjustment. When discrete tests or ties are present, methods for handling them should be noted.

10.4 Pitfalls in practice (e.g., double-dipping)

A common pitfall is performing selection or feature screening and then using the same data for significance testing without accounting for the selection step. This “double-dipping” can lead to overly optimistic adjusted results and invalid error control. Another issue is inconsistent family definitions—adjusting within one subgroup but interpreting across multiple subgroups without correction. Robust workflows track these choices explicitly and, when needed, separate discovery and validation stages.

11 Mathematical Foundations and Theorems (High-Level)

11.1 Definitions of controlled error rates

At a high level, the main controlled quantities—FWER, FDR, and variants like local false discovery rate—are defined in terms of the random sets of rejected hypotheses. These definitions formalize what it means for a method to be “valid”: the procedure must bound the expected frequency of certain error types under specified model assumptions. Additional mathematical objects include the numbers of true nulls, false rejections, and the relationship between rejection thresholds and ordered p-values.

11.2 Key properties of stepwise procedures

Stepwise procedures have structural properties that make analysis tractable. For many methods, rejection sets can be characterized by monotonicity with respect to p-value ordering, and the error rate can be expressed using probabilities of crossing specific thresholds. For step-up and step-down algorithms, the mathematical proofs often rely on how the set of rejected hypotheses changes as p-values vary, ensuring that the procedure behaves predictably as evidence strengthens or weakens.

11.3 Consistency and asymptotic behavior (overview)

Asymptotic theory examines what happens as the number of hypotheses grows, often under assumptions about sparsity (e.g., only a small fraction of hypotheses are non-null) and how test statistics behave in the limit. Consistency results address whether the procedure increasingly distinguishes signals from noise and whether error rates converge to their targets. Such results justify the use of asymptotically derived procedures in large-scale studies, though finite-sample performance still needs empirical evaluation.

11.4 Linking theoretical guarantees to empirical workflows

A central theme in multiple testing is translating theoretical guarantees into usable analysis pipelines. This involves matching assumptions to the data-generating context, specifying the hypothesis family correctly, and ensuring that p-values or test statistics meet the regularity conditions assumed in proofs. Empirical validation—through simulation under plausible dependence and through resampling-based calibration—bridges the gap between theorem-level correctness and day-to-day practice in real datasets.