1 Scope and motivation

Multiple testing correction comprises statistical procedures for situations in which many hypothesis tests are conducted at once. When each test is assessed using a conventional significance threshold, the probability of at least one false positive grows with the number of tests. Correction methods modify how evidence is judged so that reported findings have error rates that can be interpreted in the context of all tests performed.

1.1 Why multiple comparisons increase false positives

If each null hypothesis were tested at level \(\alpha\) and the nulls were all true, then even under perfect randomness, some tests will fall below the threshold simply by chance. As the number of tested hypotheses increases, the minimum p-value across tests is increasingly likely to be small, leading to an inflated chance of reporting at least one significant result even when no real effects exist.

1.2 Definitions of hypotheses and “families” of tests

A central modeling choice is how to group tests into a “family.” The family determines the scope of the error guarantee. In practice, families can be defined by the experimental question, the set of outcomes being screened, or the collection of hypotheses produced by a pre-specified analysis plan. Different choices can lead to different correction strengths and different interpretations.

1.3 Relationship between p-values and multiplicity

P-values are tail-area probabilities computed under each null model. Multiple testing correction treats p-values as a collection and applies rules that depend on their joint behavior—either through mathematical assumptions (such as independence) or through resampling. The adjustments can be expressed as modified significance thresholds, or equivalently as “adjusted p-values” that allow a uniform reporting scale.

2 Core error-rate concepts

Error-rate targets quantify different notions of false discoveries. The most common distinction is between controlling the probability of any false rejection and controlling the expected proportion of false rejections among the rejected set.

2.1 Family-wise error rate (FWER)

The family-wise error rate is the probability of making at least one false rejection among the hypotheses in the defined family. Controlling FWER is often described as protecting the “overall” conclusion from any mistaken positives.

2.1.1 Strong vs. weak control

Many FWER procedures offer either strong control or weak control. Strong control means the method maintains the target error rate for any configuration of true and false null hypotheses. Weak control requires the guarantee only when all nulls are true. In applied work, strong control is typically preferred because experiments rarely satisfy the “all nulls are true” scenario.

2.2 False discovery rate (FDR)

False discovery rate targets the expected proportion of false rejections among all rejected hypotheses. FDR is generally less conservative than FWER, making it attractive in high-throughput studies where some false positives may be tolerable in exchange for greater discovery power.

2.2.1 Key variants: mFDR, q-values, and local FDR (conceptual)

Variants refine what “rate” means or how results are reported. The marginal FDR (mFDR) focuses on a particular expectation under the null distribution of each hypothesis. The term q-value is commonly used for FDR-adjusted quantities that can be interpreted as the minimum FDR level at which a given hypothesis would be rejected by an FDR procedure. Local false discovery rate is a related conceptual approach that assigns posterior-like uncertainty to each test, often framed within mixture-model thinking rather than purely frequentist thresholding.

Beyond FWER and FDR, additional quantities help communicate error behavior from different perspectives.

2.3.1 Per-comparison error rate

The per-comparison error rate is the probability of rejecting a specific true null hypothesis. It is simpler but does not directly address the multiplicity-adjusted probability of mistakes across a whole set.

2.3.2 False discovery proportion (FDP)

The false discovery proportion is the realized fraction of false rejections among all rejections for a particular outcome of the testing procedure. It is random because the number of rejections changes with the observed data. FDR is the expectation of FDP under the procedure.

2.3.3 Positive predictive value (interpretive role)

Positive predictive value is a concept often used to describe the probability that a reported positive is truly positive. While not identical to FDR, it is closely connected in interpretive discussions because both reflect how often discoveries correspond to true effects. In practice, positive predictive value can depend on baseline rates (prevalence) and modeling assumptions, so it is frequently treated as an interpretive bridge rather than a strict error-control target.

3 p-value adjustment methods

Many correction strategies can be expressed as rules that compare ordered p-values to thresholds, often resulting in a rejection set defined by the smallest indices meeting those criteria.

3.1 Step-up procedures

Step-up methods sort p-values from smallest to largest and seek the largest index that satisfies a threshold rule. Hypotheses with p-values at or below that index are rejected.

3.1.1 Benjamini–Hochberg procedure (BH)

The Benjamini–Hochberg procedure is a canonical FDR-controlling method. After ordering p-values \(p_{(1)} \le \cdots \le p_{(m)}\), BH rejects the largest \(k\) such that \(p_{(k)} \le \frac{k}{m}\alpha\), where \(m\) is the number of tests and \(\alpha\) is the desired FDR level. Under standard independence assumptions, BH controls FDR at the target level.

3.1.2 Extensions and general step-up framework

Step-up frameworks generalize BH by altering threshold formulas, incorporating weights for different hypotheses, or adapting to specific dependence assumptions. These extensions aim to improve power while maintaining control of the chosen error metric.

3.2 Step-down procedures

Step-down methods also use ordered p-values, but they start at the smallest p-value and proceed downward until a violation occurs. Once the rule fails, larger p-values are typically not rejected.

3.2.1 Holm–Bonferroni procedure

Holm–Bonferroni is a common strong control method for FWER. It rejects hypotheses sequentially with progressively less stringent thresholds: for the \(i\)-th smallest p-value, the threshold is \(\frac{\alpha}{m-i+1}\). This produces a procedure that is uniformly at least as powerful as the basic Bonferroni correction while preserving FWER control.

Hochberg’s procedure is another step-down-like method (often described in the step-up form depending on presentation) that can provide improved power relative to Holm under certain dependence conditions. Its use depends on assumptions about the joint distribution of test statistics or p-values.

3.3 Direct multiplicity adjustments

Some approaches adjust individual significance levels directly without ordering, or via a simple analytic transformation of p-values.

3.3.1 Bonferroni correction

Bonferroni controls FWER by testing each hypothesis at \(\alpha/m\). Equivalently, a p-value can be multiplied by \(m\) and compared to \(\alpha\). The method is simple and broadly applicable, but often conservative, especially when many hypotheses are correlated.

3.3.2 Šidák correction

Šidák’s correction is derived under independence and provides a threshold that can be slightly less conservative than Bonferroni. It is based on converting the desired FWER into a per-test level using the independence structure.

3.4 Conservativeness and power trade-offs

Correction strength directly affects power, the probability of rejecting false null hypotheses. Stricter error control targets (such as FWER versus FDR) and conservative thresholds (such as Bonferroni under dependence) tend to reduce false positives at the cost of fewer true discoveries. The choice of method thus involves balancing interpretability, expected false discoveries, and sensitivity to real effects.

4 Controlling FWER

FWER procedures aim to keep the probability of even a single false rejection below the specified level. This is often important when downstream consequences of any false positive are severe.

4.1 Bonferroni-based approaches

Bonferroni provides baseline FWER control by dividing the significance level across tests. Its key advantage is minimal assumptions. However, its conservativeness can be pronounced in large testing problems, especially when many hypotheses are correlated.

4.2 Holm’s method

Holm’s procedure improves upon Bonferroni by using order information. Because it adjusts thresholds depending on how small earlier p-values are, it can reject more hypotheses while maintaining the same FWER target under standard settings.

4.3 Sidak-based approaches

Sidak’s method uses a per-test threshold derived from the independence assumption to achieve the same overall FWER target. Under independence it can be more permissive than Bonferroni, but the guarantee may fail when dependence is not compatible with the derivation.

4.4 Dependencies and impact on FWER control

Dependence among tests can affect the conservativeness of procedures. For FWER, many methods retain validity under wide conditions, but some refined methods rely on stronger assumptions. When dependence is complex, conservative approaches may remain safe, whereas dependence-aware procedures aim to avoid unnecessary loss of power.

5 Controlling FDR

FDR control is designed for settings where some false positives are acceptable as long as they remain a small fraction on average.

5.1 Benjamini–Hochberg under independence

Under independence or certain positive dependence conditions, BH controls FDR at the desired level. In typical workflows, it is used when p-values can be reasonably modeled as sufficiently independent or when the dependence structure is expected to be favorable for BH’s theoretical conditions.

5.2 Benjamini–Yekutieli under dependence (conceptual)

When dependence is arbitrary, BH’s standard guarantees may not hold. Benjamini–Yekutieli provides a more conservative FDR control strategy for general dependence by scaling thresholds to account for worst-case behavior. It is often viewed as a safe default when dependence assumptions cannot be justified.

5.3 Adaptive and two-stage FDR procedures

Adaptive methods attempt to recover power by estimating components of the multiple-testing setup from the data. Two-stage strategies commonly screen hypotheses using a rough criterion and then apply a refined FDR rule within the selected subset.

5.3.1 Estimation of the proportion of nulls

Many adaptive procedures incorporate an estimate of how many hypotheses are likely to be null. If the fraction of non-null hypotheses is larger than expected, adaptive strategies can loosen thresholds for discoveries while maintaining FDR control, provided the estimation and procedure are correctly coupled.

5.4 q-value estimation and interpretation

q-values summarize FDR control in a per-hypothesis reporting format. A common interpretation is that selecting all hypotheses with q-values below a threshold \(q^*\) yields an expected false discovery proportion at most \(q^*\), under the assumptions of the chosen FDR procedure. This framing helps non-specialists understand the meaning of “significance” under FDR.

6 Dependence structures and assumptions

Dependence influences the calibration of multiplicity corrections. Some methods rely on independence-like behavior, while others are designed to be robust.

6.1 Independence vs. positive dependence

Independence is rarely exact in complex data. Positive dependence—where small p-values tend to occur together—can sometimes make FDR control easier than in the worst-case arbitrary dependence scenario. Many theoretical results distinguish independence-like assumptions from more adversarial dependence patterns.

6.2 Arbitrary dependence and robust methods

Robust FDR methods target validity without assuming independence. They typically use conservative adjustments or resampling strategies that approximate the null distribution under the observed dependence.

6.3 Effective number of tests

In correlated testing problems, the number of effectively distinct comparisons may be smaller than the literal count \(m\). Some modern approaches incorporate this idea through weighting or model-based adjustments, aiming to avoid excessive penalization caused by treating correlated tests as independent.

6.4 Resampling approaches under dependence

Permutation and bootstrap methods can preserve aspects of dependence by resampling in a way that respects the data-generating mechanism under the null. These approaches can be computationally expensive but are often helpful when analytic assumptions about dependence are questionable.

7 Resampling and permutation-based correction

Resampling methods approximate the distribution of test statistics under null hypotheses while retaining dependence structures present in the observed data.

7.1 Permutation tests for adjusted significance

Permutation tests repeatedly shuffle labels (or otherwise re-randomize under a null model) to generate a reference distribution. Multiplicity-adjusted significance can be obtained by comparing observed test statistics or p-value aggregates to their permuted counterparts, enabling control targets that are difficult to justify analytically.

7.2 Westfall–Young type procedures (conceptual)

Westfall–Young procedures are a class of methods that use permutation-based estimates to derive strong control for family-wise error in multiple testing. Conceptually, they aim to capture how the joint distribution affects the probability of observing extreme patterns of p-values by chance.

7.3 Bootstrap approaches for FDR/FWER (overview)

Bootstrap methods resample with replacement or simulate from fitted models to approximate uncertainty under dependence. In the multiple-testing context, bootstrap-based adjustments can target FWER or FDR, though correctness depends on how the null is represented and whether the resampling scheme matches the testing assumptions.

7.4 Computational considerations

Resampling requires many iterations to estimate tail probabilities accurately, especially for small \(\alpha\) or large families. Computational cost can become substantial, motivating approximations, parallel computation, or fewer permutations with careful calibration.

8 Modeling-based approaches

Instead of only adjusting p-values directly, modeling approaches attempt to represent the data as coming from a mixture of null and non-null components.

8.1 Empirical Bayes perspective (overview)

Empirical Bayes treats the proportion of null hypotheses and the distribution of signals as unknown quantities estimated from the data. This perspective can yield procedures for selecting thresholds that target FDR-like quantities, using estimated mixture parameters rather than only rank-based adjustments.

8.2 Mixture models and local false discovery rate (conceptual)

Mixture models posit that observed test statistics arise from multiple distributions—typically a null component and one or more alternative components. The local false discovery rate conceptually represents the posterior probability that a particular hypothesis is null given its test statistic, offering a probabilistic measure for each feature.

8.3 Comparison with classical p-value adjustment

Classical corrections rely on ordering and error-rate control without modeling the shape of the alternatives. Modeling-based methods can adapt more flexibly and may improve power, but they depend on correct model specification or at least adequate approximations, so misspecification can impact validity.

8.4 Assumption checking and sensitivity

Because modeling approaches are sensitive to assumptions about null distribution or mixture form, practitioners often use diagnostic checks, calibration by null resampling, and sensitivity analyses. Robustness can be assessed by comparing results under alternative null models or by evaluating stability across resampling schemes.

9 Choosing a correction strategy

Selecting a method involves aligning statistical goals with practical constraints such as dependence, interpretability, and computational budget.

9.1 Selecting the error metric (FWER vs. FDR)

FWER control emphasizes avoidance of any false positive in the family. FDR control allows a controlled average rate of false discoveries, often making it more suitable for exploratory screening or large-scale studies where follow-up is expected.

9.2 Effect size considerations and power

If true effects are rare or small, conservative corrections can strongly reduce power. Conversely, when signals are substantial or the number of null hypotheses is limited, less restrictive FDR procedures can yield more actionable results. Evaluating power typically requires either simulation or assumptions about effect distributions.

9.3 Sample size and number of tests

Large numbers of tests can dominate the multiplicity penalty, even with moderate sample sizes. With limited sample sizes, p-values may be noisy, and dependence estimation for some procedures can be unreliable. This motivates either robust methods or resampling schemes that do not require fragile analytic assumptions.

9.4 Practical workflow recommendations

A common workflow is to define the family based on the analysis plan, choose an error target, assess dependence plausibility, then select a method accordingly. Reporting both the multiplicity-adjusted decision rule (or adjusted p-values) and the chosen error metric helps readers interpret results consistently.

10 Implementation details

Correct implementation affects both numerical validity and interpretability of reported adjusted results.

10.1 Input requirements (p-values, test statistics, dependence info)

Many procedures accept p-values directly, while others require test statistics or additional information such as groupings, weights, or the form of dependence. Dependence knowledge can determine whether resampling or robust corrections are preferable.

10.2 Common software options and functions (conceptual)

Statistical software packages commonly include implementations of Bonferroni, Holm, BH, and related procedures, along with functions for producing adjusted p-values or q-values. Some environments also provide resampling or permutation capabilities for multiplicity control.

10.3 Handling ties and numerical precision

Ties in p-values can occur due to discrete test statistics or rounding. Many algorithms handle ties gracefully, but small differences in implementation can affect the exact rejection set, especially near thresholds. Numerical precision considerations are particularly relevant when working with extremely small p-values.

10.4 Reproducibility and reporting adjusted results

Reproducibility benefits from explicitly recording the correction method, the error target, how the family was defined, and software versions. Reporting adjusted p-values or the final rejection set with the chosen \(\alpha\) or q-level enables downstream readers to verify and interpret conclusions.

11 Reporting and interpretation

Reporting should communicate both which hypotheses were rejected and how uncertainty was controlled across the full set.

11.1 How to present adjusted p-values or rejection sets

Adjusted p-values provide a direct comparison to a desired significance level under the chosen correction. Alternatively, researchers may report the rejection set along with the threshold used. Clear notation regarding the number of tests and the family definition prevents misinterpretation.

11.2 Interpreting FDR-controlled findings

Under FDR control, “significant” results are not guaranteed to be all true positives. Instead, they carry an expected error rate interpretation: among the rejected hypotheses, the proportion of false discoveries is controlled in expectation. This helps calibrate confidence when follow-up and validation are planned.

11.3 Common reporting mistakes

Common errors include applying corrections to a different set of hypotheses than the family intended, mixing p-values computed under one testing scheme with corrections derived for another, and failing to state the correction target (FWER vs FDR). Another issue is presenting raw p-values without specifying whether correction was applied.

11.4 Communicating uncertainty to non-statistical audiences

For broad audiences, it is often helpful to translate the error metric into plain language: FWER relates to the chance of any false claim, while FDR relates to how frequently false claims may appear among reported positives on average. Using consistent terminology such as “adjusted significance” and “expected false discoveries” supports understanding.

12 Special cases and extensions

Multiple testing correction extends beyond basic one-shot hypothesis testing by adapting to structured experimental designs.

12.1 Hierarchical testing and nested hypotheses (overview)

When hypotheses form a hierarchy—such as hypotheses grouped by pathway, region, or model components—testing can be organized to respect logical nesting. Hierarchical methods can prevent rejecting lower-level hypotheses without support from higher-level context, while still controlling global error rates.

12.2 Multiple endpoints and multivariate testing (overview)

Experiments often test multiple outcomes simultaneously. Multivariate settings require methods that can handle correlations among endpoints and possibly among features. Error control may target families defined by endpoints, by time points, or by combined criteria.

12.3 Time-series and spatial dependence (overview)

In time-series and spatial data, observations are typically correlated across time or location. Multiple testing corrections must account for structured dependence to avoid overly optimistic significance. Approaches may include dependence-aware resampling, block permutations, or model-based methods.

12.4 High-dimensional settings (overview)

High-dimensional problems—where the number of tests can be very large relative to sample size—stress both theoretical assumptions and computational capacity. Methods for FDR and FWER often focus on scalable procedures and on assumptions compatible with the high-dimensional regime.

13 Worked examples (illustrative)

The following examples are illustrative and meant to demonstrate how correction strategies are applied conceptually rather than to provide exhaustive mathematical derivations.

13.1 Illustrative small-scale FWER control

Suppose four hypotheses are tested with unadjusted p-values. A Bonferroni correction would compare each p-value to \(\alpha/4\), whereas Holm’s method would order the p-values and use progressively adjusted thresholds. Both aim to ensure the probability of at least one false rejection remains below \(\alpha\), but Holm typically rejects more hypotheses because it is less conservative when early p-values are small.

13.2 Illustrative FDR control in large-scale testing

Consider a study with many feature-level hypotheses, where the goal is discovery followed by validation. Using BH at a target FDR level \(\alpha\), one orders p-values and finds the largest index satisfying the BH inequality. The resulting rejection set is then interpreted as discoveries whose expected proportion of false discoveries is controlled at the chosen level under applicable assumptions.

13.3 Comparing methods on the same p-value set

Given a fixed set of p-values, different methods can produce different rejection sets. FWER methods like Bonferroni and Holm often reject fewer hypotheses than FDR methods. In contrast, FDR procedures can reject more features, especially when many p-values are moderately small. Comparing outputs helps highlight the practical impact of choosing FWER versus FDR.

13.4 Sensitivity checks with resampling

If dependence is unclear or if p-values might be sensitive to distributional assumptions, sensitivity can be assessed with permutation-based adjustments. By repeating resampling under a null-consistent mechanism, one can verify whether the empirical frequency of extreme rejection patterns aligns with the claimed error control. When discrepancies occur, this indicates that dependence or model assumptions may not be appropriate.