1 Multiple Hypothesis Testing Motivation

1.1 Why p-values Need Adjustment

A p-value quantifies how incompatible observed data are with a null hypothesis for a single test. When many hypotheses are evaluated on the same data set, the probability of observing at least one small p-value purely by chance increases. Adjusted p-values address this by modifying the evidence metric so that the reported significance reflects the fact that multiple comparisons were conducted.

1.2 Error Rates: FWER vs FDR

Different scientific goals motivate different error targets. The family-wise error rate (FWER) controls the probability of making one or more false rejections within a specified family of tests. The false discovery rate (FDR) controls the expected proportion of false rejections among all rejections. These two quantities emphasize different tolerances: FWER is strict about any mistake, while FDR allows some false positives in exchange for higher power.

1.3 Families of Tests and Scope Choices

Adjustment depends on how the tests are grouped into a “family.” For instance, all hypotheses tested within a single study analysis might form the family, but a narrower or broader scope could be chosen. The choice of family is not purely mathematical; it reflects the experimental design, pre-specified analysis plan, and what the analyst considers comparable comparisons.

2 Definitions and Notation

2.1 Raw p-values and Test Statistics

Consider m statistical tests producing raw p-values p₁, p₂, …, p_m. Each p_i is obtained from a test statistic T_i using a reference distribution under the null model for that test. Raw p-values are typically computed independently or under an assumed dependency structure, but the adjustment step explicitly targets the multiple-testing setting.

2.2 “Adjusted” as a Function of Multiple Comparisons

An adjusted p-value for test i is a mapping from the collection of raw p-values to a new set that accounts for multiplicity. Depending on the method, the adjustment can be monotone in p_i (so smaller raw p-values do not become larger adjusted values) and can target a specified error rate. In practice, adjusted p-values are often produced by converting an underlying rejection rule into a p-value-like quantity.

2.3 Common Terms: q-value and Corrected p-value

Terminology varies by community. “Corrected p-value” typically refers to an FWER-style correction applied to raw p-values. “q-value” is often used for FDR-controlled procedures, representing the minimum FDR level at which a given hypothesis would be rejected under the method.

3 Adjustment Methods (FWER Control)

3.1 Bonferroni Correction

The Bonferroni correction controls FWER by scaling significance thresholds or by equivalently transforming p-values. If an overall level α is desired across m tests, each test is compared to α/m. As an adjusted p-value representation, a common form is p_i,adj = min(1, m p_i). This yields a conservative correction because it treats all tests as if they could produce false positives in the worst-case manner.

3.1.1 Assumptions and Conservativeness

Bonferroni’s validity does not require independence among tests, which contributes to its broad applicability. The conservativeness arises because it can overestimate the joint chance of false rejections, particularly when tests are positively correlated or share structure that reduces the likelihood of simultaneous extreme outcomes.

3.2 Holm–Bonferroni Method

The Holm–Bonferroni procedure improves on Bonferroni by using a step-down strategy. Raw p-values are ordered from smallest to largest: p_(1) ≤ p_(2) ≤ … ≤ p_(m). Starting with p_(1), the method compares p_(k) to α/(m − k + 1). If p_(k) is larger than its threshold, the procedure stops; larger p-values are not rejected.

3.2.1 Step-down Procedure Intuition

The step-down logic gradually relaxes thresholds as the ordered p-values increase, but only if earlier hypotheses were sufficiently strong to justify continuing. This typically increases power over plain Bonferroni while still controlling FWER under broad conditions.

Hochberg’s procedure is a step-up alternative for FWER control. It orders p-values from largest to smallest and begins with the least stringent comparison. For ordered values p_(m), p_(m−1), …, p_(1), the procedure finds the largest index k such that p_(k) is below α/(m − k + 1), then rejects all hypotheses with p-values at most p_(k).

3.3.1 When Step-up Variants Apply

Step-up methods can be more powerful than step-down methods but may require additional assumptions about dependence among test statistics. A typical requirement is control under certain positive dependence structures (often phrased in terms of monotone likelihood ratio or similar conditions, depending on the formulation). When these conditions hold, step-up behavior can provide a favorable balance of error control and sensitivity.

4 Adjustment Methods (FDR Control)

4.1 Benjamini–Hochberg (BH) Procedure

The Benjamini–Hochberg procedure controls FDR at level q (often written α in software). After sorting p-values in ascending order p_(1) ≤ … ≤ p_(m), BH computes the largest index k such that p_(k) ≤ (k/m) q. It then rejects the hypotheses corresponding to p_(1), …, p_(k). BH can be expressed as providing adjusted p-values that correspond to these thresholds.

4.1.1 Rank-Based Thresholding

The BH rule uses the rank of each p-value rather than a uniform cutoff. This means that hypotheses with better (smaller) p-values receive less stringent thresholds, while weaker signals face stronger requirements. The approach is designed to manage the expected fraction of false discoveries among all rejections.

4.2 Benjamini–Yekutieli (BY) Procedure

The Benjamini–Yekutieli method extends FDR control to settings with arbitrary dependence by using a more conservative scaling of the thresholds. BY retains the same general rank-based structure as BH but replaces q with q / c(m), where c(m) is a harmonic-like factor that depends on the number of tests.

4.2.1 Dependency Considerations

BY is often used when dependence structure is complex or unknown and the analyst wants a guarantee that FDR control holds without requiring special positive dependence assumptions. The trade-off is typically reduced power compared with BH.

4.3 Storey’s Procedure and Estimating π0

Storey’s approach targets FDR control by estimating the proportion π0 of true null hypotheses among the m tests. If π0 is less than 1, one can use more permissive thresholds than standard BH while maintaining FDR control, because the expected number of false rejections can be reduced under sparsity.

4.3.1 q-values from π0 Estimates

In Storey-type workflows, q-values can be computed using π0 estimates derived from the distribution of p-values (for example, the mass near 1). The resulting q-value for each hypothesis reflects the estimated minimal FDR level at which the hypothesis would be called significant.

5 Selecting a Significance Threshold

5.1 Using Adjusted p-values Directly

Once adjusted p-values are available, a common workflow is to select a target error level (FWER α or FDR q) and declare hypotheses significant when the corresponding adjusted p-value is less than or equal to that level. Because many adjusted p-value constructions are monotone in the raw p-value, this rule is typically aligned with the underlying rejection mechanism.

5.2 Mapping Between α Levels and Adjusted Outputs

Adjusted p-values can be interpreted as “how much” multiple-testing allowance is required before a rejection becomes acceptable. For FDR methods, q-values can be compared to an FDR target directly; for FWER methods, adjusted p-values can be compared to a family-wise α. The mapping is method-specific but generally consistent with the method’s threshold rule.

5.3 Interpreting “Significant” Results with Adjustments

A significant adjusted p-value indicates that—under the assumptions and within the chosen family—the error control criterion is satisfied at the specified level. It does not mean the null is true or false with certainty, nor does it provide the posterior probability of a hypothesis. Instead, it summarizes evidence while accounting for the multiplicity of tests.

6 Assumptions, Dependence, and Validity

6.1 Independence vs Positive Dependance

Many multiple-testing results depend on the relationship among test statistics. Some procedures remain valid under independence; others require forms of positive dependence to preserve error control. When dependence violates these conditions, the advertised control may no longer hold exactly.

6.2 Effect of Correlation Among Tests

Correlation among tests affects both the distribution of p-values and the performance of rejection rules. Positive correlation can sometimes make extreme p-values less likely to occur simultaneously, which may reduce false rejections and increase power. Negative dependence can have the opposite effect, potentially increasing the probability of crossing significance thresholds.

6.3 Practical Diagnostics and Robustness

Because the true dependence structure is rarely known perfectly, analysts often rely on robustness heuristics. These may include using more conservative procedures when dependence is suspected to be arbitrary, checking stability across data splits or bootstrap resampling, and verifying that results do not rely on a small subset of correlated features.

7 Computational and Implementation Considerations

7.1 Software Workflows and Defaults

Most statistical software provides functions to compute adjusted p-values for standard methods such as Bonferroni, Holm, BH, and BY. Default settings can vary by package, so implementations should be checked for the chosen family definition, method parameters, and whether the reported adjusted values correspond to the same rejection logic described in the method.

7.2 Handling Missing Values and Ties

Real datasets may yield missing p-values due to incomplete covariates, convergence issues, or invalid model fits. Handling missingness typically requires excluding those tests consistently within the family or applying a predefined imputation/repair strategy. Ties in p-values can arise from discrete tests or rounding; many algorithms specify deterministic tie handling to preserve monotonicity.

7.3 Performance for Large-Scale Testing

When m is very large (e.g., high-throughput experiments), sorting and vector operations dominate runtime. Efficient implementations avoid unnecessary re-computation and exploit rank-based calculations. Memory management and batching can be important when storing intermediate quantities such as ordered indices or π0-related estimates.

8 Reporting and Interpretation in Practice

8.1 How to Present Results in Tables

Common reporting formats include columns for the raw p-value, the adjusted p-value (or q-value), test statistic, and effect estimate. Clear labeling of the adjustment method and the error control target is essential. For readability, some reports also include a boolean “rejected” indicator based on the chosen cutoff.

8.2 Distinguishing Discovery vs Confirmation

In many research workflows, adjusted results are treated as discoveries that guide further validation. Confirmation may involve follow-up experiments, independent datasets, or pre-registered analyses. Adjusted p-values address multiplicity within the analyzed family, but they do not by themselves guarantee reproducibility across new studies.

8.3 Common Misinterpretations to Avoid

A frequent misunderstanding is treating adjusted p-values as if they were probabilities that the null hypothesis is true. Another is using adjusted significance without correctly defining the test family or applying the method to a subset after seeing results in a way that changes the multiplicity implicitly. Adjusted p-values also should not be compared across studies without accounting for differences in methods, family choices, and dependence patterns.

9 Worked Examples

9.1 Small Demonstration with Bonferroni

Suppose four tests yield raw p-values: 0.01, 0.03, 0.20, and 0.60. With Bonferroni correction over m = 4, adjusted p-values become

  • 4 × 0.01 = 0.04,
  • 4 × 0.03 = 0.12,
  • 4 × 0.20 = 0.80,
  • 4 × 0.60 = 2.40 → truncated to 1.00.

If the family-wise level is α = 0.05, only the first test remains significant (adjusted p-value ≤ 0.05).

9.2 Multiple Tests with BH FDR Control

Consider m = 10 tests with target FDR q = 0.10. After sorting p-values, suppose the largest index satisfying p_(k) ≤ (k/10) × 0.10 is k = 3. The BH procedure then rejects the three hypotheses with the smallest p-values. In an adjusted p-value representation, each of those three tests would have adjusted p-values at most 0.10, while the remaining tests would exceed the q target.

9.3 Comparing Methods and Outcomes

In a typical comparison, Bonferroni and Holm control FWER and therefore are often more conservative than BH for the same nominal level, especially when many tests are performed. BH may yield more rejections because it targets the expected proportion of false discoveries rather than a strict “no false positives” guarantee. If dependence is suspected to be arbitrary, BY can be used instead, often producing fewer rejections than BH but with stronger dependence-free guarantees.

10 Relationship to Other Correction Concepts

10.1 Confidence Intervals After Adjustment

Multiplicity adjustments are not limited to hypothesis tests. Confidence intervals can be constructed to maintain coverage properties under selection or multiple comparisons, using methods such as simultaneous inference or interval families aligned with the error control goal. These intervals provide a range estimate while accounting for the fact that many hypotheses were examined.

10.2 Multiplicity in Regression and ANOVA Contexts

In regression and analysis of variance workflows, multiplicity appears when testing multiple coefficients, contrasts, or group effects. Adjusted p-values can be applied to the resulting set of tests, provided the family of contrasts is clearly defined. Often, the structure of the model and the design matrices influence the dependence among test statistics.

10.3 Contrast with Effect Size Corrections

Effect sizes measure magnitude rather than evidence against a null. While some fields apply bias corrections to effect estimates (for example, due to small-sample bias), these are conceptually distinct from p-value adjustments. Multiple-testing procedures adjust the inferential threshold or evidence metric for decisions; effect size corrections adjust estimation properties of the magnitude itself.