1 Background and Motivation

1.1 Multiple testing and family-wise error rate (FWER)

In many statistical studies, researchers test several hypotheses at once, such as comparing multiple groups or examining many predictors. When hypotheses are tested simultaneously, the chance of making at least one Type I error (rejecting a true null hypothesis) increases beyond the nominal per-test significance level. A common target for controlling this overall risk is the family-wise error rate (FWER), defined as the probability of making one or more Type I errors within a specified family of tests.

1.2 Why simple corrections may be overly conservative

A simple strategy is to apply the same significance threshold to each test after multiplying or scaling it to account for the number of hypotheses. The most well-known example is Bonferroni correction, which ensures FWER control by using an adjusted threshold that is often quite small. While safe, this can substantially reduce statistical power, especially when many hypotheses are tested but only a few are expected to be false nulls, or when the tests exhibit certain favorable structures.

1.3 Relation to controlling Type I error

Holm’s method addresses the same Type I error control objective as Bonferroni—namely limiting FWER—but does so by using the ordered set of p-values to adapt the thresholds. Rather than treating all hypotheses symmetrically with a uniformly strict cutoff, Holm’s procedure “steps down” through increasingly less stringent comparisons as evidence against the null accumulates. This adaptivity typically yields stronger power while retaining the essential error-rate guarantee.

2 Definition of Holm’s Method

2.1 Setup: hypotheses, p-values, and ordering

Consider \(m\) null hypotheses \(H_1,\dots,H_m\), each with an associated p-value \(p_1,\dots,p_m\). Let the p-values be ordered from smallest to largest: \[ p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}, \] where \(p_{(k)}\) is the k-th smallest p-value, corresponding to some hypothesis index. The goal is to decide which hypotheses to reject while ensuring control of the FWER at a chosen level \(\alpha\).

2.2 Step-down adjustment rules

Holm’s method compares each ordered p-value to a sequence of decreasing critical values. Specifically, for \(k = 1,2,\dots,m\), the k-th ordered p-value is tested against \[ \frac{\alpha}{m-k+1}. \] The thresholds are larger early in the sequence and become smaller? In fact, for step-down they start at \(\alpha/m\) for the smallest p-value and then increase for later p-values because the denominator decreases; equivalently, the rejection criteria become less stringent as \(k\) increases, but only after earlier rejections have occurred.

2.3 Decision procedure and stopping criterion

The step-down decision rule is:

  1. Start with the smallest p-value \(p_{(1)}\). If \(p_{(1)} > \alpha/m\), then reject none of the hypotheses.
  2. If \(p_{(1)} \le \alpha/m\), reject \(H\) corresponding to \(p_{(1)}\), then test \(p_{(2)}\) against \(\alpha/(m-1)\).
  3. Continue in this fashion. At the first index \(k\) where

\[ p_{(k)} > \frac{\alpha}{m-k+1}, \] stop and reject all hypotheses corresponding only to \(p_{(1)},\dots,p_{(k-1)}\).

This stopping rule—reject while thresholds are met, halt at the first non-passing p-value—is the core of the method.

2.4 When Holm’s method is applied (typical workflow)

Holm’s method is used after a set of p-values has been computed for a pre-defined family of hypotheses. A typical workflow is:

  • Choose the family of tests (e.g., all pairwise contrasts within a study, or all features selected for screening).
  • Compute each hypothesis’ p-value using the analysis model appropriate to the study design.
  • Order the p-values and apply the step-down comparisons to determine rejected hypotheses.
  • Optionally report adjusted p-values that correspond to the Holm-adjusted significance levels, enabling readers to see at which \(\alpha\) each hypothesis would be rejected.

3 Statistical Properties

3.1 FWER control guarantee

A key property of Holm’s method is strong control of the family-wise error rate under broad conditions. In its standard formulation, it controls FWER at level \(\alpha\) regardless of whether hypotheses are independent or dependent, provided the usual validity of each individual p-value holds (for example, that the p-values are valid under their respective null hypotheses). This guarantee makes the method a widely adopted “default” for FWER-oriented multiple testing.

3.2 Comparison with Bonferroni correction

Bonferroni correction rejects hypotheses whose p-values satisfy \(p_i \le \alpha/m\), typically using the same threshold for every hypothesis. Holm’s method begins with that same strict threshold for the smallest p-value but then allows less conservative thresholds for subsequent p-values only if earlier rejections occurred. Because it uses the observed ordering rather than applying a uniform cutoff, Holm’s procedure is uniformly at least as powerful as Bonferroni under the usual assumptions, meaning it tends to reject more false nulls while maintaining the same FWER control.

3.3 Power characteristics (vs. other FWER methods)

Power in multiple testing depends on the configuration of true and false null hypotheses and on the distribution of p-values under alternatives. Holm’s step-down logic generally improves power compared with procedures that apply a single conservative threshold, particularly when the smallest p-values are sufficiently small to trigger early rejections. Compared with other FWER-controlling methods, Holm is often competitive without requiring special assumptions about dependence. In situations where more assumptions hold, some alternative approaches can be more powerful, but Holm’s method offers a robust balance between error control and practical performance.

3.4 Assumptions and interpretation of results

Interpreting Holm’s results assumes that each p-value is computed from a valid test statistic for its hypothesis and that the chosen family of tests matches the scope of error control desired. Importantly, “controlling FWER” addresses the probability of one or more false rejections in the family, not the expected number of false rejections. As a result, it is possible to reject several hypotheses while still having a controlled probability of at least one mistake; the procedure does not guarantee that every non-rejected hypothesis is truly true, nor that rejected hypotheses are certainly false.

4 Computational Aspects

4.1 Algorithmic implementation

Holm’s method is straightforward to implement:

  • Sort p-values with their hypothesis labels.
  • For each position \(k\), compute the threshold \(\alpha/(m-k+1)\).
  • Reject all hypotheses up to (but not including) the first index where \(p_{(k)}\) exceeds the threshold.

The algorithm relies only on ordering and comparisons, not on iterative optimization.

4.2 Complexity and practical scalability

The computational cost is dominated by sorting the p-values. For \(m\) tests, sorting requires \(O(m \log m)\) time. After sorting, the step-down comparisons require \(O(m)\) additional time. This scaling is typically efficient for modern applications, including feature screening where \(m\) can be large, though very large-scale settings may warrant specialized workflows and careful numerical handling.

4.3 Handling ties and numerical precision issues

In practice, p-values may tie due to discretization, finite sample sizes, or rounding. When ties occur, a consistent ordering rule is needed so that hypotheses with equal p-values are treated coherently. Numerical precision can also affect comparisons near the threshold boundaries. A common approach is to use sufficient floating-point precision and to define comparisons in a way that is stable under small rounding errors (for example, by applying tolerances if the software’s documentation permits).

4.4 Software and function interfaces (conceptual)

Many statistical packages provide functions that compute Holm-adjusted p-values or directly perform Holm step-down rejection. Conceptually, interfaces typically accept:

  • a vector of raw p-values,
  • a significance level \(\alpha\),
  • and optionally the direction or type of testing (one- vs two-sided), depending on how p-values were generated.

Some functions return rejected hypothesis indices; others return adjusted p-values that can be used for flexible reporting and downstream selection.

5.1 Holm–Bonferroni (naming conventions)

Holm’s method is sometimes referred to as the Holm–Bonferroni procedure because it refines Bonferroni’s principle using an ordered step-down approach. The naming emphasizes its relationship to Bonferroni correction and its shared objective of controlling FWER.

5.2 Step-up vs. step-down logic

The defining feature of Holm’s procedure is step-down behavior: rejection decisions proceed from the smallest p-value upward and stop at the first failure. This contrasts with step-up methods, which typically begin with the largest admissible evidence threshold and then may continue rejecting as long as criteria are satisfied in a different order. The two designs can have different power characteristics and can respond differently to patterns in the p-value distribution.

5.3 Connections to Hochberg’s procedure (conceptual comparison)

Hochberg’s procedure is a prominent FWER-controlling method that uses a step-up logic. Conceptually, it can be more powerful than Bonferroni under certain dependence structures, because it attempts to use larger p-values efficiently when strong overall signals are present. Holm’s method is often preferred for robustness because it does not rely on the same dependence conditions as Hochberg’s more aggressive design. Both procedures are frequently discussed together in texts on multiple testing.

6 Applications and Examples

6.1 Multiple comparisons in factorial designs

In factorial experiments, researchers may test main effects and interactions and then conduct multiple post hoc comparisons among factor levels. Holm’s method is used to maintain an overall FWER across the set of related comparisons. For example, if a study examines several pairwise group differences following a model fit, Holm’s step-down thresholds help ensure that the family-wide probability of at least one false rejection does not exceed \(\alpha\).

6.2 Post hoc testing after omnibus tests

An omnibus test (such as an overall ANOVA or global regression test) may be followed by multiple specific comparisons only when the omnibus test indicates potential differences. Holm’s method can be applied to the post hoc set to control error across those follow-up hypotheses. This practice aligns with the general idea of separating the global question from the more granular set of targeted comparisons while maintaining a rigorous error bound for the family of follow-up tests.

6.3 Feature screening with controlled error rates

When screening many features—such as variables in a high-dimensional dataset—one may want to control FWER to limit the chance of false discoveries across the entire family. Holm’s method is relevant when the scientific goal emphasizes “no false alarms” in the set being reported. Although false discovery rate methods are common in very large feature-selection tasks, Holm’s procedure remains appropriate when controlling the probability of any false rejection is the primary objective.

6.4 Worked-out example with ordered p-values

Suppose \(m=5\) hypotheses are tested at \(\alpha=0.05\), producing p-values that order as: \[ p_{(1)}=0.003,\; p_{(2)}=0.012,\; p_{(3)}=0.018,\; p_{(4)}=0.040,\; p_{(5)}=0.070. \] Compute Holm thresholds:

  • For \(k=1\): \(\alpha/5 = 0.01\). Since \(0.003 \le 0.01\), reject \(H_{(1)}\).
  • For \(k=2\): \(\alpha/4 = 0.0125\). Since \(0.012 \le 0.0125\), reject \(H_{(2)}\).
  • For \(k=3\): \(\alpha/3 \approx 0.0167\). Since \(0.018 > 0.0167\), stop.

Therefore, only the hypotheses corresponding to \(p_{(1)}\) and \(p_{(2)}\) are rejected, and the remaining three are not rejected.

7 Extensions and Practical Considerations

7.1 One-sided vs. two-sided testing impacts

Holm’s method operates on p-values, so the key impact of one-sided versus two-sided testing comes from how the p-values were obtained for each hypothesis. Two-sided p-values typically have larger values than one-sided p-values for the same observed statistic under symmetric alternatives, affecting the ordering and thus the rejection sequence. Once valid p-values are produced, the Holm step-down machinery applies uniformly.

7.2 Dependence structure between tests (practical notes)

While the FWER guarantee of Holm’s method is strong in standard settings, real analyses can involve complex dependence created by shared data, common nuisance parameters, or resampling procedures. In practical terms, researchers typically ensure that p-values are valid for the tests used and that the “family” is defined consistently. When p-values come from permutation or resampling methods, dependence may be naturally accommodated, but validity still depends on the resampling scheme matching the testing goal.

7.3 Reporting adjusted p-values (“Holm-adjusted”)

Besides reporting which hypotheses are rejected, it is common to report Holm-adjusted p-values. These are adjusted significance values such that a hypothesis is rejected at level \(\alpha\) if and only if its Holm-adjusted p-value is at most \(\alpha\). Reporting adjusted p-values supports transparency: readers can compare results across different intended significance levels without rerunning the adjustment.

7.4 Common pitfalls and best practices

Common issues include:

  • Misdefining the “family” of tests: Holm controls FWER within the specified set. Expanding or changing the family after analysis can invalidate the intended error control.
  • Using p-values that are not valid under the null: the guarantee depends on the validity of each p-value for its hypothesis.
  • Forgetting to order correctly: the step-down stopping rule depends on the ordered p-values; incorrect handling can change the rejection set.
  • Over-interpreting: FWER control does not quantify effect sizes or the probability that a rejected hypothesis is “true” in an absolute sense.

Best practice is to pre-specify the set of hypotheses in the family, compute valid p-values with a consistent model, apply Holm’s ordering carefully, and report the adjusted results in a way that matches the intended error control objective.