1 Motivation and Multiple Testing Context

1.1 What “familywise error rate” means

In multiple hypothesis testing, investigators often test many null hypotheses in parallel. The familywise error rate (FWER) is the probability that at least one null hypothesis is incorrectly rejected. Controlling FWER is therefore aimed at limiting the chance of making any false discoveries within the entire collection (the “family”) of tests.

1.2 Why unadjusted testing inflates false discoveries

If each hypothesis is tested at a nominal level (for example, 0.05) without adjustment, the likelihood of encountering at least one false rejection increases as the number of hypotheses grows. This occurs because repeated testing introduces multiple opportunities for random fluctuations to appear statistically significant, even when all null hypotheses are true.

1.3 Comparing common correction goals (FWER vs FDR)

Correction methods can be designed to control different error metrics. FWER control targets the probability of one or more errors across the full set. A related objective is control of the false discovery rate (FDR), which limits the expected proportion of false rejections among the rejected hypotheses. FWER control is typically more stringent and less permissive than FDR control, trading reduced false positives for potentially lower power.

2 Core Idea of the Holm Correction

2.1 Step-down structure overview

The Holm correction, also called Holm’s step-down method, implements a sequentially stricter evaluation of hypotheses. Starting from the most significant p-value, it compares each ordered p-value to a threshold that becomes progressively less stringent as the procedure moves down the ranking. This structure protects against false rejections while allowing more discoveries than uniformly conservative rules.

2.2 Ordering p-values and associated hypotheses

Let there be \(m\) hypotheses with corresponding p-values. The Holm procedure sorts the p-values in nondecreasing order: \[ p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}. \] Each ordered p-value \(p_{(i)}\) corresponds to a particular hypothesis in the sorted list. The method then evaluates these hypotheses in that same order.

2.3 Decision rule using adjusted thresholds

Holm’s decision rule uses an adjusted set of significance levels. When the ordered p-value \(p_{(i)}\) is tested, it is compared to a threshold \(\alpha/(m-i+1)\). Hypotheses are rejected only while the comparisons continue to satisfy the criterion. Once a p-value fails its threshold, the procedure stops rejecting further hypotheses in the ordered list.

2.4 Relation to Bonferroni (connection and improvement)

Bonferroni correction applies the same global adjustment to every hypothesis, using the uniform threshold \(\alpha/m\). Holm can be viewed as an improvement on Bonferroni because it uses a sequence of thresholds: the earliest tests are guarded with stringent criteria, and later tests—if earlier ones pass—receive relatively less conservative thresholds. Under standard conditions, Holm maintains strong FWER control while typically offering greater power than Bonferroni.

3 Mathematical Formulation

3.1 Hypothesis setup and notation

Consider testing \(m\) null hypotheses \(H_1,\dots,H_m\) with an overall significance level \(\alpha\). Each hypothesis \(H_j\) has an associated p-value \(p_j\). After sorting, the ordered p-values are denoted \(p_{(1)},\dots,p_{(m)}\), where \(p_{(1)}\) is the smallest.

3.2 Adjusted significance thresholds (α sequence)

Define the sequence of adjusted thresholds: \[ \alpha_i = \frac{\alpha}{m-i+1}, \quad i=1,\dots,m. \] These thresholds satisfy \(\alpha_1 \le \alpha_2 \le \cdots \le \alpha_m\), meaning the test becomes less strict as \(i\) increases.

3.3 Algorithmic description (ordered testing)

Algorithmically, Holm’s procedure proceeds as follows:

  1. Sort the p-values to obtain the order \(p_{(1)} \le \cdots \le p_{(m)}\).
  2. For each \(i=1,\dots,m\), compute the adjusted threshold \(\alpha_i = \alpha/(m-i+1)\).
  3. Compare \(p_{(i)}\) with \(\alpha_i\) in increasing order of \(i\).

3.4 Stopping rule and rejection set

Define \(k\) as the largest index such that for all \(i \le k\), \[ p_{(i)} \le \alpha_i. \] Then the rejection set consists of the hypotheses associated with the ordered p-values \(p_{(1)},\dots,p_{(k)}\). If the very first comparison fails (i.e., \(p_{(1)} > \alpha_1\)), then no hypotheses are rejected.

4 Properties and Statistical Guarantees

4.1 Strong control of FWER

A central property of Holm’s method is its strong control of the familywise error rate: it controls the probability of making one or more false rejections not only when all null hypotheses are true (the “global” null) but also for any configuration of true and false null hypotheses. This robustness is one reason it is widely used in practice.

4.2 Assumptions and typical conditions

Strong FWER control for Holm’s step-down procedure is often stated under general assumptions about the validity of the individual p-values, such as that each p-value is valid in the sense of being stochastically not smaller than it would be under its null. Many presentations also frame guarantees in terms of nonnegatively dependent test statistics or broadly valid p-value constructions, depending on the exact testing framework.

4.3 When Holm is guaranteed to be valid

The procedure’s error control is guaranteed when the p-values satisfy the standard requirements needed for multiplicity procedures: they must be computed so that, under each true null hypothesis, the resulting p-value behaves appropriately (e.g., has the correct super-uniform property). In addition, the theoretical guarantees for Holm are typically formulated for testing setups where p-values are derived from tests that preserve the conditions underlying FWER control.

4.4 Power considerations versus other methods

Because Holm uses a data-adaptive sequence of thresholds—becoming less conservative after early rejections—it usually has higher power than Bonferroni for the same target FWER. Relative to some step-up and other refined procedures, its power can vary with the dependence structure among tests and the distribution of true and false alternatives. In many common scenarios, however, Holm offers a practical balance of reliability and sensitivity.

5 Practical Implementation

5.1 Step-by-step example with numeric p-values

Suppose \(m=5\) hypotheses are tested at overall level \(\alpha=0.05\), yielding p-values: \[ 0.01,\; 0.03,\; 0.04,\; 0.20,\; 0.30. \] Sorted p-values are already in order. The adjusted thresholds are: \[ \alpha_1=0.05/5=0.01,\quad \alpha_2=0.05/4=0.0125,\quad \alpha_3=0.05/3\approx 0.0167,\quad \alpha_4=0.05/2=0.025,\quad \alpha_5=0.05/1=0.05. \] Comparisons:

  • \(p_{(1)}=0.01 \le 0.01\): reject.
  • \(p_{(2)}=0.03 > 0.0125\): stop and do not reject the remaining hypotheses.

Final result: only the first hypothesis is rejected.

5.2 Handling ties and edge cases

If a p-value equals its corresponding threshold (\(p_{(i)}=\alpha_i\)), the usual convention is to treat it as satisfying the inequality and thus continue rejecting up to that point. Ties among p-values do not affect validity because the procedure depends on the ordered sequence; implementations typically break ties consistently but any tie-ordering that preserves nondecreasing order yields the same set of accepted/rejected indices under standard inequality comparisons.

Edge cases include \(m=1\), where Holm reduces to the ordinary single-test rule at level \(\alpha\), and situations where many p-values are identical, in which case the same stopping point logic determines how many hypotheses are rejected.

5.3 Software workflow (input/output expectations)

In statistical software, Holm correction is typically implemented as:

  1. Input: a vector of p-values and the desired overall \(\alpha\) (or a significance level).
  2. Processing: sort p-values, compute the Holm thresholds, apply the step-down stopping rule.
  3. Output: either a logical rejection indicator for each hypothesis at level \(\alpha\), and/or Holm-adjusted p-values (“adjusted significance levels”) that can be compared to \(\alpha\).

The “adjusted p-values” representation is often convenient because it allows one to decide rejection for any \(\alpha\) after a single computation.

5.4 Interpreting adjusted p-values

Holm-adjusted p-values are constructed so that rejecting hypotheses whose adjusted p-values are \(\le \alpha\) reproduces the same rejection set as the original step-down procedure. Conceptually, an adjusted p-value answers: “At what overall significance level would this hypothesis be rejected by the Holm method?” This interpretation supports clear reporting and comparison across studies.

6.1 Holm–Šidák variant (overview-level comparison)

A closely related approach is the Holm–Šidák procedure, which modifies the threshold sequence using Šidák-type adjustments rather than Bonferroni-style linear thresholds. It is designed to control FWER under certain dependence assumptions and can be viewed as an alternative family of stepwise methods that share Holm’s ordered, step-down character.

6.2 Hommel method (brief relationship)

The Hommel procedure is another stepwise multiplicity method aimed at strong FWER control. It can be more powerful than Holm in some settings, particularly when there is information suggesting certain patterns in the ordering of p-values. Like Holm, it belongs to the broader category of FWER-controlling procedures but uses a different combination of local and global comparisons.

6.3 Hochberg procedure (step-up comparison)

The Hochberg procedure is a step-up alternative that, unlike Holm’s step-down design, considers hypotheses from the largest p-value toward the smallest. Under specific dependence conditions, Hochberg can be more powerful than Holm while maintaining strong FWER control. Because the procedure’s direction differs, it can behave differently when the true alternatives are relatively strong or relatively weak.

6.4 Transition to FDR methods (e.g., Benjamini–Hochberg)

For applications where controlling the false discovery rate is preferable to strict control of the probability of any false rejection, FDR procedures such as the Benjamini–Hochberg method are commonly used. These methods generally permit more rejections by tolerating a controlled proportion of false positives among discoveries. While Holm focuses on FWER, FDR methods address a distinct statistical goal and are typically chosen based on the scientific context and acceptable risk profile.