1 Introduction

1.1 Multiple testing and the need for FWER control

In many statistical studies, researchers evaluate a collection of hypotheses rather than a single question. When multiple tests are performed, the probability of making at least one false rejection can increase beyond the nominal level used for an individual test. A common target for controlling this inflation is the family-wise error rate (FWER), defined as the probability of rejecting one or more true null hypotheses within a specified family of tests. Procedures designed for FWER aim to keep that probability bounded by a prechosen significance level.

1.2 Relationship to Holm’s and Šidák’s methods

The Holm–Šidák variant is a hybrid construction that blends the stepwise logic of Holm’s method with the error-rate control motivation behind Šidák-type adjustments. Holm’s procedure uses a step-down sequence of increasingly permissive critical values for the ordered p-values, producing strong control under minimal assumptions. Šidák’s approach traditionally arises from multiplicative reasoning for independent tests and yields a different style of threshold adjustment. Holm–Šidák incorporates these ideas so that its sequential thresholds are tuned to achieve FWER control under dependence settings that are stronger than those required for Bonferroni, while retaining a stepwise rejection mechanism rather than a single global adjustment.

1.3 When the variant is typically applied

Holm–Šidák is typically considered in contexts where investigators want FWER control for multiple hypotheses and are willing to adopt dependence assumptions under which Šidák-type arguments are valid or approximately valid. It is often used when practitioners need p-value adjustments that are tighter than conservative Bonferroni-style corrections but still provide formal error guarantees. It is most natural in studies with a moderate-to-large number of hypotheses where ranking p-values and performing a sequential check is straightforward.

2 Statistical Setup

2.1 Hypotheses and test statistics

Suppose a family of \(m\) hypotheses \(H_{1},\dots,H_{m}\) is tested simultaneously. Each hypothesis is associated with a test statistic whose distribution under the null provides a p-value. The procedure does not require a specific form of test statistics; it only uses the resulting p-values (or equivalently, test rejection indicators based on critical values).

2.2 Family-wise error rate (FWER) definition

Let \(V\) denote the number of true null hypotheses that are rejected. The family-wise error rate is \[ \mathrm{FWER}=\Pr(V\ge 1). \] A multiple-comparisons procedure is said to control FWER at level \(\alpha\) if \(\mathrm{FWER}\le \alpha\) under the stated model and dependence assumptions.

2.3 Assumptions about dependence among tests

The validity of FWER control depends on how the test statistics relate to one another. Some procedures provide “strong control,” meaning the guarantee holds regardless of which subset of hypotheses are true. Other methods require assumptions about dependence, such as independence or certain positive dependence structures. In Holm–Šidák constructions, the thresholds are designed so that the probability of at least one false rejection is kept below \(\alpha\) under the dependence regime for which the underlying Šidák-type calibration is appropriate.

2.4 Input quantities: p-values and significance levels

The typical inputs are:

  • a significance level \(\alpha\) (the target FWER control level),
  • the collection of p-values \(p_{1},\dots,p_{m}\).

The procedure then orders p-values and compares them to a sequence of thresholds derived from \(\alpha\) and the number of remaining hypotheses at each step.

3 Stepwise Rejection Mechanism

3.1 Ordering of p-values

Let \(p_{(1)}\le \cdots \le p_{(m)}\) denote the ordered p-values, and let \(H_{(i)}\) be the hypothesis corresponding to \(p_{(i)}\). The stepwise rule checks these p-values from smallest to largest (or equivalently uses a step-down structure on the ordered list) to determine how many hypotheses can be rejected while preserving the desired FWER property.

3.2 Step-down Holm–Šidák thresholds

At each position \(i\), the method uses a threshold that depends on \(\alpha\) and the remaining number of tests. Conceptually, the Holm component enforces a sequential safeguard: if a particular p-value is not small enough, larger p-values will not be rejected either. The Šidák element modifies the functional form of the threshold to reflect multiplicative calibration associated with dependence compatible with Šidák-type control.

A common representation expresses the step-down critical values as an increasing sequence in \(i\), so that earlier rejections are harder to justify and later decisions become progressively less stringent.

3.3 Stopping rule and decision logic

Define the largest index \(k\) such that the ordered p-values satisfy the corresponding threshold inequalities for all \(i\le k\). Then the procedure rejects exactly the hypotheses \(H_{(1)},\dots,H_{(k)}\). If the smallest p-value fails its first comparison, then \(k=0\) and no hypotheses are rejected. This stopping rule is the core mechanism that yields the stepwise error control.

3.4 Equivalent formulation in terms of adjusted significance levels

Instead of checking p-values directly, one can express the same decisions as adjusted per-comparison significance levels. For each ordered p-value \(p_{(i)}\), there is a level \(\alpha_{i}\) such that comparing \(p_{(i)}\) to its threshold is equivalent to comparing the p-value to that adjusted level. This equivalence is useful for interpretation and for connecting the procedure to adjusted-p-value reporting.

4 Adjusted p-values (Holm–Šidák p-value construction)

4.1 Definition of adjusted p-values

Adjusted p-values provide an interpretable scale for reporting multiple-testing results. For each hypothesis \(H_{j}\) with p-value \(p_{j}\), the Holm–Šidák adjusted p-value \(p^{*}_{j}\) is constructed so that rejecting \(H_{j}\) at FWER level \(\alpha\) is equivalent to requiring \(p^{*}_{j}\le \alpha\). Typically, adjusted values are defined first on the ordered p-values and then mapped back to the original hypothesis labels.

4.2 Monotonicity and how adjusted values are computed

The adjusted p-values are defined to preserve monotonic behavior: if \(p_{(i)}\le p_{(i+1)}\), then the corresponding adjusted values should not decrease. In practice, the construction uses the same thresholds as in the stepwise rule, with the adjusted p-value at position \(i\) obtained by applying a transformation of \(\alpha\) that inverts the threshold inequality. Because the step-down nature can induce non-monotonic raw computations, the final adjusted values are usually enforced via a monotone envelope (e.g., taking minima over suffix indices) so that the reported adjusted p-values are coherent with the sequential decision logic.

4.3 Interpreting adjusted p-values versus thresholds

An adjusted p-value \(p^{*}_{j}\) can be interpreted as the smallest FWER level \(\alpha\) at which hypothesis \(H_{j}\) would be rejected by the Holm–Šidák procedure. This interpretation is distinct from the original p-value, which measures evidence against a single null hypothesis in isolation. Adjusted p-values therefore incorporate the multiplicity correction and the stepwise structure.

4.4 Comparison to unadjusted p-values

Unadjusted p-values often appear more “optimistic” because they ignore the multiplicity of tests. The Holm–Šidák correction generally increases p-values or at least does not decrease them, reflecting the need to protect against false discoveries at the family level. For highly significant p-values, the adjustment may be modest; for borderline results, adjusted values can increase substantially, potentially changing which hypotheses are rejected at a fixed \(\alpha\).

5 Theoretical Properties

5.1 FWER control guarantees

The primary theoretical guarantee is that the procedure controls FWER at level \(\alpha\) under the dependence assumptions under which its calibration is justified. The step-down structure supports this by ensuring that once a p-value fails a threshold, all later hypotheses are automatically retained, preventing error accumulation that would occur under naive unadjusted testing.

5.2 Strong vs. weak control considerations

A procedure offers strong FWER control if the bound holds for every configuration of true and false null hypotheses. Many stepwise FWER procedures are designed to provide this stronger property, but achieving it depends on the exact form of thresholds and the dependence model. Holm-type methods are often favored for strong control under broad conditions, while Holm–Šidák variants may require additional assumptions to match the strength of Holm’s guarantee.

5.3 Role of dependence structure in validity

Dependence affects how likely it is that many test statistics simultaneously appear extreme. In Šidák-type reasoning, independence (or a closely related positive dependence condition) enables multiplicative probability bounds. If tests are more negatively dependent than allowed by the method’s assumptions, the thresholds may no longer deliver the advertised FWER level. As a result, theoretical properties should be interpreted in conjunction with how the procedure’s dependence criteria map onto the study at hand.

5.4 Consistency and limiting behavior (high-level)

At a high level, stepwise FWER methods are expected to behave consistently in regimes where sample sizes grow and p-values under false nulls tend to 0 while those under true nulls remain stochastically bounded. Under such asymptotic behavior, the number of rejections typically increases for non-null effects while the probability of any erroneous rejection remains controlled. The precise limiting statements depend on assumptions about effect sizes, test validity, and dependence.

6 Practical Implementation

6.1 Algorithmic pseudocode

A generic outline uses ordered p-values and sequential threshold comparisons:

  1. Input p-values \(p_{1},\dots,p_{m}\) and target level \(\alpha\).
  2. Sort p-values to obtain \(p_{(1)}\le \cdots \le p_{(m)}\).
  3. For \(i=1,\dots,m\), compute the step-down threshold \(t_{i}\) specified by the Holm–Šidák calibration.
  4. Find the smallest index \(i\) where \(p_{(i)} > t_{i}\); if no such index exists, set \(k=m\). Otherwise set \(k=i-1\).
  5. Reject hypotheses \(H_{(1)},\dots,H_{(k)}\). Optionally compute adjusted p-values using the inverted/monotonized thresholds.

6.2 Computational complexity and scaling with number of tests

The dominant cost is sorting the p-values, which is typically \(O(m\log m)\). Threshold computation and the sequential comparison are linear in \(m\). For large-scale testing problems, this overhead is small relative to the cost of generating p-values from the underlying statistical models.

6.3 Common pitfalls (e.g., ties, one-sided vs two-sided tests)

Several implementation issues can affect correctness:

  • Ties in p-values: Sorting with ties is fine, but adjusted p-values must follow the monotonic construction consistently across tied values.
  • One-sided vs two-sided tests: P-values must be on the same scale and correspond to the same tail convention used by the method. Mixing one-sided and two-sided p-values can distort the intended error calibration.
  • P-value validity: The procedure assumes that each p-value is properly calibrated under its null (i.e., valid marginally). Miscomputed p-values can break FWER guarantees.

6.4 Software availability and usage notes

Holm–Šidák adjustments may be available in statistical software packages that support multiple-testing corrections beyond standard Bonferroni/Holm. When supported, usage typically requires specifying the procedure name (or parameters matching the Holm–Šidák definition) and indicating whether the correction should be applied to one-sided or two-sided p-values. If a software implementation uses a specific dependence model or variant definition, the documentation should be checked to ensure it matches the intended error control setting.

7.1 Versus Holm (step-down Bonferroni)

Holm’s method uses step-down thresholds derived from a Bonferroni-type guarantee. Holm–Šidák can yield less conservative thresholds under suitable dependence conditions, potentially increasing power while still protecting FWER. However, Holm is often considered more robust to dependence deviations because Bonferroni-style bounds do not rely on the same independence structure as Šidák-type calibrations.

7.2 Versus Šidák (single-step approach)

Šidák’s single-step procedure applies a global threshold to all hypotheses simultaneously (after ordering is not required for the basic idea). The stepwise Holm–Šidák variant generally provides a more flexible decision rule by allowing early stopping: if some hypotheses fail to meet stringent initial thresholds, later hypotheses are not automatically rejected. This typically improves practical decision-making compared with a rigid one-shot threshold.

7.3 Versus Hochberg and other step-up methods

Step-up methods, such as Hochberg-type procedures, reject hypotheses starting from the largest p-values that could still be considered significant under an error constraint. Holm–Šidák is step-down, which can behave differently depending on the distribution of p-values under the null and alternative. In practice, step-up procedures may be more powerful under certain dependence and sparsity patterns, while step-down procedures provide a conservative monotonic safeguard.

7.4 Versus Bonferroni and Holm–Bonferroni variants

Bonferroni uniformly applies \(\alpha/m\) as a threshold to each p-value and is often conservative, especially when many tests are likely to be null. Holm–Bonferroni (the classical Holm method) improves power by replacing the single threshold with a stepwise sequence. Holm–Šidák is positioned as an alternative stepwise correction that incorporates Šidák-type calibration, aiming for tighter thresholds under appropriate conditions.

8 Worked Example

8.1 Example data and multiple hypotheses

Consider \(m=5\) hypotheses with unadjusted p-values: \[ p = (0.012,\ 0.031,\ 0.041,\ 0.090,\ 0.200). \] Assume the target FWER level is \(\alpha=0.05\). Order the p-values to identify their ranks. Here they are already ordered as \[ p_{(1)}=0.012,\ p_{(2)}=0.031,\ p_{(3)}=0.041,\ p_{(4)}=0.090,\ p_{(5)}=0.200. \]

8.2 Computing ordered p-values

With the ordered list in hand, the next step is to compute the Holm–Šidák thresholds \(t_{1},\dots,t_{5}\). The exact numeric thresholds depend on the specific Holm–Šidák calibration formula used by the procedure definition in the source being followed.

8.3 Applying Holm–Šidák thresholds step-by-step

Using the computed thresholds, compare sequentially:

  • Check whether \(p_{(1)} \le t_{1}\). If yes, proceed.
  • Check whether \(p_{(2)} \le t_{2}\). If yes, proceed.
  • Continue until encountering the first index \(i\) such that \(p_{(i)} > t_{i}\).

Suppose (illustratively) that the first three comparisons succeed while the fourth fails: \[ p_{(1)} \le t_{1},\quad p_{(2)} \le t_{2},\quad p_{(3)} \le t_{3},\quad p_{(4)} > t_{4}. \] Then the stopping rule yields \(k=3\).

8.4 Determining the final set of rejected hypotheses

The rejected set is \(\{H_{(1)},H_{(2)},H_{(3)}\}\), corresponding to the three smallest p-values (0.012, 0.031, 0.041). The hypotheses associated with \(p_{(4)}\) and \(p_{(5)}\) are not rejected at FWER level \(\alpha=0.05\).

9 Extensions and Variants

9.1 Variations in threshold parameterization

Different presentations of Holm–Šidák may parameterize the thresholds slightly differently depending on how the Šidák adjustment is embedded into the step-down sequence. These differences can affect the exact functional form of the per-step critical values and, consequently, the adjusted p-values. Users should match the variant to the definition implemented in their chosen reference or software.

Holm–Šidák sits within a broader family of stepwise FWER controls that translate between ordered p-value comparisons and monotone adjusted p-values. Many related methods share a common skeleton—order, compare sequentially, stop, and optionally compute adjusted p-values—while differing in how thresholds are derived. This makes them comparable in behavior and in implementation patterns.

9.3 Extensions to special test structures (overview)

In some settings, tests have additional structure, such as grouped hypotheses, hierarchical models, or constraints that induce specific dependence patterns. While a full development depends on the statistical framework, Holm–Šidák-like logic can sometimes be extended by redefining the effective number of hypotheses or by applying the correction within structured families (e.g., within groups), provided the resulting error guarantees remain valid under the assumed dependence model.

9.4 Notes on adaptation to different testing frameworks

The method requires p-values that are valid for their null hypotheses. If p-values come from resampling (e.g., permutation) or from asymptotic approximations, the validity should be assessed in the context of multiple testing. When p-values are estimated from complex models, care is needed to ensure that the p-values meet the assumptions needed for the stated FWER control. Under heavy model misspecification, the formal guarantees may not translate into practice.

10 Summary

10.1 Key takeaways on error control and procedure behavior

Holm–Šidák is a stepwise multiple-comparisons procedure designed to control the family-wise error rate while leveraging a Šidák-type calibration inside Holm’s step-down framework. Its sequential stopping rule rejects a prefix of the ordered p-values, and its adjusted p-values enable direct reporting at any target FWER level.

10.2 When Holm–Šidák is a good choice

It is a good choice when one wants FWER control with corrections that can be less conservative than Bonferroni-based step-down methods, and when the dependence structure among tests aligns reasonably well with the conditions under which Šidák-type bounds are justified. It is particularly attractive when the analysis naturally yields p-values for each hypothesis and a ranked, stepwise decision is easy to implement.

10.3 Guidance for reporting results in multiple testing contexts

In reporting, practitioners commonly present both the rejected hypotheses and either the Holm–Šidák adjusted p-values or the fact that a given hypothesis is significant at FWER level \(\alpha\). The dependence assumptions or the intended validity conditions for the method should be documented when relevant, and comparisons against unadjusted p-values can help clarify how multiplicity influenced the final conclusions.