1. Overview of multiple testing and FWER

1.1 Family-wise error rate (FWER) and why it matters

In multiple hypothesis testing, the investigator evaluates many null hypotheses at once. The family-wise error rate (FWER) is the probability of making at least one false rejection among the entire set of hypotheses. Controlling FWER is especially important in settings where a single mistaken “discovery” is costly, or where confirmatory follow-up is intended to be limited to a small set of trustworthy claims.

1.2 Null hypotheses, a family of tests, and multiplicity

A “family” of tests is the collection of hypotheses considered together for the purpose of error control. Multiplicity arises because testing repeatedly inflates the chance of false positives even when each individual test is calibrated at a desired significance level. Procedures that adjust for multiplicity aim to maintain an overall error rate that is meaningful at the family level rather than only for each single test.

1.3 Dependence among test statistics and its impact

Test statistics in practice are often correlated: sharing subjects, measurements, covariates, or underlying structure can induce dependence. This dependence affects how often extreme values occur jointly. Some classical corrections (e.g., those based on worst-case bounds) remain valid under minimal assumptions, but may be conservative. Resampling-based methods, including Westfall–Young, are designed to use the observed dependence structure more directly to achieve sharper control.

2. Core idea of the Westfall–Young framework

2.1 Permutation/resampling under the global null

The Westfall–Young approach estimates the null distribution of multiple-test summaries by resampling in a way that is valid under the global null hypothesis (all nulls true). The method relies on the idea that, under the null, the labels or generating mechanism can be permuted (or otherwise resampled) without changing the distribution of the test statistics in a way that matters for error control.

2.2 Estimating adjusted p-values and critical values

Rather than calibrating each hypothesis separately, Westfall–Young constructs adjusted p-values or a common rejection threshold by comparing observed test statistics to their resampled counterparts. A standard construction uses the resampled distribution of a maximum-type statistic across the family, producing multiplicity-aware cutoffs that control FWER.

2.3 Step-down versus step-up strategies

Westfall–Young procedures can be organized as step-down or step-up methods. Step-down algorithms typically start with the most significant hypothesis (largest evidence against its null) and move to less significant ones, updating critical values to refine which hypotheses can be rejected while maintaining the desired error control. Step-up variants reverse the direction, potentially offering different trade-offs between power and conservativeness under dependence.

2.4 Handling dependence via resampling

The defining feature of the framework is that dependence is handled through the resampling mechanism. By repeatedly generating the entire set of test statistics under the null through the same resampling scheme, the procedure preserves the dependence pattern that occurs under the null. This allows the max-type distribution to reflect correlation among tests, often producing less conservative adjustments than methods that ignore dependence.

3. Statistical ingredients

3.1 Choice of test statistics

The procedure begins by defining a test statistic for each hypothesis. Typical choices depend on the modeling context (e.g., t-statistics in regression, rank-based scores, or other summary measures). The key requirement is that the collection of test statistics is computed for every resampled dataset in the same way as for the original data.

3.2 Test statistics used for max-type adjustments

For max-type adjustments, one common ingredient is the maximum of the collection of (possibly transformed) test statistics across hypotheses. This max statistic is used because controlling the probability of exceeding a threshold for the maximum directly translates into controlling the probability of at least one false rejection. Variants may instead use subset-restricted maxima or other monotone statistics compatible with step-down logic.

3.3 Exchangeability and resampling validity

Resampling validity typically depends on an exchangeability property: under the null, the resampling operation should not systematically bias the joint distribution of test statistics. In permutation tests, exchangeability is often implied by symmetry of labels under the null. In other resampling settings, the justification is analogous: the resampling must mimic the distribution of test statistics that would occur if all null hypotheses were true.

3.4 Implementation details (randomization schemes)

Implementation requires a concrete resampling scheme. For permutations, this could mean shuffling treatment labels, outcome labels, or other components subject to the validity constraints of the null. For bootstrap-like analogs, it could involve resampling subjects or residuals under a fitted model. Each scheme determines what “under the null” means operationally and therefore affects the accuracy and validity of the estimated adjusted p-values.

4. Error rate control guarantees

4.1 Strong FWER control vs. weak control

“Strong” FWER control means the procedure controls FWER at the target level for any configuration of true and false nulls. “Weak” control typically refers to guaranteeing the error rate only when the global null holds (all nulls true). Westfall–Young procedures often aim for strong control, but the guarantees depend on the exact variant and assumptions about dependence and structure.

4.2 Conditions under which control is expected

Control is usually tied to conditions such as monotonicity of the rejection rule with respect to test statistics, and properties of the joint null distribution captured by exchangeability or related assumptions. Some versions rely on subset pivotality or related concepts that ensure the distribution of the relevant maxima behaves consistently when some nulls are false. When these conditions are met, the resampling-based critical values can be shown to yield the desired error control.

4.3 Connections to monotonicity and subset pivotality

Many proofs use monotonicity: if rejection thresholds move in a way that is consistent with increasing evidence against nulls, then the stepwise procedure can maintain control. Subset pivotality (informally: the distribution of test statistics for a subset of hypotheses does not depend on nuisance aspects of the other hypotheses being true or false) is another common technical requirement in the theoretical justification of max-type resampling methods.

5.1 Westfall–Young step-down (maxT) style

A well-known variant uses a step-down strategy with a maxT adjustment. The algorithm orders test statistics and uses resampled maxima to compute increasingly permissive or restrictive cutoffs as it steps through the ordered hypotheses. The step-down nature often yields a set of rejected hypotheses that is coherent with the idea of rejecting only when enough evidence exists even after accounting for multiplicity.

5.2 Westfall–Young step-up variants

Step-up versions also use resampling to form adjusted criteria but differ in the direction of the stepwise logic. These can lead to different power characteristics across scenarios, especially when dependence among tests is nontrivial. As with other stepwise multiple testing methods, the choice between step-down and step-up is usually guided by theoretical properties available for the specific setting and by practical performance.

5.3 Bootstrap-based analogs and alternative resampling

Beyond straightforward permutations, there are bootstrap-based and other resampling analogs that attempt to reproduce the null distribution of the multiple-test summaries. These approaches can be advantageous when permutations are difficult to define or when exchangeability is not naturally satisfied. However, validity depends on whether the bootstrap mechanism properly reflects the null distribution of the test statistics and their dependence structure.

5.4 Comparisons with Bonferroni, Holm, and Hochberg

Bonferroni’s method controls FWER by a simple union bound and often does not exploit dependence; it can be conservative. Holm’s step-down correction improves power over Bonferroni while still controlling FWER under minimal assumptions. Hochberg’s step-up correction can be more powerful under additional dependence conditions (often phrased in terms of positive dependence). Westfall–Young can outperform these classical methods when the dependence structure is accurately captured by resampling and when the theoretical assumptions for resampling-based control are plausibly satisfied.

Conceptually, Westfall–Young belongs to a broader family of resampling-based multiple testing frameworks that build adjusted thresholds from empirical null distributions of maxima or related summaries. Romano–Wolf methods also use resampling to control error rates, sometimes targeting false discovery rate variants or other multi-test metrics. While the exact mathematical targets and assumptions differ, the overarching idea—using resampling to capture joint behavior under the null—creates a conceptual bridge between these approaches.

6. Practical workflow

6.1 Preparing hypotheses and test statistics

The investigator specifies the set of hypotheses and computes a test statistic for each one using the original data. The statistics must be comparable across hypotheses in how they measure evidence against their respective nulls (e.g., all oriented so that larger values indicate stronger evidence against the null).

6.2 Running permutations and storing resampled maxima

Next, the investigator performs many resampling iterations (often permutations), recomputing the entire vector of test statistics each time. For each resample, one stores the maximum-type quantity (or the set of quantities required by the chosen step-down or step-up variant). These stored values become an empirical estimate of the null distribution relevant to multiplicity control.

6.3 Computing adjusted p-values

Adjusted p-values are obtained by comparing each observed statistic to its resampled reference distribution. In maxT-style logic, an adjusted p-value reflects the probability—estimated from resampling—that the maximum across the family would be at least as extreme as what is observed for a particular hypothesis, possibly after accounting for stepwise constraints.

6.4 Selecting a rejection threshold and interpreting results

Finally, the investigator compares adjusted p-values to the target significance level to decide which hypotheses to reject, or uses adjusted critical values to apply the stepwise algorithm. Interpretation emphasizes that rejecting hypotheses should maintain the controlled probability of at least one false rejection within the predefined family, subject to the validity of the resampling assumptions.

7. Computational considerations

7.1 Number of permutations and accuracy trade-offs

Accuracy depends on the number of resampling iterations. With too few permutations, the empirical adjusted p-values can be coarse, especially when the desired significance level is small. Increasing the permutation count reduces Monte Carlo error but increases computation time.

7.2 Efficiency for large numbers of tests

When the family contains many hypotheses, recomputing all test statistics for every resample can be burdensome. Efficiency may rely on optimizing computation of statistics, exploiting matrix operations, or reusing intermediate quantities when possible. Some implementations also store only the needed maxima rather than the full vector of statistics for each resample.

7.3 Parallelization and reproducibility (random seeds)

Resampling iterations are typically embarrassingly parallel. Practical workflows often parallelize over iterations to reduce wall-clock time. For reproducibility, random seeds should be controlled so that the same resampling sequence can be regenerated if needed.

7.4 Memory and runtime considerations

Storing large arrays of resampled statistics can strain memory. In many max-based algorithms, keeping only the maxima (or the stepwise critical summaries) for each iteration is sufficient. Runtime and memory balance depends on the number of hypotheses, the number of permutations, and the computational cost of recomputing test statistics.

8. Worked example (high-level)

8.1 Setting up a multiple testing problem

Consider a scenario where several related features are tested simultaneously for association with an outcome. Each hypothesis corresponds to a feature, and a test statistic is computed per feature, producing an evidence score across the family.

8.2 Performing the resampling steps

The procedure generates resampled datasets under the global null using an exchangeable or permutation-respecting scheme. For each resample, all per-feature test statistics are recalculated, and the maximum-type statistic across features is recorded.

8.3 Obtaining adjusted p-values and decisions

The empirical distribution of the resampled maxima is then used to compute adjusted p-values for each observed feature statistic (or to apply a step-down rule). Features with adjusted p-values below the chosen significance level are rejected, yielding a set of claims with controlled FWER.

8.4 Reporting results transparently

Reporting typically includes the family definition, the test statistics used, the resampling scheme, the number of resamples, and the achieved control target (the nominal FWER level). Transparent disclosure helps readers assess whether assumptions for resampling validity are plausible and whether Monte Carlo accuracy is sufficient.

9. Assumptions, pitfalls, and diagnostics

9.1 When exchangeability may fail

Exchangeability can fail when the resampling scheme changes the data-generating mechanism in a way that is not compatible with the null. For example, if resampling breaks a structural constraint that would remain under the null, the estimated null distribution may be distorted, jeopardizing error control.

9.2 Effects of small permutation counts

With very few permutations, adjusted p-values may take only a limited set of values. This can lead to unstable decisions near the threshold and may understate uncertainty about the achieved FWER.

9.3 Discrete test statistics and ties

Some tests yield discrete statistics or produce ties. The handling of ties in the computation of adjusted p-values (e.g., using “greater than or equal to” comparisons and potential continuity corrections) influences the resulting p-values and can slightly alter conservativeness.

9.4 Sensitivity to dependence structure mismatches

Even when individual resampling is reasonable, the dependence structure among test statistics is crucial for max-based adjustments. If the resampling mechanism does not reproduce the correct dependence under the null, the empirical max distribution may not match reality, leading to miscalibration.

9.5 Common reporting mistakes (what to disclose)

Common shortcomings include not stating the family of hypotheses, omitting the exact resampling scheme, using too few resamples without acknowledging resolution limits, and failing to describe how step-down or step-up decisions were computed. Clear documentation is important for reproducibility and for evaluating the credibility of error control.

10. Extensions and modern applications

10.1 High-dimensional testing contexts (general)

In high-dimensional problems—where the number of hypotheses can be large—resampling-based multiple testing remains attractive because it can adapt to dependence. However, computational cost becomes a dominant concern, motivating careful implementation, reduced statistics computation, or more efficient resampling variants.

10.2 Adaptive choices of statistics and their implications

Investigators sometimes consider data-driven transformations or statistic choices. Such adaptivity can complicate theoretical guarantees because exchangeability and monotonicity assumptions may no longer hold exactly. When adaptations are used, they are typically designed to preserve the resampling validity conditions as much as possible.

10.3 Software and implementation ecosystems (overview)

Multiple statistical software ecosystems provide implementations of Westfall–Young-type procedures, often exposing parameters for the resampling scheme, test statistics, and stepwise strategy. Practical selection depends on compatibility with the data structure (e.g., permutation feasibility) and transparency of the underlying algorithm.

10.4 Guidance for selecting a resampling-based method

Selection guidance typically emphasizes verifying that the resampling scheme is justified under the null, choosing enough resamples for stability, and matching the procedure variant to the desired balance of error control strength and power. Where theoretical conditions are uncertain, comparing with classical corrections and conducting sensitivity checks can help assess robustness.