1 Family-wise error rate (FWER) basics

1.1 Definition and interpretation

1.1.1 Family of hypotheses and “at least one” error

In multiple hypothesis testing, researchers consider several hypotheses together rather than in isolation. A *family* is a prespecified collection of hypotheses treated as one decision unit. The family-wise error rate summarizes the risk across that entire collection: it is the probability of producing one or more false rejections within the family.

This “one-or-more” structure reflects the idea that even a single false positive may be consequential. For example, if each hypothesis corresponds to a candidate effect that could trigger a costly follow-up, the analyst may prioritize preventing any erroneous discoveries over allowing a limited number.

1.1.2 Relationship to Type I error

A Type I error occurs when a true null hypothesis is incorrectly rejected. FWER aggregates these events across the family. Under the usual definition, it targets the probability that *at least one* true null is rejected, meaning at least one Type I error occurs somewhere in the set. As a result, FWER is a way to control the chance of making any Type I mistake within the family, rather than controlling Type I error separately for each individual test.

1.2 Notation and probability statement

1.2.1 Global null scenario

Let the family contain \(m\) hypotheses \(H_1,\dots,H_m\), each with an associated hypothesis test and p-value. Let \(V\) denote the number of false rejections (equivalently, the number of rejected hypotheses whose nulls are actually true). In the global null scenario, all null hypotheses are true, so any rejection is a Type I error. Then \(V\ge 1\) is equivalent to rejecting at least one hypothesis.

The family-wise error rate under this scenario is commonly written as \[ \mathrm{FWER} = \Pr(V \ge 1). \] Because every null is true in the global null, controlling \(\Pr(V\ge 1)\) ensures control of the probability of at least one false discovery when all effects are absent.

1.2.2 General (partial null) scenario

In practice, not all null hypotheses are necessarily true. For a general configuration, some subset of nulls may be true while others are false. FWER is still defined as the probability of at least one false rejection among the true nulls: \[ \mathrm{FWER} = \Pr(V \ge 1), \] where \(V\) counts false rejections only among those nulls that are actually true. This formulation highlights that the risk depends on which hypotheses are truly null, not merely on the number of hypotheses tested.

1.3 Strong vs. weak control

1.3.1 Strong FWER control under all configurations

A procedure offers *strong* control of FWER if it controls \(\Pr(V\ge 1)\) at a specified level \(\alpha\) for every possible configuration of true and false null hypotheses. In other words, it works both when all nulls are true and when some nulls are false.

Strong control is often the practical requirement in applied settings because analysts typically cannot assume that all tested effects are absent. Methods such as Holm’s step-down procedure are designed to provide this type of guarantee under broad conditions.

1.3.2 Weak FWER control under the complete null

A procedure provides *weak* control if it controls FWER at level \(\alpha\) only under the complete null configuration (all hypotheses are true). Under partial alternatives—where some nulls are false—the error probability may exceed \(\alpha\).

Weak control can still be useful in narrowly defined scenarios, but it is generally less reassuring when the analyst expects a mix of true and false nulls.

2 Controlling FWER in practice

2.1 Stepwise adjustment methods overview

2.1.1 Bonferroni correction

The Bonferroni correction controls FWER by adjusting the significance level used for each test. If the family contains \(m\) hypotheses and the target FWER is \(\alpha\), the method compares each individual p-value \(p_i\) to \(\alpha/m\). Equivalently, one rejects \(H_i\) when \(p_i \le \alpha/m\).

Bonferroni is straightforward and widely used because it does not require complex assumptions about dependence between tests. Its main trade-off is conservativeness: tightening each threshold can reduce power, especially when \(m\) is large.

2.1.2 Holm’s step-down procedure

Holm’s method is a stepwise refinement of Bonferroni. First, sort the p-values: \[ p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}. \] Then test sequentially from the smallest p-value, comparing \(p_{(j)}\) to \(\alpha/(m-j+1)\). Starting at \(j=1\), once a comparison fails (i.e., the current p-value exceeds its threshold), no further hypotheses are rejected.

This ordering logic yields greater power than Bonferroni in many settings while retaining strong FWER control for typical p-value validity conditions.

2.1.3 Hochberg’s step-up procedure (when applicable)

Hochberg’s procedure is a complementary stepwise approach that rejects hypotheses starting from the largest p-values among those not yet rejected, using thresholds that become less stringent as the process steps upward. It is typically applicable when certain dependence structures hold (often summarized by positive dependence assumptions), rather than under arbitrary dependence.

When those conditions are met, Hochberg can be more powerful than Holm because it allows earlier rejections guided by larger p-values that still pass sufficiently strict criteria.

2.2 Choosing which p-value to compare

2.2.1 Adjusted significance level vs. adjusted p-values

Two equivalent implementation styles are commonly used. One can adjust the significance thresholds (e.g., \(\alpha/m\), \(\alpha/(m-j+1)\)) and compare each raw p-value to its corresponding threshold. Alternatively, one can compute *adjusted p-values* that translate the multiplicity correction directly into a new p-value scale; each hypothesis is rejected if its adjusted p-value is at most \(\alpha\).

Adjusted p-values are often convenient for reporting because they summarize the correction in a single number per hypothesis and support later re-interpretation under different \(\alpha\) levels.

2.2.2 Critical values and ordering logic

Stepwise methods depend on ordering. For Bonferroni, there is no ordering: all hypotheses use the same threshold. For Holm and Hochberg, the thresholds vary with rank, so which p-values are compared at each stage depends on the sorted arrangement.

This creates a decision rule that is sensitive to how many hypotheses appear promising (small p-values). The resulting rejection set is not necessarily the same as what would be obtained by applying a single fixed threshold to all p-values.

2.3 Assumptions and dependence considerations

2.3.1 Independence vs. arbitrary dependence

Some procedures guarantee strong FWER control regardless of dependence among test statistics, while others rely on specific dependence properties. Bonferroni and Holm are frequently described as robust in the sense that they do not require independence assumptions to maintain FWER control at their target level, making them safer defaults when the dependence structure is complex or unknown.

Procedures like Hochberg may require conditions that effectively ensure that test statistics are not “too negatively correlated” in ways that could inflate the chance of false rejection.

2.3.2 Conservativeness and power trade-offs

“Conservativeness” refers to how much below the nominal \(\alpha\) a method’s true error probability can sit. More conservative methods provide stronger protection but may reject fewer false nulls, reducing power.

Dependence patterns and the number of hypotheses influence this balance: with large \(m\), Bonferroni-type corrections may become substantially restrictive, prompting analysts to consider alternative error criteria or more efficient FWER methods when dependence assumptions are credible.

3.1 FWER vs. false discovery rate (FDR)

3.1.1 How the two criteria differ conceptually

FWER controls the probability of making *any* false discovery: it is a stringent, event-based criterion focused on the event \(V\ge 1\). FDR controls the *expected proportion* of false discoveries among rejected hypotheses, typically formulated as \(\mathbb{E}[V/R]\) where \(R\) is the number of rejections.

Because FDR allows some false positives as long as their proportion remains controlled on average, it can be less conservative than FWER when many hypotheses are tested.

3.1.2 When each control is preferred

FWER is often preferred when the cost of even a single false positive is high, or when follow-up actions are extremely targeted. FDR is more common in exploratory or large-scale screening contexts where some false positives are tolerable and the main goal is to enrich discoveries with a controlled contamination level.

The choice depends on the practical consequences of errors and the expected number of true signals.

3.2 FWER vs. per-comparison error rate (PCER)

3.2.1 Separate testing vs. family-level control

Per-comparison error rate describes error control for each individual comparison or, in some formulations, the probability of a false positive for a test taken singly. In contrast, FWER addresses the family-level probability that at least one false rejection occurs somewhere in the set.

Thus, PCER can be thought of as “local” protection, whereas FWER provides “global” protection across the entire group of hypotheses.

3.2.2 Practical implications in study design

If researchers apply PCER-like thresholds independently without accounting for the multiplicity, the overall probability of at least one false positive can grow quickly with the number of hypotheses. FWER-based procedures mitigate this escalation by explicitly incorporating the family structure into the decision rule.

This distinction influences how analysts define the family and how they plan downstream interpretation and follow-up.

3.3.1 FWER variants (e.g., generalized approaches)

Several variants of FWER exist, including generalized notions that alter how the error event is aggregated. These can accommodate different weighting schemes, broader “family” definitions, or alternative ways to define error in structured testing problems.

Even when the core aim remains “avoid false rejections,” the precise mathematical target can differ, producing procedures that are tuned to specific scientific goals.

3.3.2 Error control in multiple testing frameworks

FWER control is one element of the broader literature on multiple testing. Modern frameworks often combine multiplicity adjustment with hierarchical testing, resampling-based calibration, or modeling of test statistics. Such approaches aim to maintain valid error control while improving efficiency or enabling more complex experimental designs.

Although the mathematical details vary, the guiding concept remains similar: multiplicity must be accounted for to prevent inflated error rates.

4 Worked examples and intuition

4.1 Simple two-hypothesis illustration

4.1.1 Computing FWER for small families

Consider two hypotheses tested at an unadjusted level \(\alpha\). Let \(E_1\) be the event of rejecting a true \(H_1\), and \(E_2\) be the event of rejecting a true \(H_2\). If both nulls are true, then \(V\ge 1\) occurs exactly when \(E_1\cup E_2\) occurs. Therefore, \[ \mathrm{FWER}=\Pr(E_1\cup E_2)=\Pr(E_1)+\Pr(E_2)-\Pr(E_1\cap E_2). \] If the tests are independent and each has Type I error probability \(\alpha\), then \(\Pr(E_1\cap E_2)=\alpha^2\), giving \(\mathrm{FWER}=2\alpha-\alpha^2\). This exceeds \(\alpha\) for any \(\alpha\in(0,1)\), illustrating why multiplicity adjustment is needed.

4.1.2 Comparing unadjusted vs. adjusted testing

With Bonferroni correction for \(m=2\), each test uses threshold \(\alpha/2\). In the global null with independent tests, \[ \mathrm{FWER} = 2(\alpha/2) - (\alpha/2)^2 = \alpha - \alpha^2/4 \le \alpha. \] The inequality reflects conservativeness: FWER is at most \(\alpha\) rather than exactly equal. Holm’s method, applied to two hypotheses, produces the same rejection behavior as using these rank-dependent thresholds and typically retains the family-level protection.

4.2 Multi-hypothesis scenario

4.2.1 Applying Bonferroni/Holm step logic

Suppose a family of \(m\) hypotheses yields sorted p-values \(p_{(1)},\dots,p_{(m)}\). Under Bonferroni, one rejects every hypothesis whose p-value is no larger than \(\alpha/m\). Under Holm, one compares \(p_{(1)}\) to \(\alpha/m\); if it passes, then \(p_{(2)}\) is compared to \(\alpha/(m-1)\), and so on.

The step-down nature matters: even if some later p-values are small, rejection cannot continue once a p-value fails its threshold. This guards against overstating evidence when the early evidence is insufficient.

4.2.2 Interpreting adjusted results

After applying an FWER-controlling method, each rejected hypothesis can be interpreted as having been selected while maintaining the controlled probability of at least one false rejection in the entire family. Adjusted p-values (if computed) indicate how strongly each hypothesis clears the multiplicity-adjusted bar.

In reporting, the key interpretive point is that the correction is tied to the family definition: the same hypothesis may have different adjusted significance if grouped into a different family.

4.3 Visual and conceptual aids

4.3.1 Thresholding and “union of events” view

A helpful conceptual picture treats each individual hypothesis test as defining an event of “rejecting a true null.” FWER then corresponds to the probability of the union of these events across the family. Methods like Bonferroni enforce a bound on that union probability by ensuring the thresholds are low enough that even the worst-case accumulation of risks stays under \(\alpha\).

This “union of events” viewpoint aligns with the mathematical structure \(V\ge 1\), reinforcing why FWER is inherently more stringent than proportion-based criteria.

4.3.2 Power considerations under control

Power refers to the probability of correctly rejecting false null hypotheses. FWER control generally reduces power compared with naive unadjusted testing, especially when many hypotheses are tested. However, stepwise refinements (notably Holm, and sometimes Hochberg under assumptions) can recover power by using thresholds that are less strict for later ranks or by exploiting information in the ordering of p-values.

Thus, intuition about power often depends on both the number of hypotheses and how many are expected to show real effects.

5 Implementation and reporting

5.1 Software and workflow

5.1.1 Obtaining p-values and defining the family

Implementation starts with valid p-values for each hypothesis. These p-values typically come from statistical tests computed under a specified model and inference scheme. Before correction, analysts must explicitly define the hypothesis family—the collection of hypotheses whose multiplicity will be controlled together.

A common source of error is ambiguity in family definition; different choices lead to different adjusted thresholds and different error guarantees.

5.1.2 Selecting an FWER-controlling method

Method selection depends on the dependence structure assumptions that can be justified, the number of hypotheses, and the desired balance between conservativeness and power. Bonferroni is broadly applicable and easy to use when dependence is unknown. Holm is often chosen for improved power without requiring strong additional assumptions. Hochberg can be attractive under positive-dependence scenarios, though the analyst must be able to defend those conditions.

5.2 Reporting conventions

5.2.1 Stating the error-control target

Reports typically state the target error rate (e.g., “FWER controlled at \(\alpha=0.05\)”) and the specific method used (e.g., Bonferroni or Holm). This makes it clear that the probability of at least one false rejection within the defined family is controlled at the nominal level.

5.2.2 Communicating adjusted p-values and decisions

For each hypothesis, results are often presented as raw p-values along with adjusted p-values, or as decisions made under the multiplicity-adjusted rule. When adjusted p-values are provided, readers can assess the margin by which each hypothesis passes the corrected threshold.

Equally important is clarifying the family definition so that readers can interpret what the multiplicity correction actually covered.

5.3 Common pitfalls

5.3.1 Misdefining the hypothesis family

If the family is defined after looking at results, or if additional hypotheses are implicitly included in interpretation without being included in the correction, the nominal FWER guarantee may no longer apply. The method’s validity hinges on the family being determined according to a prespecified plan.

5.3.2 Overlooking dependence or conservative choices

Analysts may either ignore dependence assumptions needed by a particular method or choose overly conservative approaches without considering the impact on power. While conservative methods are safer for error control, excessive conservatism can obscure real effects. Conversely, using a more powerful procedure without the required conditions can lead to incorrect error control.