1 Concept and motivation

1.1 Multiple hypothesis testing problem

In many statistical studies, researchers do not test a single hypothesis. Instead, they evaluate many candidate hypotheses simultaneously—for example, testing each variable in a dataset, comparing multiple groups, or assessing many possible predictors in a model. When the same nominal significance level (such as 0.05) is applied to every test, some false rejections become likely simply by chance. The Bonferroni correction addresses this by tightening the decision threshold for each individual test.

1.2 Family-wise error rate (FWER)

A central quantity in multiple-comparisons control is the family-wise error rate (FWER), defined as the probability of making at least one false rejection among a prespecified set of hypotheses (the “family”). For instance, if none of the null hypotheses are true, FWER measures how likely the procedure is to still produce any significant result. Bonferroni correction is designed to control FWER at a chosen overall level.

1.3 Why per-test thresholds must change

The need to modify per-test thresholds arises because each test contributes to the overall chance of error. If there are \(m\) tests and each is conducted at level \(\alpha\) without adjustment, the probability of observing at least one significant result increases with \(m\). Bonferroni’s key idea is to replace the common threshold \(\alpha\) with a smaller “adjusted” threshold so that the family-level error rate stays at or below \(\alpha\).

2 Bonferroni correction method

2.1 Basic formula

Let there be \(m\) hypotheses tested simultaneously, and let the desired overall significance level be \(\alpha\). The Bonferroni correction uses an individual-test threshold \[ \alpha_{\text{Bonf}}=\frac{\alpha}{m}. \] Under this rule, a null hypothesis is rejected only if its p-value is less than or equal to \(\alpha/m\).

An equivalent way to report the adjustment is through p-values: each unadjusted p-value \(p_i\) can be transformed to \[ p_i^{\text{adj}}=\min(1,\; m\,p_i), \] and hypotheses are rejected if \(p_i^{\text{adj}}\le \alpha\).

2.1.1 Deriving the adjusted alpha level

The derivation relies on a general probability bound. If \(V\) is the indicator of the event that at least one null hypothesis is falsely rejected, then \[ \Pr(V=1)=\Pr\left(\bigcup_{i=1}^m \{\text{reject }H_i \text{ when }H_i \text{ is true}\}\right). \] By the union bound (Boole’s inequality), \[ \Pr(V=1)\le \sum_{i=1}^m \Pr(\text{reject }H_i \text{ when }H_i \text{ is true}). \] If each true null is tested at level \(\alpha/m\), then each term in the sum is at most \(\alpha/m\), giving \[ \Pr(V=1)\le m\cdot(\alpha/m)=\alpha. \] Thus, choosing \(\alpha_{\text{Bonf}}=\alpha/m\) guarantees FWER control at level \(\alpha\) under minimal assumptions about dependence.

2.2 Interpretation of adjusted p-values

Adjusted p-values convey how strong a result must be to withstand the multiple-testing procedure. An adjusted p-value \(p_i^{\text{adj}}=\min(1, mp_i)\) can be interpreted as the smallest overall significance level \(\alpha\) at which that hypothesis would be rejected using Bonferroni. Because the adjustment multiplies p-values by \(m\), results that are only marginally significant at the unadjusted level may become non-significant after correction.

2.3 Relationship to hypothesis testing frameworks

2.3.1 Single-step vs direct correction

Bonferroni is commonly implemented as a single-step method: all adjusted thresholds are computed from \(m\), and the decision rule is applied uniformly across the family. While the correction can be framed as “directly adjusting alpha” or “directly adjusting p-values,” both produce the same rejection set when using the threshold-and-compare logic described above.

3 Implementation details

3.1 Choosing the number of tests (m)

The parameter \(m\) is the number of hypotheses in the family. Correct specification matters: using too small an \(m\) can under-control the error rate, while using too large an \(m\) can unnecessarily reduce power. In practice, \(m\) may reflect planned comparisons (predefined in a protocol) rather than all possible exploratory checks performed after seeing data.

3.2 Handling different test statistics

Bonferroni can be applied regardless of the type of test statistic, as long as each hypothesis produces a valid p-value for testing its null. This flexibility means it can be used with t-tests, z-tests, chi-square tests, likelihood-ratio-derived tests, and other standard procedures. The method’s central requirement is that p-values correspond to each individual test’s null hypothesis.

3.3 One-sided vs two-sided tests

If hypotheses are tested in a one-sided manner, the p-values used in the correction should reflect that directionality. For two-sided tests, p-values typically account for departures in either direction. The correction itself does not decide one- or two-sidedness; that choice belongs to the individual hypothesis tests, while Bonferroni adjusts the resulting p-values or thresholds consistently.

3.4 Special cases (e.g., ties and discrete data)

With continuous test statistics, ties in p-values are uncommon, but in discrete settings (such as small-sample counts or rank-based statistics), p-values can take on only a limited set of values. Discreteness can make p-values conservative relative to exact uniformity under the null, which affects error rates. Bonferroni still provides valid FWER control via the union bound, but the actual achieved FWER may be below the target \(\alpha\), sometimes substantially. Handling ties typically does not require special machinery beyond computing the p-values correctly, though careful reporting is needed when many p-values coincide.

4 Properties and limitations

4.1 Conservativeness and statistical power

A known characteristic of Bonferroni is conservativeness: by using a worst-case bound, it ensures FWER control even when dependence among tests could be favorable. The main cost is reduced statistical power, especially when \(m\) is large or when true effects are modest. In such scenarios, the tightened thresholds may prevent detection of real signals, increasing false negatives.

4.2 Dependence between tests

Unlike some alternative procedures that rely on particular dependence structures, Bonferroni does not require independence to control FWER. This robustness is a strength: even with strong correlations between tests, the union-bound argument still yields validity. However, because the method is conservative under many dependence patterns, the gain in robustness can translate into a larger power penalty than methods tailored to dependence.

4.3 When Bonferroni performs well

Bonferroni tends to be attractive when the number of comparisons is modest, when the stakes of any false discovery are high, or when researchers want guaranteed error control without complex assumptions. It also performs well in settings where the hypothesis family is clearly defined and tests are heterogeneous, because a uniform p-value adjustment is straightforward to apply.

4.4 When alternative methods may be preferable

When the number of tests is very large, the conservativeness can become severe. Alternatives may offer better power while still addressing multiple testing. For example, procedures controlling FWER with improved efficiency under certain conditions, or methods controlling the false discovery rate (FDR), can be preferable when it is acceptable to allow some false positives in exchange for greater sensitivity.

5.1 Holm–Bonferroni procedure

The Holm–Bonferroni method improves on Bonferroni while still controlling FWER. It uses a step-down strategy: p-values are ordered, and each is compared to a threshold that becomes less stringent as one proceeds through the ordered list. In many practical cases, Holm’s approach yields more rejections than the basic Bonferroni rule while maintaining the same family-wise guarantee.

5.2 Hochberg method

The Hochberg procedure is another step-up method related to Bonferroni. It typically offers additional power compared with Holm and Bonferroni under specific assumptions about dependence. Like Holm, it organizes tests by p-value and then applies thresholds in a manner that can produce more discoveries than the single-step Bonferroni approach.

5.3 Šidák correction

The Šidák correction is also derived from controlling the probability of false rejections, but it uses a different mathematical relationship that depends on the assumption of independence among tests. Under independence, Šidák can be less conservative than Bonferroni because it effectively accounts for the joint distribution more directly. When independence is not plausible, Bonferroni’s assumption-minimal validity is often favored.

5.4 False discovery rate (FDR) approaches

FDR-oriented methods, such as the Benjamini–Hochberg procedure, control the expected proportion of false rejections among all rejections. This changes the target quantity from “no false rejections with high probability” (FWER) to “a tolerable fraction of false positives on average.” As a result, FDR methods often have higher power in large-scale testing, though they do not guarantee that the probability of any false discovery remains below a fixed level.

6 Practical examples

6.1 Example: many independent t-tests

Suppose a researcher tests ten independent t-statistics for differences between treatment and control across ten features. If the overall significance level is \(\alpha=0.05\), then Bonferroni uses \(\alpha/10=0.005\) for each individual test. A feature with unadjusted p-value \(0.004\) would be significant after correction, while one with p-value \(0.02\) would not, even though \(0.02<0.05\) for an unadjusted analysis.

6.2 Example: comparing several group means

Consider comparing \(k\) group means pairwise after an initial screening. If all pairwise comparisons are treated as a single family, then the number of tests is \(m=\binom{k}{2}\). For instance, with \(k=5\) groups there are \(m=10\) pairwise comparisons. Bonferroni would set the per-comparison threshold to \(0.05/10=0.005\). This approach prevents a scenario where multiple pairwise contrasts collectively produce at least one apparent difference purely due to chance.

6.3 Example: post-hoc testing after ANOVA

In ANOVA, an overall omnibus test evaluates whether any group means differ, but it does not identify which groups. Post-hoc pairwise tests are often used afterward. When performing these comparisons, the family should typically include all post-hoc contrasts you intend to interpret together. If you conduct six pairwise comparisons at the \(\alpha=0.05\) family level, Bonferroni sets the per-comparison threshold to \(0.05/6\approx 0.0083\).

6.4 Example: regression coefficient significance checks

In a multiple regression model with several predictors, a common practice is to test coefficients for significance. If there are \(m\) regression coefficients being tested (excluding an intercept if desired), Bonferroni can adjust p-values for those \(m\) tests. For example, testing 8 predictors at an overall level \(0.05\) yields an individual threshold of \(0.00625\). This guards against concluding that some coefficients differ from zero when the model’s predictors are correlated and the search across multiple coefficients increases the likelihood of spurious significance.

7 Reporting and best practices

7.1 Communicating adjusted results

Reports should state that Bonferroni correction was used, specify the family-wise significance level \(\alpha\), and clearly indicate how \(m\) was determined. Tables of results can present both unadjusted and adjusted p-values, along with the final significance decision. Including the adjustment formula or a brief description helps readers interpret why some effects may fail to reach significance.

7.2 Choosing an overall significance level

The overall \(\alpha\) represents the tolerated probability of at least one false rejection among the chosen family. Common defaults like 0.05 may be used, but the choice should reflect the study context and the consequences of false positives. If the analysis includes different sets of hypotheses for different questions, it may be appropriate to apply different families and thus different \(m\) values.

7.3 Effect of correction on interpretation

After Bonferroni adjustment, a “significant” result means the hypothesis survived a stricter threshold that controls FWER. Researchers should avoid interpreting adjusted significance as evidence of effect size or practical importance; it only addresses the statistical error criterion within the multiple-testing framework. Conversely, non-significant adjusted results do not imply the absence of effects, especially given the power loss that can accompany large numbers of comparisons.

7.4 Common pitfalls and troubleshooting

A frequent mistake is using an incorrect \(m\), such as counting tests that were not part of the planned hypothesis family or failing to include all comparisons intended for interpretation. Another pitfall is mixing families—for example, applying Bonferroni across unrelated sets of hypotheses or treating an exploratory search as if it were a single planned family. Troubleshooting typically involves re-defining the hypothesis family, confirming that p-values correspond to the intended one- or two-sided tests, and verifying that adjusted p-values are computed consistently as \(m p_i\) capped at 1.