1 Introduction to Multiple Comparisons

1.1 Why multiple testing increases false positives

When many hypotheses are tested at once, the chance of observing at least one false “significant” result grows with the number of tests, even if all null hypotheses are true. This inflation occurs because random variation produces some unusually small p-values purely by chance, and the more opportunities one has to see a small value, the higher the likelihood that at least one crossing occurs.

Multiple testing is common in fields such as genomics, imaging, and large-scale experiments, where analysts routinely screen many candidate effects. The key statistical problem is that nominal significance levels (e.g., 0.05 per test) do not translate into the same overall error probability once tests are combined.

1.2 Family-wise error rate (FWER) and significance thresholds

A standard target for error control is the family-wise error rate (FWER): the probability of making one or more false rejections among a specified set of hypotheses (the “family”). To control FWER at level \(\alpha\), methods adjust significance criteria so that, under assumptions, the overall probability of at least one false positive does not exceed \(\alpha\).

In practice, this is implemented through thresholds for p-values. Because tests are performed simultaneously, these thresholds typically depend on the number of hypotheses and sometimes on the rank (ordering) of each p-value.

1.3 Stepwise correction methods in statistics

Stepwise multiple-comparisons procedures use the ordered structure of p-values rather than treating each hypothesis independently. Broadly, there are:

  • Step-down procedures, which start with the most stringent threshold and relax it as ranks proceed.
  • Step-up procedures, which start with a less stringent threshold and become more permissive as ranks increase.

The Hochberg correction is a step-up method, meaning it leverages the largest p-values that can still be rejected given the ordered thresholds, potentially allowing more discoveries than fully conservative approaches while maintaining the designated FWER control under suitable dependence assumptions.

2 The Hochberg Procedure

2.1 Assumptions and applicability

2.1.1 Independence or positive dependence conditions

The Hochberg procedure is typically described for settings where the test statistics (or equivalently the p-values) satisfy specific dependence properties. A commonly cited condition is independence. In broader formulations, the method can also be valid under forms of positive dependence, which broadly prevent p-values from behaving “too negatively” together.

Because dependence governs how likely it is that multiple p-values simultaneously take extreme values, these assumptions are crucial. If the dependence structure strongly violates the required conditions, FWER may exceed the nominal \(\alpha\), making results less reliable.

2.2 Ordering p-values and step-up logic

Suppose there are \(m\) hypotheses with corresponding p-values \(p_1, p_2, \ldots, p_m\). Order the p-values from smallest to largest: \[ p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}. \] The step-up logic evaluates rejection possibilities starting from the largest p-value that might still be allowed under the Hochberg thresholds. In effect, the procedure seeks the greatest index \(k\) for which the p-value \(p_{(k)}\) remains small enough to justify rejecting all hypotheses with ranks up to \(k\).

2.3 Critical values and decision rule

For a desired FWER level \(\alpha\), Hochberg’s decision rule can be expressed in terms of comparing ordered p-values to scaled thresholds. One standard formulation is:

  • Find the largest \(k\) such that

\[ p_{(k)} \le \frac{\alpha}{m-k+1}. \]

  • Reject hypotheses corresponding to \(p_{(1)}, p_{(2)}, \ldots, p_{(k)}\).

Equivalently, many presentations express the rule as checks applied in descending order from \(p_{(m)}\) downward until a threshold is satisfied; once found, all smaller p-values are rejected as well. This search-and-stop behavior is the defining feature of a step-up method.

2.4 Interpreting results after adjustment

After applying Hochberg correction, the rejected hypotheses are those deemed significant while controlling the FWER at level \(\alpha\) under the method’s assumptions. Interpreting results usually focuses on the set of rejections rather than individual p-values at face value.

Because Hochberg can be less conservative than some alternatives, the resulting significant set may be larger. However, that advantage is contingent on the correctness of assumptions; when dependence assumptions fail, the apparent gain in power may come with a loss of guaranteed error control.

3 Relationship to Other Error-Control Methods

3.1 Comparison with Bonferroni correction

The Bonferroni correction controls FWER by testing each hypothesis at level \(\alpha/m\) (or equivalently by multiplying p-values by \(m\) and capping at 1). This approach is simple and broadly applicable, often requiring minimal assumptions.

Relative to Bonferroni, Hochberg can be less conservative because it uses the ordered information from the p-value ranking. Under appropriate dependence conditions, Hochberg’s step-up structure can yield more rejections than Bonferroni while keeping the FWER controlled.

3.2 Comparison with Holm correction (step-down)

Holm’s method is a step-down procedure that controls FWER and often performs better than Bonferroni. It works by comparing the smallest p-value to the most stringent threshold and then relaxing as necessary.

In contrast, Hochberg is step-up. Depending on the realized p-value configuration and dependence structure, one method may reject more hypotheses than the other. In many practical scenarios, they are close; the differences arise from the directionality of the stepwise logic and how thresholds are engaged.

3.3 Comparison with Benjamini–Hochberg (FDR control)

The Benjamini–Hochberg procedure targets a different error metric: the false discovery rate (FDR), which is the expected proportion of false discoveries among rejected hypotheses. This is not the same as controlling FWER, and it typically allows more rejections.

Although the names are similar, the Hochberg correction discussed here is for FWER control, while the Benjamini–Hochberg method is for FDR control. Analysts must therefore match the procedure to the intended error criterion; using an FDR method when FWER control is required may produce more false positives than allowed under a strict family-wise definition.

3.4 When Hochberg is more or less conservative

Hochberg’s relative conservatism depends on:

  • Dependence conditions: under suitable positive dependence, it can maintain error control while being less conservative.
  • Observed p-value pattern: because the method selects the largest acceptable rank \(k\), certain p-value configurations can lead to more rejections than step-down alternatives.
  • Number of tests: with larger \(m\), the thresholds change, affecting how easily candidates pass early or late comparisons.

If dependence assumptions are not met, Hochberg may become effectively too liberal. Conversely, in some data structures, step-down procedures like Holm can be at least as powerful as Hochberg, making Hochberg no better—or occasionally more conservative—on that specific realization.

4 Practical Implementation

4.1 Input requirements (p-values, number of tests)

Implementation requires:

  • The set of p-values for all \(m\) hypotheses in the family.
  • The target FWER level \(\alpha\).

The method assumes these p-values are computed consistently (e.g., from the same type of test and under a compatible null model). If p-values are not comparable due to differing test forms or unequal null interpretations, the family should be defined carefully.

4.2 Handling ties and numerical stability

In real computations, p-values can coincide exactly due to rounding, discretization, or permutation-based methods. Ties affect the ordering \(p_{(i)}\), but Hochberg’s decision rule typically remains well-defined as long as the implementation treats tied values in a consistent manner.

Numerical stability issues can arise when p-values are extremely small or extremely close to thresholds. A robust implementation should use adequate floating-point precision and careful comparisons. When thresholds are computed as \(\alpha/(m-k+1)\), rounding can affect whether a comparison is considered true.

4.3 Edge cases (e.g., all non-significant p-values)

If none of the ordered p-values satisfy the inequality for any \(k\), then \(k\) is effectively zero and no hypotheses are rejected. This outcome corresponds to all adjusted significance decisions being non-significant under the procedure.

At the other extreme, if even the largest p-value meets the applicable threshold at \(k=m\), then all hypotheses are rejected. These edge cases are straightforward but should be handled cleanly by software to avoid off-by-one indexing errors.

4.4 Computing adjusted p-values vs. threshold-based decisions

Two common workflows exist:

  1. Threshold-based decision: compute \(k\) via the Hochberg rule and reject all hypotheses up to that rank.
  2. Adjusted p-values: produce a corrected p-value for each hypothesis such that rejecting at level \(\alpha\) is equivalent to comparing the adjusted p-value to \(\alpha\).

Adjusted p-values are often useful for reporting and for downstream comparisons at varying significance levels. Many implementations derive them from the step-up structure by identifying, for each rank, the smallest \(\alpha\) that would make the hypothesis rejectable under Hochberg.

5 Theoretical Background

5.1 Controlling FWER: intuition and outline

FWER control means that, under the null configurations considered and the dependence assumptions imposed by the method, the procedure’s chance of producing even a single false rejection is bounded by \(\alpha\).

Intuitively, Hochberg reduces the probability of false positives by incorporating rank information: it does not require every small p-value to pass a harsh threshold; instead, it allows rejection once a sufficiently small p-value appears at a high rank, which still constrains the collective error probability. The formal proof is usually constructed by bounding the probability that the rejection set includes at least one true null, using properties of the ordered p-values under the assumed dependence class.

5.2 Step-up versus step-down behavior

Step-up and step-down procedures distribute their conservatism differently across the ordered p-values. In a step-up method, the procedure searches for a plausible maximum rejection size and then commits to rejecting all smaller p-values. This can yield more power when the set of p-values that correspond to true effects includes at least one value that is sufficiently small at the right rank.

In step-down methods, the procedure begins with stringent comparisons for the smallest p-values and may stop early when conditions are not met. Differences in behavior can be especially visible when the smallest few p-values are not extremely small, but several intermediate p-values may still support additional rejections under step-up logic.

Because Hochberg is driven by ordered p-values, its theoretical development often invokes concepts from order statistics. The distribution of the minimum, maximum, and intermediate order statistics shapes how likely it is for specific ranks to fall below their respective thresholds.

The method’s thresholds are designed to align with these order-statistic behaviors under the null. The result is a calibration where the rank-dependent inequality ensures that the probability of rejecting any true null is controlled at the targeted FWER level.

6 Reporting and Visualization

6.1 How to report adjusted significance findings

Reporting typically includes:

  • The fact that Hochberg correction was applied.
  • The family of hypotheses to which the correction pertains.
  • The target \(\alpha\) value for FWER control.
  • The list (or count) of rejected hypotheses after adjustment.

In many reports, authors also clarify what “significant” means operationally (e.g., rejecting based on Hochberg-adjusted p-values or on the stepwise decision rule).

6.2 Presenting adjusted p-values in tables

A common presentation is a table with columns such as:

  • Hypothesis identifier or variable name
  • Raw p-value
  • Hochberg-adjusted p-value
  • Rejection indicator (yes/no at level \(\alpha\))

Adjusted p-values help readers compare evidence strength across hypotheses without repeatedly re-running the decision procedure for different \(\alpha\) levels.

6.3 Optional visual summaries (e.g., ordered p-value plots)

Visualization can include ordered plots where raw p-values are placed against rank and compared with Hochberg’s threshold curve. Such graphics make it easier to see which rank first satisfies the step-up inequality and where the rejection boundary occurs.

These plots are especially helpful in teaching or exploratory contexts, but they should be used alongside the formal numerical reporting to avoid ambiguity in exact threshold crossings due to plotting resolution.

7 Common Pitfalls and Best Practices

7.1 Using the wrong correction for the wrong target (FWER vs. FDR)

A frequent error is confusing Hochberg’s FWER-oriented correction with FDR-focused procedures bearing similar names. Analysts should verify:

  • Whether the goal is family-wise error control or false discovery rate control.
  • Whether the correction applied matches the intended error metric.

Because the numerical outputs can differ substantially, matching the correction to the target is essential for correct interpretation.

7.2 Violating dependence assumptions

Hochberg’s validity depends on assumptions about independence or positive dependence. Analysts should assess whether their data generation and test construction plausibly satisfy these conditions.

When tests are strongly and negatively dependent, FWER control may not hold. In such situations, it may be preferable to use methods designed for broader dependence robustness, or to re-evaluate the analysis design (e.g., through resampling-based strategies).

7.3 Misunderstanding “family” definitions for tests

The “family” is the set of hypotheses to which the multiple testing correction is applied. Choosing an overly broad family can reduce power, while choosing an overly narrow one can understate the true overall error rate.

Best practice is to predefine the family based on the study question and the analysis plan, and to document the included tests so that FWER statements are meaningful.

7.4 Reproducibility and documentation of correction choice

Reproducibility requires documenting:

  • The correction method name (Hochberg)
  • The error control target (FWER)
  • The value of \(\alpha\)
  • The hypothesis family definition
  • Software version or algorithmic implementation details if relevant

Because different software may handle ties, adjusted p-value conventions, or floating-point comparisons slightly differently, clear documentation helps ensure consistent reanalysis.

8 Variants and Extensions

8.1 Modified Hochberg-style methods under alternative assumptions

Several variants modify Hochberg’s framework to fit different dependence structures or test conditions. The central idea remains rank-based step-up rejection, but the threshold calibration may be adapted to match the dependence model being assumed.

These modifications are often motivated by the desire to retain power while preserving an appropriate error guarantee under less restrictive circumstances than strict independence.

8.2 Extensions to correlated test settings

In practice, test statistics are frequently correlated, especially in multivariate or spatial data. Extensions of Hochberg-type procedures aim to handle correlation more carefully by exploiting structured dependence (e.g., forms of positive regression or other conditions that limit extreme joint tail behavior).

When correlation is accounted for, the procedure can remain informative rather than reverting to overly conservative adjustments.

8.3 Software-specific implementations and conventions

Different statistical packages implement Hochberg correction with their own conventions for:

  • Computing adjusted p-values versus applying a decision rule
  • Treating tied p-values
  • Ensuring monotonicity properties of adjusted p-values
  • Naming and exposing options for dependency assumptions

For reliable interpretation, users should consult the documentation to confirm which Hochberg formulation is implemented and how it aligns with theoretical conditions.