1 Motivation and Basic Definitions

1.1 Null and alternative hypotheses

In hypothesis testing, the null hypothesis typically represents a baseline claim, such as “the treatment has no effect,” while the alternative hypothesis specifies a departure from that baseline, such as “the treatment increases the mean by a specified amount.” The choice of these hypotheses determines what counts as evidence against the null.

1.2 Type I error (significance level) and Type II error

A Type I error occurs when the test rejects the null hypothesis even though the null is true. The probability of a Type I error is controlled by the significance level, often denoted by \(\alpha\). A Type II error occurs when the test fails to reject the null while the alternative (or, more precisely, a specified data-generating mechanism) is true, with probability denoted by \(\beta\).

1.3 Definition of power as 1 − Type II error

Power is the probability of correctly rejecting the null hypothesis when the stated alternative is true. Under a particular alternative parameter value and design assumptions, power is \[ \text{Power} = 1 - \beta. \] Because \(\beta\) depends on the assumed effect size, variance, sample size, and the chosen \(\alpha\), power serves as a summary of how detectable a prespecified effect is.

1.4 Effect size and the role of assumptions

Power is not an intrinsic property of a dataset; it is a property of a test together with a set of assumptions. Chief among these is the effect size used in the alternative. Other assumptions include distributional form, variability, nuisance parameter values, and details of the design (e.g., allocation ratio, repeated measures structure). When these inputs are inaccurate, the realized power can differ from the planned value.

2 Power in Hypothesis Testing

2.1 One-sample, two-sample, and paired test settings

2.1.1 Comparing means with known/unknown variance

For mean comparisons, the standard building blocks are one-sample, two-sample, and paired t-type tests. If variance is assumed known, power calculations can use fixed standard errors. When variance is unknown and estimated from the data, power depends on how the test statistic’s reference distribution changes with estimation, usually leading to slightly different power behavior than the known-variance idealization.

2.1.2 Proportions and binary outcomes

For binary outcomes, common tests are based on approximations to the distribution of sample proportions or on large-sample test statistics such as the normal approximation. Power depends on the baseline event probability, the target difference between proportions, and the total sample size. Because binomial variability is tied to the mean level, power often varies substantially across different baseline rates.

2.2 Continuous vs categorical outcome impacts

Continuous outcomes can offer more information per observation when measurements are precise, but their power is sensitive to dispersion and distribution shape. Categorical outcomes often require grouping or thresholds, which can reduce effective information and make power depend strongly on the allocation across categories and the handling of sparsity.

2.3 Test directionality (one-sided vs two-sided)

A one-sided test restricts rejection to deviations in a single direction, typically increasing power for effects occurring in that direction, given the same \(\alpha\) and sample size. Two-sided tests allow evidence for effects in either direction and therefore generally have lower power for a specific signed effect, though they provide broader protection against wrong-direction assumptions.

2.4 Decision rules and rejection regions

Power is determined by the rejection region induced by the decision rule. For many standard tests, rejection regions correspond to critical values of a test statistic. Changing the critical value through different \(\alpha\) levels or different variance estimators changes the probability that the statistic will fall into the rejection region under the assumed alternative.

3 Computing Power

3.1 Analytical power calculations

Analytical approaches compute power directly from known or approximated sampling distributions of the test statistic under the alternative. This can produce fast results and clear dependence on inputs such as \(\alpha\), sample size, and effect size. However, analytical formulas are often limited to relatively simple models and well-behaved assumptions.

3.2 Distribution-based approaches

3.2.1 Normal approximations and test statistics

Distribution-based calculations frequently rely on asymptotic normality, resulting in formulas in terms of standard normal cumulative probabilities. These approximations can be accurate for moderate-to-large samples but may degrade when sample sizes are small, outcomes are rare, or model assumptions are strongly violated.

3.3 Simulation-based power (Monte Carlo)

3.3.1 Repeated resampling under an assumed effect

Monte Carlo power estimates approximate power by repeatedly generating datasets under the alternative hypothesis and computing the test outcome each time. The estimated power is the proportion of simulated datasets where the test rejects. This approach can handle complex statistics, nonstandard estimators, and intricate designs, provided the simulation model matches the intended data-generating process.

3.4 Numerical methods and software workflows

Numerical techniques may compute power by integrating relevant probability expressions, evaluating likelihood-based quantities, or searching over parameters to achieve a target power. In practice, software workflows often combine analytical templates with numerical evaluation, especially for generalized linear models, mixed effects models, and designs with constraints.

3.5 Planning under uncertainty of nuisance parameters

Power calculations require not only the effect of interest but also nuisance parameters, such as baseline event rates, variability, and correlations. When these nuisance parameters are uncertain, planning may involve using conservative ranges or multiple scenarios. Some designs incorporate historical data to parameterize the nuisance components, while others explicitly compute power sensitivity to alternative plausible values.

4 Power and Sample Size Planning

4.1 Sample size as the main lever

Sample size strongly affects the standard error and, consequently, the distribution of test statistics. Increasing sample size typically shifts the test statistic more decisively toward the rejection region under the alternative, raising power. Conversely, limited sample sizes increase uncertainty, often leading to greater Type II error probability.

4.2 Translating desired power into design targets

Planning often begins with a desired power level (commonly chosen to balance feasibility and risk) and then determines the sample size that achieves it under specified assumptions. This translation can be done via analytical formulas, numerical solvers, or iterative simulation until the estimated power matches the target.

4.3 Trade-offs among power, sample size, and significance level

The three main knobs—power, sample size, and significance level \(\alpha\)—are coupled. Raising \(\alpha\) generally increases power because the rejection threshold becomes less stringent, but it also increases Type I error risk. Achieving higher power usually requires more observations unless \(\alpha\) is increased or the assumed effect size is larger.

4.4 Handling multi-arm and factorial designs

Multi-arm and factorial designs introduce additional comparisons and potentially shared controls or complex covariance structures. Power calculations must reflect how many groups are compared, the allocation ratio, and which contrasts are tested. In factorial settings, power for main effects and interactions can differ, and models must account for both main and interaction components to avoid mischaracterizing detectability.

4.5 Attrition and imperfect adherence considerations

Real studies often experience dropout, missingness, or imperfect adherence to assigned interventions. Power planning can adjust for expected attrition by inflating the initially recruited sample size or by incorporating missing-data assumptions into the simulation model. If adherence affects the effective exposure, analysts may plan power for intention-to-treat effects while also considering alternative estimands that better reflect received treatment.

5 Power Curves and Sensitivity Analyses

5.1 Understanding power curves across effect sizes

A power curve shows how power changes as the assumed effect size varies. Rather than relying on a single-point effect assumption, curves reveal detectability across a spectrum, indicating whether the planned study is likely to detect modest effects or only larger ones. This helps clarify what “guaranteed” detection actually means under the model.

5.2 Minimal detectable effect (MDE) concepts

The minimal detectable effect is often defined as the smallest effect size that achieves a target power at a chosen \(\alpha\). MDE provides a concrete planning target, but it depends heavily on assumptions about variance, baseline rates, correlation, and outcome distribution. As a result, the MDE should be interpreted as model-based rather than universal.

5.3 Sensitivity to variance and outcome dispersion

Variance and dispersion determine how quickly uncertainty shrinks with additional observations. If the true variability is larger than expected, power typically decreases and the effective MDE rises. Sensitivity analyses can compute power under alternative variance estimates to show how robust the design is to miscalibration.

5.4 Robustness to model misspecification

If the statistical model used for power calculations differs from the data-generating process—such as wrong distributional assumptions, incorrect link functions, or unmodeled dependence—power may deviate from predictions. Robustness checks can involve alternative model forms, different variance structures, or simulation models that incorporate plausible deviations from the idealized assumptions.

5.5 Graphical interpretation and reporting conventions

Power curves are commonly presented with effect size on the horizontal axis and power on the vertical axis, often including markers for the target power threshold. Reporting conventions usually emphasize the assumed \(\alpha\), the definition of effect size, and the scenario used to compute the curve. Clear labeling helps prevent misunderstanding, especially when effect size units vary across domains.

6 Multiple Testing and Complex Designs

6.1 Family-wise error rate vs false discovery rate

When many hypotheses are tested, controlling error becomes more complicated than selecting a single \(\alpha\). The family-wise error rate limits the probability of at least one Type I error across a family of tests. The false discovery rate targets the expected proportion of rejected nulls that are false among all rejections. Different control goals lead to different power and threshold choices.

Bonferroni adjustment reduces the effective significance level per test by dividing \(\alpha\) by the number of comparisons, generally lowering power. Related procedures, such as Holm’s method, can provide more power than strict Bonferroni while still controlling family-wise error. In planning, these adjustments must match the structure of the hypothesis family and the intended testing strategy.

6.3 Correlation among tests and its consequences

Tests within a family are often correlated due to shared data sources, overlapping covariates, or clustered sampling. Positive correlation can change the behavior of error rates and power under multiple-testing corrections. As a result, power assessments that assume independence may misestimate detectability, making simulation or structured approximations valuable.

6.4 Interim analyses and spending alpha

Interim analyses evaluate results partway through data collection. Because multiple looks inflate the chance of Type I error if handled naively, alpha spending frameworks distribute the overall \(\alpha\) across interim and final analyses. Planning requires computing power under the timing and stopping rules, since early stopping affects the distribution of observed test statistics.

6.5 Clustered data and intraclass correlation effects

Clustered designs—where observations within a group share similarities—introduce dependence measured by intraclass correlation. Dependence reduces the effective information compared with independent observations, often requiring larger total sample sizes to achieve the same power. Power planning for clustered data therefore typically incorporates design effects derived from intraclass correlation and cluster sizes.

7 Power for Estimation vs Testing

7.1 Distinguishing “power” for detection from confidence interval precision

Although “power” is commonly used for hypothesis testing, related planning targets exist for estimation accuracy. Confidence interval precision focuses on the width of intervals for parameters, such as means or regression coefficients. Detection-oriented power and estimation-oriented precision can lead to different sample size choices, particularly when the primary goal is accurate measurement rather than binary decision-making.

7.2 Margin of error and interval width targets

Estimation planning may set requirements like a maximum margin of error for a parameter or a desired upper bound on average interval width. These targets depend on variance, sample size, and the chosen confidence level. For some parameters and models, interval width can behave differently across the parameter space, so sensitivity calculations may be necessary.

7.3 Equivalence and non-inferiority contexts (high level)

In equivalence or non-inferiority settings, the goal is not to prove superiority over a null of “no difference,” but to demonstrate that differences fall within acceptable bounds. This changes the logic of testing and typically alters how power is defined and computed, since the rejection region corresponds to meeting margin-based criteria.

7.4 Relationship between power and effect estimation quality

Detection of an effect and quality of its estimated magnitude are related but not identical. A study can have high power to detect a statistically significant difference while still producing a noisy estimate of its size, especially when the effect is small or sample size is only just sufficient. Conversely, precision can be strong even when the decision to reject a null may not occur, depending on threshold settings and uncertainty.

8 Interpretation and Common Pitfalls

8.1 “Low power” vs “no effect” misunderstanding

A common misconception is interpreting a non-significant result as evidence that there is no effect. In reality, low power means the test may simply be unlikely to detect the effect size under consideration. Distinguishing between “not detected” and “no effect” requires attention to the assumed alternative and the power under that alternative.

8.2 Post hoc power and why it can mislead

Post hoc power attempts to re-estimate power after observing the data, often using the observed effect size. This practice can be misleading because it tends to echo the observed test statistic and does not add new information beyond the original p-value and confidence interval. For interpretation, confidence intervals and preplanned design calculations are typically more informative.

8.3 Publication bias and selective reporting

Selective reporting can interact with power. If studies with low power are less likely to yield statistically significant results, then the published literature may overrepresent detectable effects, giving a distorted view of typical effect sizes. Planning and reporting standards aim to reduce these biases by making study intentions transparent.

8.4 Overfitting assumptions and optimistic effect sizes

Power calculations rely on effect size assumptions that may be informed by prior studies or pilot data. When those assumptions are overly optimistic—due to small pilot samples, publication of extreme estimates, or model flexibility—the planned power can be inflated and the study may underperform relative to expectations.

8.5 Reporting standards for power and sample size rationale

Transparent reporting typically includes the chosen \(\alpha\), the planned effect size, the test type, and the method used to compute power (analytical or simulation). If multiple scenarios were considered, reporting should describe the range of assumptions. Such documentation allows readers to assess the credibility of the planning target.

9 Practical Reporting and Best Practices

9.1 Stating effect size assumptions clearly

The effect size used in power computations should be stated with units and directionality. When multiple effect definitions are possible—such as odds ratios versus risk differences, or absolute versus standardized differences—the article or protocol should specify which definition is used to avoid ambiguity.

9.2 Documenting significance level and test type

Reporting should include the significance level and whether the procedure is one-sided or two-sided, along with the statistical test employed. For complex models, the regression contrast and the variance estimation strategy should be described sufficiently to reproduce the decision rule.

9.3 Reporting power calculation method (analytical vs simulation)

Since analytical and simulation-based approaches can yield different results under complex assumptions, the calculation method should be stated. For simulation, reporting the number of simulated replicates and the data-generating assumptions helps establish accuracy and repeatability.

9.4 Providing sensitivity ranges rather than single-point claims

Because nuisance parameters and variance estimates may be uncertain, best practice is to present power across plausible ranges. Sensitivity information communicates how much confidence planners can reasonably have in the ability to detect effects if conditions differ from the nominal scenario.

9.5 Reproducible planning checklists

Reproducibility improves when planning includes a checklist of inputs: hypotheses, effect size, \(\alpha\), model and assumptions, nuisance parameter values, design details, missingness or attrition assumptions, and the power computation code or workflow. This reduces the gap between planned and implemented analysis.