1 Foundations of Hypothesis Testing
1.1 Statistical hypotheses and their roles
Statistical hypothesis testing provides a formal way to evaluate a claim about a population parameter or data-generating process using a sample. The hypotheses are statements that can be contrasted using probability models: one represents a baseline explanation (the null), and the other represents an alternative explanation. The goal is not to prove a hypothesis true, but to quantify how consistent observed data are with the null under specified modeling choices.
1.2 Null hypothesis (H0) and alternative hypothesis (H1)
The null hypothesis, often denoted H0, typically expresses “no effect” or “no difference” in a way that is mathematically convenient for deriving sampling behavior of a test statistic. The alternative hypothesis, H1, specifies the direction or form of an effect the analyst wants evidence for. Depending on the context, H1 can be one-sided (greater than or less than) or two-sided (different in either direction). The definition of H1 determines whether departures in specific directions are treated as more informative than others.
1.3 Test statistics and decision rules
A test statistic is a function of the observed sample designed to measure compatibility with H0. Examples include standardized mean differences, variance-normalized statistics, or discrepancy measures between observed and expected counts. A decision rule then maps values of the test statistic to outcomes such as “reject H0” or “fail to reject H0.” In many classic formulations, the rule is framed through either a critical region (reject when the statistic falls in a designated set) or through p-values (reject when p is below a threshold).
1.4 Significance level (α) and critical values
The significance level α is the pre-specified probability of rejecting H0 when H0 is actually true. In other words, α controls the long-run rate of false positives under the null model. For tests with critical values, one chooses cutoffs of the test statistic so that the total probability of rejection under H0 equals α. This connects the abstract threshold to a tangible criterion in the sampling distribution of the statistic.
1.5 p-values and their interpretation
A p-value is the probability, under H0 and the assumed model, of observing a result at least as extreme as the one obtained. “Extreme” is defined by the direction and construction of the test statistic. Proper interpretation treats the p-value as conditional on H0: it measures how surprising the data would be if H0 were correct, not the probability that H0 is true. Lower p-values indicate greater tension between the observed data and the null, but they do not automatically quantify effect size.
2 Error Types and Test Performance
2.1 Type I error (false positive)
A Type I error occurs when H0 is rejected even though H0 is true. By design, the probability of a Type I error is controlled at approximately α (exactly α for many idealized tests under their assumptions). The practical implication is that even with correct modeling, some rejections will occur purely by chance, especially when many tests are performed.
2.2 Type II error (false negative)
A Type II error happens when H0 is not rejected despite H0 being false. The probability of a Type II error is denoted β, and it depends on the true parameter values under alternatives, the chosen α, and the variability of the sampling distribution. A test can have low Type II error for particular alternatives while still having substantial Type II error for other alternatives.
2.3 Power of a test
Power is the probability of rejecting H0 when a specified alternative is true, often written as 1 − β. It summarizes sensitivity: higher power means the test is more likely to detect departures from H0. Power depends on effect magnitude, sample size, variability, and the chosen significance level. In practice, analysts may use power calculations to plan experiments or to assess whether a nonsignificant result is informative.
2.4 Trade-offs between α and β
Selecting a larger α typically reduces Type II error by making rejection easier, thus increasing power. Conversely, a smaller α reduces false positives but can increase the chance of missing real effects. This trade-off is fundamental to hypothesis testing design: the “right” α depends on the costs of false positives versus false negatives, the plausibility of assumptions, and the scientific goals of the study.
2.5 Sample size and detectability
Sample size affects detectability because it influences the spread of the sampling distribution of the test statistic. With more observations, random noise averages out and smaller effects become distinguishable from sampling variability. Detectability is also tied to variability in the data: higher variance typically requires a larger sample to achieve comparable power for the same effect size.
3 Assumptions and Model Conditions
3.1 Independence and identically distributed data
Many common hypothesis tests rely on independence among observations and, in the simplest settings, identical distribution under the model. Independence ensures that the variability of aggregated statistics is correctly characterized. Identically distributed assumptions often appear as “i.i.d.” conditions, supporting standard formulas for standard errors and enabling reference distributions for test statistics.
3.2 Distributional assumptions (e.g., normality)
Parametric tests frequently assume that certain quantities derived from the data follow known distributions, such as normality of residuals or of mean estimates. When the sample size is large, central limit arguments can make these assumptions less critical, but with small samples distributional misfit can materially affect p-values and Type I error control.
3.3 Variance assumptions and homoscedasticity
Variance assumptions, such as homoscedasticity (equal variances across groups), are used to justify pooled or shared variance estimators. When variances differ substantially, some test statistics can be distorted, leading to inaccurate error probabilities. Alternatives that adjust standard errors or use different variance estimators can provide improved behavior under heteroscedasticity.
3.4 Robustness and sensitivity to assumptions
Robustness refers to how resistant a test is to departures from its ideal conditions. Some procedures remain reasonably valid under moderate violations, while others can fail dramatically. Sensitivity analysis may involve diagnostic plots, alternative model choices, or comparing results across different methods (e.g., parametric versus nonparametric), while keeping in mind that different methods target different assumptions.
3.5 Checking conditions in practice
Condition checking often includes assessing residuals, verifying independence through study design or time-series diagnostics, and evaluating distributional fit using plots or formal tests. For practical work, diagnostics should be planned before looking at p-values, to avoid confirmation bias. When violations are detected, analysts may transform variables, use robust or variance-adjusted methods, or choose procedures with weaker assumptions.
4 Common Hypothesis Tests
4.1 Z-tests and their typical use cases
Z-tests evaluate hypotheses about means or proportions when the relevant standard error is known or can be treated as known. They commonly arise in large-sample settings or when modeling assumptions imply a normal reference distribution for the standardized statistic. Z-tests are frequently introduced as “standard normal” tests where critical values come from the normal distribution.
4.2 t-tests (one-sample, two-sample, paired)
t-tests are designed for situations where the population standard deviation is unknown and is estimated from the data. The reference distribution for the standardized statistic follows a t distribution, with degrees of freedom depending on the sample structure.
4.2.1 Paired t-tests for matched observations
Paired t-tests analyze differences between matched observations, such as before-and-after measurements on the same units or comparisons between twins. By focusing on within-pair differences, this approach removes between-subject variation and can provide greater sensitivity when pairing is meaningful.
4.3 Tests for proportions
Proportion tests address claims about a success rate in a population. One-sample tests compare an observed proportion to a hypothesized value, and two-sample tests compare proportions across groups. The validity of such tests depends on adequate expected counts and appropriate approximations, especially for smaller samples.
4.4 Chi-square tests (goodness-of-fit, independence)
Chi-square tests use a discrepancy measure between observed counts and expected counts under a null model. In goodness-of-fit testing, the null specifies the distribution of categories. In independence testing, the null asserts that two categorical variables are unrelated. The chi-square approximation works best when expected counts are not too small, and analysts should check that condition.
4.5 Nonparametric alternatives
Nonparametric tests aim to reduce reliance on strict distributional assumptions by working with ranks or general distributional properties. They are often used when data are skewed, ordinal, or when variances are difficult to model parametrically. While they may sacrifice some efficiency under ideal parametric conditions, they can provide better calibration of error rates when assumptions are doubtful.
4.5.1 Rank-based tests (e.g., Wilcoxon)
Rank-based tests, including the Wilcoxon rank-sum test and Wilcoxon signed-rank test, compare central tendencies between groups or assess median-like shifts in paired settings. These methods use the order of observations rather than their exact values, making them less sensitive to outliers and some forms of non-normality.
4.6 Analysis of variance (ANOVA) for multiple groups
ANOVA generalizes mean comparisons to more than two groups by partitioning total variability into within-group and between-group components. Under the null hypothesis that all group means are equal, the ratio of between-group to within-group variability follows an F distribution (under classic assumptions). ANOVA also allows for testing specific structured alternatives depending on the model design.
4.6.1 Post-hoc comparisons and multiple group follow-ups
When ANOVA indicates that not all means are equal, follow-up comparisons identify which groups differ. Post-hoc procedures often incorporate adjustments for multiple comparisons to preserve appropriate error control. Choices among methods depend on assumptions like equal variances and on how conservative the analyst wants to be.
5 Confidence Intervals and Connections to Testing
5.1 Confidence intervals as inverse tests
Confidence intervals can be viewed as the set of parameter values that would not be rejected by a corresponding hypothesis test at a given significance level. This duality connects estimation and testing: rather than focusing solely on a yes/no decision, interval methods describe uncertainty about plausible parameter values compatible with the data.
5.2 Relationship between p-values and intervals
A p-value can often be derived from a confidence interval by checking whether a hypothesized parameter value lies inside or outside the interval. For two-sided tests, if the hypothesized value is outside the (1 − α) confidence interval, the p-value will be less than α. For one-sided tests, the mapping depends on the direction of the alternative and the interval type.
5.3 One-sided vs two-sided intervals
Two-sided tests and intervals consider deviations in either direction, producing wider intervals than one-sided counterparts at the same confidence level. One-sided alternatives focus on departures in a specific direction, yielding tighter intervals because fewer deviations are treated as compatible with the null. Selecting interval type should align with the scientific question and pre-analysis expectations.
5.4 Effect sizes and practical significance
Effect sizes quantify the magnitude of an observed difference or association, separate from whether it reaches statistical significance. Confidence intervals for effect sizes provide a range of plausible magnitudes and can be evaluated against substantive thresholds. Practical significance considers whether the estimated effect would matter in the real-world context, not merely whether sampling variability can explain it.
6 Multiple Testing and Error Control
6.1 Why multiple comparisons matter
Testing many hypotheses increases the probability of encountering at least one false positive by chance, even if each individual test is controlled at a nominal α. This phenomenon is especially relevant in exploratory settings, high-dimensional data analysis, and routine model diagnostics where numerous tests may be performed.
6.2 Family-wise error rate control
The family-wise error rate (FWER) is the probability of making at least one Type I error within a specified set of tests. Controlling FWER ensures that the overall chance of any false discovery remains bounded, which often leads to conservative procedures, particularly when the number of hypotheses is large.
6.3 False discovery rate (FDR) control
The false discovery rate (FDR) is the expected proportion of false positives among all rejected hypotheses. FDR control provides a compromise between discovery and error management, typically offering greater power than FWER-focused methods when many hypotheses are tested. It is commonly used in fields such as genomics and other settings where large-scale screening is standard.
6.4 Adjustment methods (e.g., Bonferroni, Holm)
Adjustment methods modify rejection thresholds to account for multiple testing. Bonferroni correction sets stricter per-test α levels by dividing by the number of tests, controlling FWER but often being conservative. Holm’s method improves on simple Bonferroni by using a stepwise procedure that is uniformly at least as powerful while still controlling FWER under typical assumptions.
6.5 Interpreting adjusted p-values
Adjusted p-values (or adjusted significance thresholds) represent p-values after accounting for the multiplicity structure. Interpretation follows the same logic as unadjusted p-values but with a different calibration: an adjusted p-value below α indicates that the corresponding null would be rejected while maintaining the chosen error control criterion (e.g., FWER or FDR) across the full set.
7 Bayesian vs Frequentist Perspectives (High-Level)
7.1 Contrasting evidence and probability interpretations
Frequentist hypothesis testing evaluates evidence through the behavior of test statistics under repeated sampling, treating parameters as fixed and randomness as residing in the data. Bayesian analysis treats parameters as uncertain quantities with prior distributions and updates them using observed data to obtain posterior probabilities, producing direct probability statements about parameters or hypotheses under a chosen model.
7.2 Credible intervals vs confidence intervals
Credible intervals are Bayesian intervals that contain a parameter with a specified posterior probability under the posterior distribution. Confidence intervals are frequentist constructs with coverage guarantees across repeated samples from the underlying process. Both summarize uncertainty, but their interpretive meanings differ: credible intervals refer to probability given the data and prior, whereas confidence intervals refer to long-run coverage under the model.
7.3 When each framework is commonly used
Frequentist methods are widely used in classical statistical practice, especially where error rates and long-run control are central to the analysis plan. Bayesian approaches are commonly used when incorporating prior knowledge, handling complex models, or requiring posterior probability statements for decision-making. In many modern workflows, analysts may use both perspectives to cross-check conclusions.
8 Reporting and Reproducibility
8.1 Writing results: test statistic, df, p-value
Clear reporting typically includes the test statistic value, relevant degrees of freedom (where applicable), and the p-value. Providing these components enables readers to verify the computation and compare results across studies or analyses. When possible, reporting the direction of the effect and whether the test was one-sided or two-sided reduces ambiguity.
8.2 Stating assumptions and data conditions
A complete account includes the modeling conditions required for the selected test, such as independence, distributional assumptions, or variance structure. If diagnostics were performed, brief summaries of those checks help readers judge whether the test is appropriate for the data. Where assumptions are uncertain, stating what was attempted and what alternative methods were considered improves transparency.
8.3 Effect size reporting and uncertainty
Reporting effect sizes, along with uncertainty quantification such as confidence intervals (or credible intervals in Bayesian settings), supports evaluation of practical relevance. Effect sizes contextualize the magnitude of findings and prevent overemphasis on statistical significance. Including uncertainty also clarifies how sharply the data pinpoint the estimated effect.
8.4 Common pitfalls and misinterpretations
Common issues include interpreting p-values as the probability that the null hypothesis is true, ignoring the distinction between statistical and practical significance, and failing to adjust for multiple testing when many hypotheses are examined. Another frequent pitfall is applying parametric tests without assessing whether key assumptions are badly violated, particularly in small-sample or heavily skewed settings.
8.5 Reproducible workflows and documentation
Reproducibility is supported by detailed documentation of data preprocessing, analysis choices, and code versioning. Analysts should record how the test was selected, which assumptions were checked, and the precise settings used for computations. Sharing scripts, specifying software versions, and saving intermediate outputs help others rerun the analysis and validate results.