1 Fundamental concepts
Hypothesis testing is a core tool of statistical inference. It provides a structured way to assess whether observed sample results are compatible with a stated claim about a larger population. Rather than proving a statement true or false, the method weighs evidence against a default position using probability and preselected decision rules.
1.1 Statistical inference
Statistical inference uses data from a sample to draw conclusions about a population. Because complete information about every member of a population is usually unavailable, analysts rely on limited observations and quantify uncertainty. Hypothesis tests are one formal approach within this broader framework, alongside estimation and confidence intervals.
1.2 Population and sample
The population is the full set of units, events, or measurements of interest. A sample is the subset actually observed. In hypothesis testing, the sample is used to estimate how well a proposed population claim fits the data. The quality of the conclusion depends heavily on how representative the sample is of the larger group.
1.3 Null and alternative hypotheses
The null hypothesis is the initial claim to be tested, often representing no effect, no difference, or no association. The alternative hypothesis states the competing possibility, such as the presence of an effect or a difference. A test examines whether the data provide enough evidence to reject the null hypothesis in favor of the alternative.
1.4 Test statistic
A test statistic is a numerical summary calculated from the sample data. It measures how far the observed result departs from what the null hypothesis would predict. Different tests use different statistics, but all serve the same purpose: converting raw observations into a value that can be evaluated against a reference distribution.
1.5 Significance level
The significance level is the threshold chosen before examining the data, commonly denoted by alpha. It sets the maximum tolerable probability of rejecting the null hypothesis when it is actually true. Lower values make rejection harder and reflect a more cautious standard for deciding that evidence is strong.
1.6 P-value
The p-value is the probability, assuming the null hypothesis is true, of obtaining a result at least as extreme as the one observed. Small p-values suggest the data are unusual under the null model. They do not measure the probability that the null hypothesis itself is true, nor do they directly indicate effect size.
1.7 Critical region
The critical region is the range of test-statistic values that leads to rejection of the null hypothesis. It is determined in advance from the chosen significance level and the test’s sampling distribution. If the observed statistic falls inside this region, the result is deemed statistically significant under the selected criterion.
2 General procedure
Hypothesis testing follows a sequence of steps designed to make decisions transparent and reproducible. Although details vary by method, most analyses begin with a stated question and end with an interpretation of the evidence in context. Clear planning before computation is essential to avoid arbitrary choices after seeing the data.
2.1 Formulating the hypotheses
The first step is to translate the research question into null and alternative hypotheses. These should be precise enough to allow direct statistical evaluation. For example, a study may test whether a mean equals a specified value, whether two groups differ, or whether an association exists between variables.
2.2 Choosing a test
The appropriate test depends on the type of data, the number of groups, the scale of measurement, and the study design. Tests for means, proportions, counts, ranks, or categorical associations each have different assumptions and reference distributions. Selecting the correct procedure is necessary for valid inference.
2.3 Setting assumptions
Before calculating results, analysts specify the conditions under which the test is valid. These may include random sampling, independence, approximate normality, or adequate expected counts. Assumptions are not mere formalities; they determine whether the probability calculations underlying the test are trustworthy.
2.4 Computing the statistic
The sample data are then summarized into the chosen test statistic. This value is compared with its expected behavior under the null hypothesis. The computation may also produce a p-value or a confidence interval, depending on the method used.
2.5 Making a decision
The result is evaluated against the significance level or critical region. If the p-value is below the threshold, the null hypothesis is rejected. If not, the evidence is considered insufficient for rejection. Failing to reject the null does not prove it true; it only indicates that the sample does not provide enough support for an alternative claim.
2.6 Reporting results
Good reporting includes the hypotheses, test used, test statistic, degrees of freedom when relevant, p-value, and a contextual interpretation. Researchers often also provide estimates and confidence intervals to show the magnitude and precision of the observed effect. Transparent reporting helps readers judge both the statistical and substantive importance of the result.
3 Types of hypothesis tests
Hypothesis tests are classified by the structure of the data and the question being asked. Some compare one sample with a known benchmark, others compare two or more groups, and still others analyze ranks or exact probabilities. The best choice depends on the measurement scale and the available assumptions.
3.1 One-sample tests
One-sample tests compare a single sample with a hypothesized population value. They are used when the aim is to determine whether the sample could plausibly have come from a population with a specified mean, proportion, or distributional feature.
3.1.1 Mean tests
One-sample mean tests examine whether the population average differs from a stated value. They are common in laboratory studies, quality control, and survey analysis. When population variability is known, a z-based approach may be used; when it is estimated from the sample, a t-based approach is more typical.
3.1.2 Proportion tests
Proportion tests assess whether the share of observations with a certain characteristic matches a hypothesized proportion. These are useful for binary outcomes such as success or failure, presence or absence, and yes-or-no responses. The method often relies on large-sample approximations, though exact alternatives exist for small datasets.
3.2 Two-sample tests
Two-sample tests compare two groups or two sets of measurements. They address whether the groups differ in central tendency, frequency, or another feature. The main distinction is whether the samples are independent or paired.
3.2.1 Independent samples
Independent-sample tests are used when observations in one group are unrelated to those in the other. Examples include comparing treatment and control groups or two separate populations. These tests examine whether observed differences are larger than would be expected from random variation alone.
3.2.2 Paired samples
Paired-sample tests are used when observations are linked, such as measurements taken before and after an intervention on the same subjects. By focusing on within-pair differences, these tests can reduce noise from individual variation. They are often more efficient than independent-sample methods when pairing is appropriate.
3.3 Multiple-sample tests
Multiple-sample tests compare three or more groups at once. They are useful when a study involves several treatments, categories, or time points. Rather than performing many separate pairwise tests, which can inflate error rates, a single overall test is often preferred first.
3.3.1 Analysis of variance
Analysis of variance, or ANOVA, tests whether group means differ beyond what would be expected from random variability. It partitions observed variation into between-group and within-group components. A significant result indicates that at least one group differs, though it does not identify which one.
3.3.2 Post hoc comparisons
Post hoc comparisons are follow-up tests conducted after an overall multiple-sample test suggests differences exist. They identify which specific groups differ while attempting to control the chance of false positives. These comparisons are usually adjusted for multiple testing to preserve the overall error rate.
3.4 Nonparametric tests
Nonparametric tests make fewer assumptions about the form of the population distribution. They are often used when data are ordinal, skewed, or contain outliers that make standard parametric methods less suitable. Many such tests rely on ranks rather than raw values.
3.4.1 Rank-based tests
Rank-based tests replace values with their ordered positions and then analyze those ranks. This approach reduces sensitivity to extreme observations and can be appropriate for non-normal data. Common rank-based methods compare central tendencies, matched pairs, or associations.
3.4.2 Distribution-free methods
Distribution-free methods do not require the data to follow a specific family of distributions under the null hypothesis. They are valued for flexibility and can be especially helpful in small or irregular samples. Exact and permutation-based procedures are often included in this category.
3.5 Exact tests
Exact tests calculate the true probability of the observed outcome, rather than relying on an approximation. They are especially useful when sample sizes are small or expected counts are low. Because they can be computationally intensive, exact methods are often reserved for situations where approximate tests may be unreliable.
4 Assumptions and conditions
Every hypothesis test depends on conditions that justify its mathematical model. When these conditions are violated, the resulting p-values and conclusions may be misleading. Careful checking of assumptions is therefore an integral part of analysis, not an optional afterthought.
4.1 Random sampling
Random sampling helps ensure that the sample represents the target population without systematic bias. If observations are chosen in a nonrandom way, the conclusions may apply only to the sample itself. Random selection strengthens the link between probability calculations and real-world inference.
4.2 Independence
Independence means that one observation does not determine another. Many tests assume that data points are unrelated, or that dependence has been properly accounted for through pairing or clustering. Violations of independence can distort standard errors and lead to overly confident conclusions.
4.3 Distributional assumptions
Some tests assume a particular distribution, such as normality or binomial behavior. These assumptions influence the expected sampling distribution of the test statistic. When the data depart strongly from the model, analysts may choose a different test or transform the data to improve fit.
4.4 Sample size considerations
Sample size affects both precision and the reliability of approximations. Small samples may produce unstable estimates and limited power, while very large samples can make trivial differences appear statistically significant. Adequate sample size should be planned in light of the expected effect and variability.
4.5 Robustness
A test is robust if it remains reasonably accurate when assumptions are only approximately met. Some procedures tolerate moderate departures from normality or equal variance better than others. Robust methods are valuable because real data rarely match idealized conditions perfectly.
5 Error and decision framework
Hypothesis testing is built around uncertainty, so decisions can be wrong. The framework distinguishes between rejecting a true null hypothesis and failing to reject a false one. Understanding these error types is essential for interpreting results responsibly.
5.1 Type I error
A Type I error occurs when the null hypothesis is rejected even though it is true. This is often called a false positive. The significance level is set to limit the long-run probability of making this mistake across repeated studies under the same conditions.
5.2 Type II error
A Type II error occurs when the null hypothesis is not rejected even though it is false. This is often called a false negative. Such errors become more likely when samples are small, effects are weak, or variability is high.
5.3 Power
Power is the probability that a test will correctly reject a false null hypothesis. It depends on sample size, effect size, variability, and the chosen significance level. High power is desirable because it increases the chance of detecting meaningful effects when they are present.
5.4 Effect size
Effect size measures the magnitude of a difference or association, not merely whether it is detectable. It complements the p-value by showing how large or important the observed phenomenon is. Two studies can have the same significance level yet very different effect sizes.
5.5 Confidence intervals
Confidence intervals provide a range of plausible values for an unknown parameter. They express both estimate and uncertainty in one summary. In many settings, intervals convey more practical information than a binary reject-or-not decision and help readers assess the precision of the result.
6 Common statistical tests
Several standard procedures are widely used across scientific fields. Each has strengths suited to particular data types and study designs. The most common tests differ in whether they handle means, counts, categorical data, or ranks.
6.1 Z-test
The z-test compares a sample statistic to a known or hypothesized value using the standard normal distribution. It is often applied when the population variance is known or when sample sizes are large enough for approximation. The method is simple and widely taught as a foundational test.
6.2 t-test
The t-test is used to compare means when the population variance is unknown and estimated from the data. It appears in one-sample, two-sample, and paired-sample forms. Because it accounts for extra uncertainty in the variance estimate, it is a central tool in applied statistics.
6.3 Chi-squared test
The chi-squared test is used for categorical data, including goodness-of-fit questions and tests of independence in contingency tables. It compares observed counts with expected counts under the null hypothesis. When counts are sufficiently large, the chi-squared approximation is effective and convenient.
6.4 F-test
The F-test compares variances or evaluates overall differences in linear models and ANOVA settings. It relies on the F distribution, which arises from ratios of variance estimates. The method is especially important in comparing nested models and in testing whether several group means differ.
6.5 Fisher's exact test
Fisher's exact test evaluates association in small contingency tables using exact probabilities. It is particularly useful when expected cell counts are low and approximate methods may be inaccurate. The test is exact because it does not rely on large-sample assumptions.
6.6 Wilcoxon tests
Wilcoxon tests are nonparametric alternatives commonly used for one-sample, paired-sample, or two-sample comparisons. They rely on ranks rather than raw values, making them suitable for ordinal data or skewed distributions. These tests are often chosen when a robust comparison of central tendency is needed.
7 Interpretation and reporting
The meaning of a hypothesis test extends beyond the calculation of a p-value. Results must be interpreted in relation to the research question, the size of the effect, and the uncertainty remaining in the data. Clear reporting helps prevent overstatement and misreading of the findings.
7.1 Statistical significance
Statistical significance means that the observed result would be unlikely under the null hypothesis according to the chosen threshold. It does not imply that the effect is large, important, or scientifically decisive. A statistically significant result is best understood as evidence against the null, not as proof of a substantive claim.
7.2 Practical significance
Practical significance concerns whether an effect matters in real-world terms. A difference can be statistically significant yet too small to have meaningful consequences. Conversely, a useful or interesting effect may fail to reach statistical significance if the sample is limited or variability is high.
7.3 Confidence and uncertainty
Interpretation should always include uncertainty. Confidence intervals, standard errors, and related measures show how much precision the study provides. A cautious report explains what the data support and where uncertainty remains, rather than presenting the outcome as definitive.
7.4 Common misinterpretations
Common mistakes include treating a p-value as the probability that the null hypothesis is true, assuming non-significance proves no effect, and equating statistical significance with practical importance. Another frequent error is to ignore the role of study design and assumptions. Avoiding these misunderstandings is crucial for sound inference.
8 Limitations and critiques
Hypothesis testing is powerful, but it has well-known limitations. Its conclusions depend on model assumptions, can be distorted by repeated testing, and may encourage overly binary thinking. These critiques have led to more cautious and nuanced use of the method in modern analysis.
8.1 Dependence on assumptions
The validity of a test depends on whether its assumptions are approximately satisfied. If the data violate independence, distributional requirements, or sampling conditions, the nominal error rates may no longer hold. Analysts must therefore examine data structure and choose methods carefully.
8.2 Multiple testing
When many hypotheses are tested simultaneously, the chance of false positives rises. This issue arises in large studies, data mining, and exploratory research. Adjustments and hierarchical testing strategies are often used to reduce the risk of misleading results.
8.3 p-hacking
P-hacking refers to selectively analyzing or reporting data in ways that increase the likelihood of finding statistical significance. Examples include trying many models, changing exclusion rules, or stopping data collection after favorable results appear. Such practices undermine the integrity of inference and can exaggerate evidence.
8.4 Reproducibility concerns
Results from hypothesis tests may fail to replicate if studies are underpowered, biased, or overly flexible in analysis. Reproducibility concerns have encouraged greater transparency, preregistration, and sharing of data and code. Robust replication is a key check on the reliability of statistical claims.
8.5 Bayesian alternatives
Bayesian approaches offer an alternative framework for statistical inference. Rather than relying on long-run error rates alone, they combine prior information with observed data to produce posterior distributions. In some settings, Bayesian methods provide a more direct interpretation of uncertainty and evidence.