1 Foundations of statistical inference
Statistical inference is the process of using data from a limited set of observations to make statements about a larger population or about the mechanism that generated the data. It rests on probability, model assumptions, and the recognition that observed results vary from sample to sample. The field aims not only to estimate unknown quantities, but also to express uncertainty in a disciplined way.
1.1 Populations and samples
A population is the full set of units, measurements, or events of interest, while a sample is the subset actually observed. Because studying every member of a population is often impractical, researchers work with samples and seek methods that preserve representativeness. The quality of an inference depends strongly on how the sample was obtained.
1.2 Parameters and statistics
Parameters are numerical characteristics of a population, such as a mean, proportion, or variance, and are typically fixed but unknown. Statistics are quantities computed from sample data, such as the sample mean or sample variance. Inference links statistics to parameters through mathematical models and probability rules.
1.3 Sampling distributions
A sampling distribution describes how a statistic behaves over repeated samples from the same population. It captures the natural fluctuation caused by random sampling. Understanding this distribution is essential for estimating uncertainty, constructing intervals, and evaluating tests.
1.4 Sources of uncertainty
Uncertainty in inference can arise from random sampling, measurement error, model misspecification, and incomplete knowledge of the population. Even when data are accurate, limited sample size introduces variability. Statistical methods try to separate signal from noise while acknowledging these limitations.
2 Probability models
Probability models provide the mathematical framework for describing random variation in data. They specify possible outcomes and assign probabilities in a way that allows predictions and inferential calculations. A good model is often an approximation rather than a literal description of reality.
2.1 Random variables
A random variable is a numerical representation of the outcome of a random process. It may count events, measure quantities, or encode categories. By assigning probabilities to possible values, random variables make data suitable for formal analysis.
2.2 Common distributions
Common distributions are standard probability models used to represent data patterns. They are selected according to the kind of variable being studied, such as counts, binary outcomes, or continuous measurements. Familiarity with standard forms helps simplify both computation and interpretation.
2.2.1 Discrete distributions
Discrete distributions describe variables that take countable values. Examples include the Bernoulli, binomial, geometric, and Poisson distributions. These models are often used for counts, event occurrences, and success-failure data.
2.2.2 Continuous distributions
Continuous distributions apply to measurements that can vary over an interval. Common examples include the normal, exponential, and uniform distributions. Such models are widely used because many natural measurements are approximately continuous.
2.3 Independence and dependence
Independent variables or observations do not influence one another probabilistically, whereas dependent ones exhibit some form of association. Independence simplifies analysis and is often assumed in introductory models. Dependence must be handled carefully because it affects variability and inferential validity.
3 Estimation
Estimation seeks to infer the value of an unknown parameter from observed data. It may produce a single best guess or a range of plausible values. Different estimators can be compared by their accuracy, stability, and theoretical properties.
3.1 Point estimation
Point estimation provides one numerical estimate for an unknown parameter. The estimate is chosen according to a rule applied to the sample. Common goals are simplicity, interpretability, and good long-run behavior.
3.1.1 Method of moments
The method of moments estimates parameters by matching sample moments, such as averages or variances, to their theoretical counterparts. It is often straightforward to apply and can yield closed-form solutions. However, it may be less efficient than other approaches in some settings.
3.1.2 Maximum likelihood estimation
Maximum likelihood estimation selects the parameter value that makes the observed data most probable under the model. It is one of the most widely used inferential techniques because it has strong theoretical foundations and broad applicability. In many models, it yields estimators with desirable large-sample properties.
3.2 Interval estimation
Interval estimation gives a range of values that is intended to capture the unknown parameter with specified confidence or probability. Unlike point estimates, intervals express uncertainty directly. Their interpretation depends on the inferential framework being used.
3.2.1 Confidence intervals
A confidence interval is a frequentist interval constructed so that, over repeated sampling, a stated proportion of such intervals would contain the true parameter. It does not assign probability to the parameter itself. Instead, it reflects the variability of the estimator and the chosen confidence level.
3.2.2 Credible intervals
A credible interval is a Bayesian interval that contains the parameter with a specified posterior probability. It is derived from the posterior distribution and has a direct probabilistic interpretation. This makes it useful when prior information is incorporated into analysis.
3.3 Properties of estimators
Estimators are judged by how well they recover the true parameter across repeated samples. Important criteria include systematic error, spread, convergence, and comparative precision. No single property guarantees overall superiority in every situation.
3.3.1 Bias
Bias is the difference between an estimator’s expected value and the true parameter value. An unbiased estimator is centered correctly on average, though it may still vary widely from sample to sample. Small bias is often desirable, but some bias may be acceptable if variance is reduced.
3.3.2 Variance
Variance measures how much an estimator changes across repeated samples. A low-variance estimator is more stable and less sensitive to random fluctuations. In practice, analysts often seek a balance between bias and variance.
3.3.3 Consistency
Consistency means that an estimator approaches the true parameter as sample size increases. It is a key large-sample property because it indicates that more data lead to better inference. A consistent estimator need not perform well in small samples.
3.3.4 Efficiency
Efficiency refers to the relative precision of an estimator compared with alternatives. An efficient estimator typically has smaller variance among estimators with similar bias. In many classical settings, efficiency is linked to optimal use of available information.
4 Hypothesis testing
Hypothesis testing is a formal procedure for assessing whether observed data are compatible with a proposed claim. It is widely used to compare models, evaluate effects, and make decisions under uncertainty. The framework depends on predefined hypotheses and rules for interpreting evidence.
4.1 Null and alternative hypotheses
The null hypothesis represents a default or baseline claim, often stating no effect or no difference. The alternative hypothesis expresses the competing possibility of an effect, difference, or association. The test is designed to measure how strongly the data conflict with the null.
4.2 Test statistics
A test statistic is a numerical summary of the data used to assess the hypotheses. It is chosen so that extreme values under the null provide evidence against it. The distribution of the statistic under the null is central to the testing procedure.
4.3 p-values and significance levels
A p-value is the probability, assuming the null hypothesis is true, of obtaining data at least as extreme as those observed. The significance level is the threshold chosen in advance for deciding whether evidence is strong enough to reject the null. These quantities are often misunderstood, so careful interpretation is essential.
4.4 Type I and Type II errors
A Type I error occurs when a true null hypothesis is rejected, while a Type II error occurs when a false null hypothesis is not rejected. These two error types reflect different kinds of inferential failure. Their relative importance depends on the practical setting.
4.5 Power analysis
Power is the probability that a test correctly rejects a false null hypothesis. Power analysis helps determine sample size, detectability of effects, and study design quality. Higher power reduces the chance of missing meaningful results.
4.6 Common tests
Common tests are standard procedures used in many applied analyses. They differ in the type of data, assumptions, and questions they address. Selection depends on measurement scale, distributional assumptions, and study design.
4.6.1 z-tests
Z-tests are commonly used when a test statistic is approximately normal and the population variance is known or large-sample methods apply. They are often introduced in basic inferential settings. Their main appeal is simplicity.
4.6.2 t-tests
T-tests compare means when the population variance is unknown and sample sizes are limited. They rely on the t-distribution, which adjusts for extra uncertainty in estimating variability. Variants include one-sample, two-sample, and paired t-tests.
4.6.3 Chi-squared tests
Chi-squared tests assess goodness of fit, independence, or homogeneity in categorical data. They compare observed counts with expected counts under a null model. These tests are especially useful for contingency tables.
4.6.4 F-tests
F-tests compare variances or nested model fits using an F-distributed statistic. They appear in analysis of variance and regression settings. The procedure is useful for evaluating whether added model complexity is justified.
5 Likelihood-based inference
Likelihood-based inference uses the likelihood function to compare parameter values according to how well they explain the observed data. It forms a unifying approach to estimation, testing, and model comparison. Many modern methods build on likelihood principles.
5.1 Likelihood functions
A likelihood function treats the observed data as fixed and views the parameter as variable. It measures the relative support that the data provide for different parameter values. Likelihood is not a probability distribution over parameters, but it plays a central inferential role.
5.2 Likelihood ratio tests
Likelihood ratio tests compare two nested models by examining the ratio of their maximized likelihoods. Large discrepancies suggest that the more complex model fits substantially better. The resulting statistic often has an approximate chi-squared distribution in large samples.
5.3 Profile likelihood
Profile likelihood focuses on one parameter of interest while optimizing over nuisance parameters. This reduces a multivariate problem to a one-dimensional summary. It is useful for constructing intervals and visualizing parameter uncertainty.
5.4 Information criteria
Information criteria, such as AIC and BIC, provide numerical measures for comparing models by balancing fit and complexity. They are especially useful when models are not nested. The preferred model is usually the one with the smallest criterion value.
6 Bayesian inference
Bayesian inference combines prior information with observed data to produce updated beliefs about unknown quantities. It treats parameters as random variables with distributions that reflect uncertainty. This framework is especially flexible when prior knowledge is meaningful.
6.1 Prior distributions
A prior distribution represents beliefs about a parameter before observing the current data. It may be informative, weakly informative, or diffuse depending on the analysis. Priors can encode scientific knowledge, previous studies, or subjective judgment.
6.2 Posterior distributions
The posterior distribution is the updated distribution of the parameter after combining prior information with the likelihood from the data. It is the main result of Bayesian analysis. Posterior summaries are used for estimation, prediction, and decision-making.
6.3 Bayesian estimation
Bayesian estimation typically uses posterior means, medians, or modes as point estimates. These summaries are chosen according to loss functions or practical convenience. The posterior distribution also provides direct measures of uncertainty.
6.4 Bayesian hypothesis testing
Bayesian hypothesis testing compares the posterior support for competing hypotheses or models. It may use posterior probabilities, Bayes factors, or decision rules based on loss. This approach differs from frequentist testing by assigning probability to hypotheses within the model framework.
6.5 Credibility and decision-making
Bayesian analysis supports decisions by combining uncertainty with explicit costs or benefits. Credible statements about parameters are often easier to interpret than indirect long-run error rates. The framework is especially useful when decisions must be made under incomplete information.
7 Asymptotic theory
Asymptotic theory studies the behavior of estimators and tests as sample size becomes large. It explains why many methods work well even when exact finite-sample results are difficult to obtain. Large-sample approximations are a cornerstone of modern inference.
7.1 Law of large numbers
The law of large numbers states that sample averages tend to approach their expected values as sample size increases. It provides a foundation for consistency and for the reliability of empirical summaries. The result helps justify using averages as estimators.
7.2 Central limit theorem
The central limit theorem shows that sums or averages of many independent random contributions are approximately normal under broad conditions. This principle underlies many familiar inferential procedures. It explains why normal approximations are often effective.
7.3 Large-sample approximations
Large-sample approximations simplify inference by replacing exact distributions with asymptotic ones. They are especially useful for complex models or large datasets. Their accuracy generally improves as the sample grows.
7.4 Consistency and asymptotic normality
Consistency ensures convergence to the true value, while asymptotic normality describes the limiting distribution of an estimator after suitable scaling. Together, these properties support interval estimation and hypothesis testing. They are standard goals in theoretical statistics.
8 Resampling and simulation methods
Resampling and simulation methods use repeated computation to approximate sampling distributions, assess uncertainty, or test assumptions. They are valuable when analytic formulas are unavailable or difficult to derive. These techniques have become especially important with modern computing.
8.1 Bootstrap
The bootstrap repeatedly resamples from observed data with replacement to estimate the variability of a statistic. It approximates the sampling distribution without relying heavily on closed-form theory. Bootstrap methods are widely used for standard errors and confidence intervals.
8.2 Jackknife
The jackknife systematically recomputes an estimate by leaving out one observation at a time, or one group at a time. It is a simpler resampling method that can reveal influence and variability. Its effectiveness is often strongest for smooth statistics.
8.3 Permutation tests
Permutation tests assess significance by rearranging labels or assignments under the assumption that the null hypothesis implies exchangeability. They generate an empirical null distribution from the data themselves. These tests are particularly appealing when parametric assumptions are weak.
8.4 Monte Carlo methods
Monte Carlo methods use random simulation to approximate quantities that are hard to compute exactly. They are widely employed in Bayesian inference, risk analysis, and complex probability calculations. Accuracy typically improves with more simulated draws.
9 Experimental design and inference
Experimental design shapes how data are collected so that inferential conclusions are more credible. A well-designed study reduces confounding, improves precision, and strengthens causal interpretation when appropriate. Design choices often matter as much as the analysis itself.
9.1 Randomization
Randomization assigns treatments or conditions by chance, reducing systematic bias and helping balance unknown factors across groups. It is a fundamental tool for drawing causal conclusions in experiments. Randomization also supports valid statistical testing.
9.2 Control groups
A control group provides a baseline for comparison against the treatment or intervention group. It helps isolate the effect of the factor under study. Controls may receive standard care, a placebo, or no treatment depending on context.
9.3 Blocking and stratification
Blocking and stratification divide subjects into more homogeneous groups before comparison. This can reduce variability and improve precision by ensuring more balanced treatment assignment. The methods are especially useful when important sources of heterogeneity are known in advance.
9.4 Sample size determination
Sample size determination seeks to choose a number of observations large enough to achieve adequate precision or power. It depends on the expected effect size, variability, desired confidence, and acceptable error rates. Planning sample size is a central part of study design.
10 Applications
Statistical inference is used whenever decisions must be made from incomplete data. Its methods adapt to different disciplines, but the core goals remain estimation, uncertainty quantification, and evidence assessment. The same conceptual tools can support both exploratory and confirmatory work.
10.1 Scientific research
In the sciences, inference helps evaluate experimental results, estimate physical or biological quantities, and compare competing theories. It is used in laboratory studies, field observations, and observational datasets. Careful modeling is essential to avoid overstating conclusions.
10.2 Medicine and public health
Medical and public health research uses inference to assess treatments, estimate risks, and study disease patterns. Clinical trials and epidemiological studies rely heavily on design, testing, and confidence measures. Clear uncertainty quantification is particularly important in health-related decisions.
10.3 Social sciences
Social science applications include survey analysis, policy evaluation, and the study of human behavior. Researchers often work with noisy data and complex causal structures. Inference helps distinguish broad patterns from random variation and measurement limitations.
10.4 Business and economics
In business and economics, statistical inference supports forecasting, market analysis, and evaluation of interventions or strategies. Companies use it to estimate demand, test product changes, and measure performance. Economic studies often combine observational data with modeling assumptions.
10.5 Engineering and quality control
Engineering uses inference to monitor processes, detect defects, and improve reliability. Quality control methods rely on sampling, control charts, and acceptance criteria. Statistical reasoning helps maintain consistency in manufacturing and technical systems.