Probability distribution

1 Fundamental concepts

A probability distribution describes how probability is assigned across the possible values of a random variable. It provides a structured way to represent uncertainty and to distinguish likely outcomes from rare ones. In practice, a distribution may be given by a formula, a table, or an empirical approximation from observed data.

1.1 Random variables

A random variable is a function that assigns a numerical value to each outcome of a random experiment. It may represent a count, a measurement, or any quantity derived from an uncertain process. Random variables are commonly classified as discrete or continuous, depending on whether their values are countable or vary over intervals.

1.2 Sample space and events

The sample space is the set of all possible outcomes of an experiment. An event is any subset of that space, such as drawing a red card or obtaining a value above a threshold. Probabilities are defined for events, and a distribution can be viewed as a systematic description of how likely different events are.

1.3 Probability mass, density, and cumulative functions

Different mathematical functions are used to describe distributions in different settings. Discrete variables are often represented by probability masses, continuous variables by probability densities, and both types by cumulative distribution functions. These representations are closely related and together provide a full description of the random variable.

1.3.1 Probability mass function

A probability mass function gives the probability that a discrete random variable takes a specific value. The values of the function are nonnegative and sum to one over the entire set of possible outcomes. It is the most direct representation for counts and other variables with separated values.

1.3.2 Probability density function

A probability density function describes how probability is distributed over a continuous range of values. The probability of an exact point is typically zero; instead, probabilities are obtained by integrating the density over an interval. Peaks in the density indicate regions where values are more concentrated.

1.3.3 Cumulative distribution function

A cumulative distribution function gives the probability that a random variable is less than or equal to a given value. It increases from 0 to 1 as the variable moves across its range. This function applies to both discrete and continuous distributions and is often used to compare or transform random variables.

1.4 Support and range

The support of a distribution is the set of values that can occur with nonzero probability or positive density. The range refers more generally to the set of values a variable may take under the model. Identifying the support is important for interpreting the distribution and for choosing appropriate mathematical tools.

2 Types of probability distributions

Probability distributions are often grouped by whether they are discrete, continuous, or a combination of both. Each type is suited to particular kinds of data and phenomena. Many common models arise because they are simple, mathematically tractable, and useful in applications.

2.1 Discrete distributions

Discrete distributions describe variables that take separate, countable values. They are used for counts, binary outcomes, and waiting times measured in numbers of trials. Their probabilities are specified by a probability mass function.

2.1.1 Bernoulli distribution

The Bernoulli distribution models a single trial with two possible outcomes, often labeled success and failure. It is determined by one parameter giving the probability of success. This distribution is a basic building block for more complex binary and binomial models.

2.1.2 Binomial distribution

The binomial distribution gives the number of successes in a fixed number of independent Bernoulli trials. It depends on the number of trials and the success probability. It is widely used for repeated yes-or-no experiments and proportion data.

2.1.3 Geometric distribution

The geometric distribution describes the number of trials needed to obtain the first success in repeated independent Bernoulli trials. It has a memoryless property, meaning the future waiting time does not depend on how long the process has already continued. This makes it useful for simple waiting-time models.

2.1.4 Poisson distribution

The Poisson distribution models the number of events occurring in a fixed interval when events happen independently at an average rate. It is often used for arrivals, counts, and rare-event processes. Its simplicity and flexibility make it one of the most common count distributions.

2.2 Continuous distributions

Continuous distributions describe variables that can take any value within an interval. Probabilities are assigned to ranges rather than to individual points. They are used for measurements such as time, length, and temperature.

2.2.1 Uniform distribution

The uniform distribution assigns equal density across a specified interval. It is often used as a simple model of complete indifference among values in that range. Although idealized, it is useful in simulation and as a starting point for more complex models.

2.2.2 Normal distribution

The normal distribution is a bell-shaped continuous distribution characterized by its mean and variance. It appears frequently in natural and measurement processes and is central to many statistical methods. Its mathematical properties make it especially important in theory and applications.

2.2.3 Exponential distribution

The exponential distribution models waiting times between events in a process with a constant average rate. It is continuous and memoryless, making it closely related to the Poisson process. It is often used in reliability and survival analysis.

2.2.4 Gamma distribution

The gamma distribution is a flexible continuous family used for positive-valued data. It includes the exponential distribution as a special case and can model skewed waiting times and durations. Its shape can vary widely depending on its parameters.

2.3 Mixed distributions

Mixed distributions contain both discrete and continuous components. A variable may have point masses at certain values and a continuous density over an interval. Such models arise in applications where some outcomes occur exactly, while others vary smoothly.

3 Properties of distributions

Distributions are often summarized by numerical characteristics that describe their center, spread, shape, and other features. These properties help compare models and interpret data. They also guide estimation and decision-making.

3.1 Parameters

Parameters are fixed numerical values that determine the form of a distribution. Changing a parameter can shift the location, stretch the scale, or alter the shape. A distribution family is often defined by the parameters it accepts.

3.1.1 Location parameters

Location parameters determine where a distribution is centered on the number line. They typically shift the entire distribution without changing its overall form. The mean or median may move as the location changes.

3.1.2 Scale parameters

Scale parameters control the spread of a distribution. Larger scale values usually produce wider, more dispersed outcomes, while smaller values concentrate probability more tightly. They are especially important when comparing distributions with similar shapes.

3.1.3 Shape parameters

Shape parameters affect asymmetry, tail weight, and the overall profile of a distribution. Unlike location and scale, they can change the curve’s form in more fundamental ways. Many flexible families use one or more shape parameters to adapt to different datasets.

3.2 Moments

Moments are numerical summaries derived from powers of a random variable. They provide information about the center and structure of a distribution. The first few moments are the most commonly used.

3.2.1 Mean

The mean is the expected or average value of a random variable. It gives a central location for the distribution when the expectation exists. In practical settings, it is often used as a representative value.

3.2.2 Variance

Variance measures the typical squared deviation from the mean. A larger variance indicates more dispersion, while a smaller variance indicates tighter clustering. The square root of the variance is the standard deviation.

3.2.3 Skewness

Skewness measures asymmetry in a distribution. Positive skewness indicates a longer right tail, while negative skewness indicates a longer left tail. A nearly symmetric distribution has skewness close to zero.

3.2.4 Kurtosis

Kurtosis describes the concentration of values near the center and the heaviness of tails. Higher kurtosis is often associated with more extreme outcomes, though interpretations depend on context. It is commonly used alongside skewness to summarize shape.

3.3 Quantiles and percentiles

Quantiles divide a distribution into portions with equal probability. Percentiles are a common quantile scale based on hundredths. These measures are useful for comparing values across distributions and for identifying thresholds such as medians and upper-tail cutoffs.

3.4 Symmetry and tail behavior

Symmetry refers to whether a distribution mirrors itself around a central point. Tail behavior describes how rapidly probabilities decrease far from the center. Heavy-tailed distributions assign more probability to extreme values than light-tailed ones.

4 Operations on distributions

Distributions can be transformed, combined, and conditioned in various ways. These operations are essential for building models from simpler components. They also help connect theory with observable quantities.

4.1 Transformation of random variables

A transformation applies a function to a random variable to produce a new one. Common examples include scaling, shifting, taking logarithms, or squaring. The resulting distribution is determined by the original distribution and the transformation rule.

4.2 Sums of random variables

The distribution of a sum depends on the distributions of the individual terms and their dependence structure. Sums of independent variables are especially important in probability and statistics. Many classical results, including approximation theorems, are based on sums.

4.3 Convolution

Convolution is the operation used to find the distribution of a sum of independent random variables. It combines two distributions into a new one by integrating or summing over all compatible pairs of values. This concept is central in signal processing, statistics, and probabilistic modeling.

4.4 Mixture distributions

A mixture distribution combines several component distributions using specified weights. It can represent heterogeneous populations or processes with multiple regimes. Mixtures often produce richer shapes than a single standard family can capture.

4.5 Conditional distributions

A conditional distribution describes the distribution of one variable after another variable has been fixed or observed. It is fundamental to inference, dependence modeling, and sequential analysis. Conditional probability provides the basis for updating uncertainty in light of new information.

5 Special families of distributions

Some distribution families play a particularly important role because of their structure, generality, or connection to other mathematical ideas. They often appear in theory, computation, and applied modeling. Many of them also serve as building blocks for more advanced methods.

5.1 Exponential family

The exponential family is a broad class of distributions that can be written in a common canonical form. It includes many familiar models such as the normal, binomial, Poisson, and exponential distributions. This family is important because it often leads to convenient estimation and inference procedures.

5.2 Multivariate distributions

Multivariate distributions describe random vectors with several components. They are used when variables are observed jointly and may influence one another. Such models capture dependence, covariance, and interactions among multiple dimensions.

5.2.1 Joint distributions

A joint distribution gives the probability structure for two or more random variables simultaneously. It contains complete information about how the variables behave together. From the joint distribution, many other summaries can be derived.

5.2.2 Marginal distributions

A marginal distribution describes one component of a multivariate system without reference to the others. It is obtained by summing or integrating over the remaining variables. Marginals show the behavior of individual variables on their own.

5.2.3 Conditional distributions

In the multivariate setting, conditional distributions describe one or more variables given the values of others. They are essential for prediction and for representing dependence structures. Conditional models also underpin many iterative estimation algorithms.

5.2.4 Covariance and correlation

Covariance measures how two variables vary together, while correlation standardizes that relationship to a dimensionless scale. Positive values indicate that variables tend to increase together, and negative values indicate opposite movement. These measures summarize linear association but not all forms of dependence.

5.3 Copulas

Copulas are functions that separate marginal behavior from dependence structure. They allow the construction of multivariate distributions with specified marginals and flexible associations. Copulas are especially useful when dependence is not well represented by simple correlation alone.

6 Estimation and fitting

In practical work, a theoretical distribution is often fitted to observed data. Estimation aims to determine unknown parameters, while fitting evaluates how well the model matches reality. These tasks are central to statistical analysis and prediction.

6.1 Parameter estimation

Parameter estimation uses data to infer the numerical values that define a distribution. Different methods emphasize different criteria, such as likelihood, moments, or prior information. The choice of method can affect accuracy, robustness, and interpretability.

6.1.1 Maximum likelihood estimation

Maximum likelihood estimation chooses parameters that make the observed data most probable under the model. It is widely used because of its generality and favorable large-sample properties. In many common distributions, it leads to efficient and interpretable estimates.

6.1.2 Method of moments

The method of moments matches theoretical moments to sample moments. It is often simpler to apply than likelihood-based methods and can provide good initial estimates. Its main advantage is algebraic ease, though it may be less efficient in some settings.

6.1.3 Bayesian estimation

Bayesian estimation combines observed data with prior beliefs about parameters. The result is a posterior distribution over parameter values rather than a single point estimate. This framework is useful when uncertainty about parameters should be represented explicitly.

6.2 Goodness of fit

Goodness of fit assesses how closely a proposed distribution matches observed data. It may involve visual checks, test statistics, or residual analysis. A good fit does not prove correctness, but it supports the model as a reasonable approximation.

6.3 Model selection

Model selection compares multiple candidate distributions or families to choose the most suitable one. Criteria may balance fit and complexity to avoid overfitting. Common approaches include information criteria, cross-validation, and diagnostic comparison.

7 Applications

Probability distributions are used wherever uncertainty, variability, or random variation must be analyzed. They provide a common language for describing data and building predictive models. Their applications extend across science, engineering, and quantitative decision-making.

7.1 Statistical inference

In statistical inference, distributions underlie estimation, hypothesis testing, and confidence intervals. They describe the sampling behavior of statistics and help quantify uncertainty. Many inferential methods rely on assumptions about the form of the data-generating distribution.

7.2 Machine learning

Machine learning uses distributions to model data generation, classification uncertainty, and prediction error. Probabilistic methods appear in generative models, Bayesian learning, and stochastic optimization. Distributions also help evaluate uncertainty in predictions and hidden structure in data.

7.3 Risk analysis

Risk analysis uses distributions to represent uncertain losses, returns, or event frequencies. Tail behavior is especially important because extreme outcomes can dominate decisions. Distributional models support simulation, stress testing, and quantitative planning.

7.4 Reliability engineering

Reliability engineering studies failure times, lifetimes, and maintenance intervals using probability distributions. Models such as the exponential and gamma distributions are often employed for components and systems. These tools help estimate durability and schedule inspections or replacements.

In the natural sciences, distributions are used to model measurements, errors, and physical processes. In the social sciences, they help describe counts, survey responses, and behavioral variation. Across both areas, distributions provide a foundation for empirical analysis and theory testing.

Probability distributions are connected to several broader ideas in mathematics and statistics. These related concepts clarify how distributions are defined, used, and generalized. They also link single-variable models to more advanced probabilistic structures.

8.1 Probability measures

A probability measure assigns probabilities to events in a mathematically consistent way. It is the abstract foundation on which distributions are built. A probability distribution can be seen as the representation of such a measure for a random variable.

8.2 Random processes

A random process describes a family of random variables indexed by time, space, or another parameter. Each fixed index has its own distribution, and the collection reveals how the system evolves. Random processes are essential in time series, stochastic modeling, and physics.

8.3 Distribution functions in statistics

Distribution functions are used in statistics to summarize data, compare models, and compute probabilities. They include empirical distribution functions derived from samples as well as theoretical forms used in inference. These functions bridge raw observations and probabilistic models.

8.4 Law of large numbers and central limit theorem

The law of large numbers explains why sample averages tend to stabilize near expected values as sample size increases. The central limit theorem shows that sums and averages often approach a normal distribution under broad conditions. Together, these results justify many standard statistical approximations and procedures.