1 Basic concepts
Joint distributions describe how several random variables behave together in one probability model. Instead of assigning probabilities to a single outcome variable, they assign probability structure to tuples of outcomes. This makes it possible to study whether variables move independently, tend to occur together, or show more complicated patterns of association.
1.1 Random variables and vectors
A random variable maps outcomes of a random experiment to numerical values. When two or more random variables are considered together, they can be arranged as a random vector. A random vector provides a compact way to represent multivariate uncertainty, such as height and weight measured on the same person or the results of several repeated trials.
Random vectors may have two components, as in a bivariate setting, or many components in higher-dimensional models. The same mathematical framework applies in each case, although calculations become more elaborate as the number of variables increases.
1.2 Joint probability distribution
The joint probability distribution of several random variables specifies the probability law for their combined values. It answers questions such as the chance that one variable falls in a range while another lies in a different range, or the likelihood of a particular pair of values occurring together.
1.2.1 Discrete joint distributions
For discrete variables, the joint distribution assigns probability to each possible combination of values. These probabilities are collected in a joint probability mass function, which must be nonnegative and sum to 1 over all allowed value pairs or tuples.
1.2.2 Continuous joint distributions
For continuous variables, the joint distribution is described by a density rather than by point probabilities. Probabilities are obtained by integrating the density over a region of the variable space. Individual points typically have probability 0, but intervals and other measurable sets may have positive probability.
1.2.3 Mixed distributions
Some random vectors contain both discrete and continuous components. Such mixed distributions arise in settings where a variable may include a point mass at one value and a continuous spread over other values. These cases require combining summation and integration when computing probabilities.
1.3 Support and domain
The support of a joint distribution is the set of values where the probability or density is nonzero. The domain is the set of all values the variables are allowed to take, which may be larger than the support. Understanding support helps identify impossible combinations, geometric constraints, and regions where calculations must be performed.
2 Representation of joint distributions
Joint distributions can be represented in several equivalent or complementary ways. The most convenient representation depends on whether the variables are discrete, continuous, or mixed, and on what kind of probability question is being asked.
2.1 Joint probability mass function
For discrete random variables, the joint probability mass function gives the probability of each exact outcome combination. It is commonly written as a function of all variables and can be displayed in a table for two variables or in higher-dimensional arrays for more variables.
2.2 Joint probability density function
For continuous random variables, the joint probability density function describes how probability is distributed across a region of space. It is not itself a probability, but integrating it over a set yields the probability that the random vector lies in that set. The shape of the density often reveals concentration, spread, and dependence patterns.
2.3 Joint cumulative distribution function
The joint cumulative distribution function gives the probability that each variable is less than or equal to a specified value. It provides a complete description of the distribution and is defined for both discrete and continuous cases.
2.3.1 Properties of the joint CDF
A joint cumulative distribution function is nondecreasing in each argument and takes values between 0 and 1. It approaches 0 when any threshold goes to the lower limit of the support and approaches 1 when all thresholds move beyond the upper range of the distribution. It also satisfies consistency conditions that make it suitable for describing valid probability laws.
2.3.2 Relationship to marginal distributions
Marginal distribution functions can be obtained from the joint cumulative distribution function by letting the other variables move to their full range. This relationship makes the joint CDF a comprehensive object from which individual behavior can be recovered.
2.4 Probability tables and contour plots
Probability tables are useful for low-dimensional discrete distributions because they show all joint probabilities in a compact form. For continuous distributions, contour plots and surface plots provide a visual summary of how density is distributed across the plane or higher-dimensional slices. These graphical tools help identify concentration, symmetry, and dependence.
3 Marginal distributions
Marginal distributions describe the behavior of one variable taken alone, even when it is part of a larger joint model. They are obtained by aggregating over the possible values of the other variables.
3.1 Derivation from a joint distribution
The marginal distribution of a variable follows directly from the joint distribution. In a discrete setting, this is done by summing over the values of the remaining variables. In a continuous setting, it is done by integrating over them. Marginals summarize individual behavior without discarding the influence of the larger system.
3.2 Marginalization in discrete cases
For discrete random variables, marginalization is performed by summing joint probabilities across rows, columns, or higher-dimensional index sets. This process collapses the joint table into a one-variable distribution. The resulting probabilities still total 1.
3.3 Marginalization in continuous cases
For continuous variables, marginal densities are obtained by integrating the joint density over the unwanted coordinates. This can often be viewed as projecting the density onto one axis. The marginal may have a shape very different from the joint density, especially when dependence is present.
4 Conditional distributions
Conditional distributions describe the behavior of one random variable when the value of another is known or fixed. They are central to prediction, updating, and sequential reasoning.
4.1 Conditional probability mass function
In the discrete case, the conditional probability mass function gives the probability of one outcome given a specific value of another variable. It is computed by dividing the joint probability by the appropriate marginal probability, provided the conditioning event has positive probability.
4.2 Conditional probability density function
For continuous variables, the conditional density is defined using the ratio of the joint density to the marginal density of the conditioning variable, where the marginal is positive. It describes how probability is distributed after incorporating information about another variable.
4.3 Conditional cumulative distribution function
The conditional cumulative distribution function gives the probability that one variable falls below a threshold given the value of another variable. It is derived by integrating the conditional density or summing the conditional mass function and is useful for threshold-based queries.
4.4 Bayes' theorem and conditioning
Bayes' theorem relates conditional probabilities in opposite directions. In joint-distribution settings, it allows one to update beliefs about one variable after observing another. This idea underlies many statistical models, classification rules, and inference procedures.
5 Dependence and independence
Joint distributions reveal whether variables act independently or show dependence. This distinction affects prediction, estimation, and interpretation of data.
5.1 Independence of random variables
Random variables are independent when their joint distribution factorizes into the product of their marginals. Independence means that knowledge of one variable does not alter the probability distribution of another. It is a strong property and is often assumed in simplified models.
5.2 Dependence structure
When variables are not independent, the joint distribution encodes their dependence structure. Dependence may be positive, negative, nonlinear, symmetric, or localized in certain regions of the support. Different joint models can have the same marginals but very different dependence behavior.
5.3 Correlation and covariance
Covariance and correlation summarize linear association between variables. They do not fully describe a joint distribution, but they provide useful numerical measures of how variables move together.
5.3.1 Covariance matrix
For a random vector, the covariance matrix contains the covariances of every pair of components. Its diagonal entries are variances, and its off-diagonal entries show pairwise relationships. The matrix is fundamental in multivariate analysis, especially for normal models and dimensionality reduction.
5.3.2 Pearson correlation coefficient
The Pearson correlation coefficient standardizes covariance by the product of the standard deviations. Its value lies between -1 and 1, with values near the extremes indicating strong linear association. A value near 0 indicates little linear relation, though other forms of dependence may still be present.
5.4 Joint moments
Joint moments are expectations of products of powers of random variables. They include means, variances, covariances, and higher-order quantities. Joint moments can characterize shape, spread, and interaction effects, and are frequently used in theoretical and applied probability.
6 Transformations of random variables
New random variables are often formed by applying functions to jointly distributed variables. The resulting distributions can be derived from the original joint distribution.
6.1 Functions of jointly distributed variables
If several variables are jointly distributed, any function of them is also a random variable. This includes sums, products, ratios, maxima, and indicator functions. Determining the distribution of such functions often begins with the original joint law.
6.2 Distribution of sums and differences
The distribution of a sum or difference of random variables depends on the joint distribution. If the variables are independent, convolution methods are often used. If they are dependent, their association must be incorporated into the calculation. Such transformations are common in error propagation and signal processing.
6.3 Change of variables
A change of variables replaces one set of random variables with another related set. The transformed distribution is obtained by adjusting the original density or mass according to the mapping between the variables.
6.3.1 Jacobian method
For continuous transformations, the Jacobian determinant accounts for stretching or compression under the change of coordinates. It is essential when converting densities between coordinate systems or transforming multivariate random vectors.
6.3.2 Multivariate transformations
Multivariate transformations may involve several new variables defined from several old ones. When the mapping is one-to-one and differentiable, the transformed joint density can often be computed directly. If the mapping is not one-to-one, contributions from all valid preimages must be included.
7 Expectation with joint distributions
Expectations based on joint distributions generalize ordinary averages to multivariate settings. They provide numerical summaries of combined behavior.
7.1 Joint expectation
Joint expectation refers to the expected value of a function of several random variables. The function may be a product, sum, or more complicated expression. This concept is broader than marginal expectation because it incorporates relationships among variables.
7.2 Mixed moments
Mixed moments involve powers of more than one variable, such as the expectation of the product of two components. They are useful for describing dependence, skewness, and higher-order structure in multivariate data.
7.3 Conditional expectation
Conditional expectation gives the average value of one variable or function of variables after conditioning on another variable. It can be seen as a probability-weighted prediction based on available information. In many models, it serves as the best mean-square predictor.
7.4 Law of total expectation
The law of total expectation states that an expectation can be computed by averaging conditional expectations. It connects local conditional behavior with global averages and is often used to simplify calculations or build recursive arguments.
8 Common examples and families
Several standard joint distributions appear repeatedly in probability and statistics. They serve as building blocks for more complex models and as reference points for understanding dependence.
8.1 Bernoulli and categorical variables
Joint distributions of Bernoulli or categorical variables describe multiple binary or class-label outcomes. These models are used for surveys, classification tasks, and repeated trials. Their discrete structure makes them straightforward to tabulate and analyze.
8.2 Bivariate normal distribution
The bivariate normal distribution is a two-variable continuous model characterized by means, variances, and correlation. Its contours are elliptical, and its marginal distributions are normal. It is widely used because of its mathematical tractability and its role as an approximation in many settings.
8.3 Multivariate normal distribution
The multivariate normal distribution extends the bivariate case to many dimensions. It is determined by a mean vector and covariance matrix and has strong analytical properties. Linear combinations of multivariate normal variables are also normally distributed, which makes the family especially useful in theory and applications.
8.4 Multivariate uniform distribution
A multivariate uniform distribution assigns equal density over a specified region, such as a rectangle, box, or more general bounded set. It is useful for modeling complete uncertainty within a known range and for generating random samples in simulation.
8.5 Binomial and multinomial settings
Joint distributions in binomial and multinomial settings arise from repeated trials with one or more categories of outcomes. The multinomial model generalizes the binomial case to several categories and is often used in counts, contingency tables, and categorical data analysis.
9 Applications
Joint distributions are central in areas where multiple quantities must be studied together. They provide the probabilistic foundation for inference, simulation, and modeling.
9.1 Statistical inference
In statistical inference, joint distributions describe data models for multiple observations or variables. They are used to construct estimators, test hypotheses, and build confidence procedures. Many inferential methods depend on specifying the joint law of the observed sample.
9.2 Simulation and Monte Carlo methods
Simulation methods generate samples from joint distributions to approximate probabilities, expectations, and complicated integrals. Monte Carlo techniques are especially helpful when analytic calculations are difficult. They also allow researchers to explore dependence and uncertainty numerically.
9.3 Risk analysis
In risk analysis, joint distributions help measure how several uncertain factors interact. This is important in finance, engineering, insurance, and reliability studies, where combined outcomes matter more than isolated ones. Dependence can strongly affect the likelihood of extreme events.
9.4 Machine learning and data analysis
Machine learning and data analysis often rely on joint distributions to model inputs and outputs together. Joint probabilistic models support classification, regression, clustering, and generative modeling. They also help describe missing data, latent variables, and uncertainty in predictions.