1 Definition
Covariance is a numerical measure of how two variables vary together. It summarizes whether larger values of one variable tend to occur with larger values of another, whether they tend to move in opposite directions, or whether no clear linear tendency is present. In probability and statistics, covariance is used to describe dependence in a way that is sensitive to the scale of the variables.
1.1 Random variables and joint variability
For random variables, covariance describes joint variability rather than behavior of each variable separately. It is based on the idea that paired outcomes can be compared to their typical values, and deviations from those typical values can be examined together. When two variables are observed repeatedly across many trials or cases, covariance captures the extent to which their departures from average values align.
1.2 Intuitive interpretation
If one variable tends to be above its mean when the other is also above its mean, the covariance is typically positive. If one tends to be above its mean when the other is below its mean, the covariance is typically negative. If the variables do not show a consistent linear pattern, the covariance is near zero, though this does not necessarily imply independence.
1.3 Sign and magnitude
The sign of covariance indicates direction: positive for same-direction movement and negative for opposite-direction movement. The magnitude depends on both the strength of joint movement and the units used to measure the variables. Because it is not standardized, a larger numerical value does not automatically mean a stronger relationship unless the variables are on comparable scales.
2 Mathematical formulation
Covariance can be defined for populations described by probability distributions and for samples drawn from data. The formulas differ slightly depending on whether the goal is to describe an entire distribution or estimate it from observed values.
2.1 Population covariance
Population covariance refers to the true covariance of two random variables under a probability model. It is a theoretical quantity defined in terms of expected values.
2.1.1 Expectation-based definition
For random variables \(X\) and \(Y\), the covariance is commonly defined as the expected value of the product of their centered versions:
\[ \operatorname{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])] \]
This expression measures how deviations from the mean occur together. If the centered values tend to have the same sign, the expected product is positive; if they tend to have opposite signs, it is negative.
2.1.2 Equivalent algebraic forms
Expanding the centered product gives an equivalent formula:
\[ \operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y] \]
This form is often convenient for theoretical work and derivations. It shows that covariance compares the average of the product with the product of the averages. The two expressions are mathematically identical when the required expectations exist.
2.2 Sample covariance
Sample covariance is computed from observed data and is used to estimate population covariance. Suppose paired observations \((x_1,y_1), \dots, (x_n,y_n)\) are available.
2.2.1 Unbiased estimator
A common estimator of covariance is
\[ s_{xy}=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}) \]
where \(\bar{x}\) and \(\bar{y}\) are sample means. The denominator \(n-1\) makes the estimator unbiased under standard assumptions for random sampling.
2.2.2 Alternative computational formula
An equivalent expression is
\[ s_{xy}=\frac{1}{n-1}\left(\sum_{i=1}^{n}x_i y_i-n\bar{x}\bar{y}\right) \]
This form is useful for computation because it avoids repeated subtraction of means in some settings. However, it can be less numerically stable when values are large and nearly cancel each other.
2.3 Covariance matrix entry
In multivariate settings, covariance appears as an entry in a covariance matrix. The \((i,j)\) element is the covariance between the \(i\)th and \(j\)th variables. Diagonal entries are variances, while off-diagonal entries measure pairwise joint variation.
3 Properties
Covariance has several basic algebraic properties that make it useful in theory and computation. These properties explain how it behaves under common transformations of variables.
3.1 Symmetry
Covariance is symmetric:
\[ \operatorname{Cov}(X,Y)=\operatorname{Cov}(Y,X) \]
This reflects the fact that joint variation does not depend on the order of the two variables.
3.2 Linearity
Covariance is linear in each argument. For constants \(a\), \(b\), and random variables \(X\), \(Y\), and \(Z\),
\[ \operatorname{Cov}(aX+bY,Z)=a\,\operatorname{Cov}(X,Z)+b\,\operatorname{Cov}(Y,Z) \]
A similar rule holds in the second argument. This property is widely used in derivations involving sums of random variables and linear models.
3.3 Effect of shifting variables
Adding a constant to one variable does not change covariance:
\[ \operatorname{Cov}(X+c,Y)=\operatorname{Cov}(X,Y) \]
for any constant \(c\). Covariance depends on deviations from the mean, so simple shifts in location have no effect.
3.4 Effect of scaling variables
Multiplying a variable by a constant scales covariance by the same factor:
\[ \operatorname{Cov}(aX,bY)=աբ\,\operatorname{Cov}(X,Y) \]
where \(a\) and \(b\) are constants. This shows why covariance depends on measurement units; changing units such as meters to centimeters alters the numerical value.
3.5 Covariance of a variable with itself
The covariance of a variable with itself equals its variance:
\[ \operatorname{Cov}(X,X)=\operatorname{Var}(X) \]
Variance is therefore a special case of covariance. It measures spread around a single mean rather than joint variation between two variables.
4 Relationship to correlation
Covariance and correlation are closely connected, but they serve slightly different purposes. Correlation is a standardized version of covariance that is easier to compare across different variable pairs.
4.1 Standardization
Correlation divides covariance by the product of the standard deviations:
\[ \rho_{XY}=\frac{\operatorname{Cov}(X,Y)}{\sigma_X \sigma_Y} \]
This rescales the measure so that it is unitless and bounded between -1 and 1 when the standard deviations are nonzero. Standardization removes the dependence on units.
4.2 Strength versus direction
The sign of covariance and correlation both indicate direction. Positive values suggest variables move together; negative values suggest opposite movement. Correlation is better for comparing strength across datasets because it removes scale effects, whereas covariance is more natural in algebraic formulas and matrix methods.
4.3 Covariance and Pearson correlation coefficient
The Pearson correlation coefficient is the most familiar standardized form of covariance. It measures linear association only, not all kinds of dependence. A covariance of zero may correspond to zero linear association, while correlation of zero similarly indicates no linear relationship but not necessarily independence.
5 Computational methods
Covariance is commonly computed from raw data, matrices, or streaming summaries. The chosen method depends on dataset size, numerical precision, and implementation goals.
5.1 Direct calculation from data
The most straightforward method uses the sample means and the centered products of paired observations. Each pair is adjusted by subtracting its respective mean, and the results are averaged with the appropriate denominator. This method is easy to understand and suitable for small to moderate datasets.
5.2 Matrix-based computation
For data arranged in a matrix with observations in rows and variables in columns, covariance matrices can be computed efficiently using linear algebra. Centering the data matrix and multiplying it by its transpose yields a compact representation of all pairwise covariances. This approach is common in statistical software and multivariate analysis.
5.3 Numerical stability considerations
When values are very large or nearly equal, naive formulas may lose precision through subtractive cancellation. More stable algorithms update means and covariance incrementally, reducing rounding error. Such methods are valuable in large-scale computing and in streaming contexts where data arrive sequentially.
6 Applications
Covariance is used across statistics and applied mathematics whenever joint variability matters. It helps summarize data, build models, and identify structure in multivariate systems.
6.1 Descriptive statistics
In exploratory analysis, covariance provides a quick summary of whether two measured quantities move together. It is often used alongside scatterplots and correlation coefficients to understand patterns in data before formal modeling.
6.2 Regression analysis
Covariance plays a role in linear regression, especially in formulas for slopes and in the interpretation of fitted relationships. The dependence between predictors and responses influences estimated coefficients, and covariance matrices are central to assessing uncertainty in parameter estimates.
6.3 Principal component analysis
Principal component analysis relies on the covariance matrix to identify directions of greatest variability in multivariate data. Eigenvectors of the covariance matrix define principal components, while eigenvalues indicate how much variation each component captures.
6.4 Portfolio theory
In finance, covariance helps describe how asset returns move together. It is used to study diversification, since assets with low or negative covariance can reduce overall portfolio variability. Covariance matrices are central to many models of risk and asset allocation.
6.5 Time series analysis
For time-dependent data, covariance is used to examine relationships across time lags and between multiple series. It appears in autocovariance functions, cross-covariance functions, and in models that describe temporal dependence.
7 Covariance matrix
The covariance matrix is a compact representation of pairwise covariances among several variables. It is one of the most important objects in multivariate statistics.
7.1 Definition and structure
For variables \(X_1, X_2, \dots, X_p\), the covariance matrix is a \(p \times p\) matrix whose \((i,j)\) entry is \(\operatorname{Cov}(X_i,X_j)\). Because covariance is symmetric, the matrix is symmetric as well. Its diagonal contains variances, and its off-diagonal elements show pairwise joint variation.
7.2 Positive semidefinite property
A covariance matrix is positive semidefinite. This means that for any vector of constants, the corresponding quadratic form is nonnegative. The property reflects the fact that variance of any linear combination of variables cannot be negative. It also ensures that covariance matrices have real, nonnegative eigenvalues.
7.3 Interpretation of diagonal and off-diagonal terms
Diagonal entries measure the variability of each variable individually. Larger diagonal values indicate greater spread. Off-diagonal entries indicate how pairs of variables move together: positive values suggest aligned movement, negative values suggest inverse movement, and values near zero suggest weak linear association.
8 Estimation and inference
Because covariance is usually estimated from finite samples, its value is subject to uncertainty. Statistical inference addresses how reliable a computed covariance may be.
8.1 Sampling variability
Different samples from the same population can produce different covariance estimates. The amount of variation depends on sample size, the underlying distribution, and the strength of dependence between variables. Larger samples generally yield more stable estimates.
8.2 Confidence intervals and hypothesis testing
In some settings, confidence intervals can be constructed for covariance or for related quantities such as correlation. Hypothesis tests may examine whether covariance differs from zero. These procedures usually rely on assumptions about the data-generating process or on asymptotic approximations.
8.3 Bias and consistency
The standard sample covariance with denominator \(n-1\) is unbiased under simple random sampling assumptions. As sample size increases, the estimator is typically consistent, meaning it converges to the population covariance. Alternative estimators may trade bias for lower variance or computational convenience.
9 Related concepts
Several concepts are closely linked to covariance and are often discussed together in statistical analysis.
9.1 Variance
Variance is the covariance of a variable with itself. It measures dispersion around the mean and is a special case of the broader covariance concept.
9.2 Correlation
Correlation standardizes covariance by dividing by the relevant standard deviations. It provides a scale-free measure of linear association that is easier to compare across variable pairs.
9.3 Conditional covariance
Conditional covariance measures joint variability after conditioning on another variable or set of variables. It is useful in models where dependence changes once additional information is taken into account.
9.4 Cross-covariance
Cross-covariance extends the idea to two different processes or series, often at different time lags. It is used in signal processing and time series analysis to study how one sequence relates to another over time.