1 Definition and basic properties
1.1 Estimator as a random variable
In statistical inference, an estimator is a function of the observed data. Because the data are random, the estimator itself is treated as a random variable. This perspective allows uncertainty about the estimator to be quantified using tools from probability theory, including expectations, variances, and covariances.
1.2 Variance as uncertainty around the parameter
Let \(\theta\) be the target parameter and \(\hat{\theta}\) an estimator of \(\theta\). The variance of \(\hat{\theta}\), written \(\mathrm{Var}(\hat{\theta})\), measures the dispersion of the estimator’s sampling distribution around its mean \(\mathbb{E}[\hat{\theta}]\). When an estimator is unbiased, this dispersion is directly around the parameter value. More generally, variance captures one component of total estimation error.
1.3 Relationship to standard deviation and confidence intervals
The standard deviation \(\mathrm{sd}(\hat{\theta})=\sqrt{\mathrm{Var}(\hat{\theta})}\) is the square root of the variance and is often easier to interpret in the same units as the parameter. Under approximate normality (or exact normality in some settings), variance determines the scale of confidence intervals: wider intervals correspond to larger variance.
1.4 Variance under linear transformations
Variance behaves predictably under linear operations. If \(a\) and \(b\) are constants, then for a scalar estimator, \[ \mathrm{Var}(a\hat{\theta}+b)=a^2\mathrm{Var}(\hat{\theta}). \] For sums of estimators, variance depends not only on each individual variance but also on their covariance. These transformation rules are foundational for deriving variance of transformed estimators.
2 Variance of common estimators
2.1 Sample mean
For independent and identically distributed observations \(X_1,\dots,X_n\) with mean \(\mu\) and variance \(\sigma^2\), the sample mean \(\bar{X}\) satisfies \[ \mathrm{Var}(\bar{X})=\frac{\sigma^2}{n}. \] This inverse sample-size relationship reflects the averaging effect: averaging reduces random fluctuation.
2.2 Sample proportion
For Bernoulli data, where \(Y_i\in\{0,1\}\) with success probability \(p\), the sample proportion \(\hat{p}=\frac{1}{n}\sum_{i=1}^n Y_i\) has \[ \mathrm{Var}(\hat{p})=\frac{p(1-p)}{n}. \] In practice, \(p\) is unknown and is often replaced by \(\hat{p}\) for estimation of variance.
2.3 Sample variance and unbiasedness considerations
The sample variance is typically formed as \[ s^2=\frac{1}{n-1}\sum_{i=1}^n (X_i-\bar{X})^2, \] which is designed to be unbiased for \(\sigma^2\) under standard assumptions. The variance of \(s^2\) depends on higher moments of the underlying distribution (e.g., kurtosis), meaning that it is not determined solely by \(\sigma^2\). This highlights why variance estimation can be more complex for variance-like statistics than for mean-like statistics.
2.4 Linear (weighted) estimators
Consider a linear estimator \(\hat{\theta}=\sum_{i=1}^n w_i X_i\) with weights \(w_i\). If observations are independent, \[ \mathrm{Var}(\hat{\theta})=\sum_{i=1}^n w_i^2\,\mathrm{Var}(X_i). \] If observations have correlation, cross terms involving covariances appear. Choice of weights can therefore meaningfully alter estimator variance, which motivates variance-reduction strategies such as weighting schemes.
2.5 Ratio-type and transformed estimators (overview)
Ratio estimators and other nonlinear transformations often have variance that cannot be written in a simple closed form without approximation. A common approach is to apply linearization (e.g., via the delta method) to obtain asymptotic variance. These estimators are frequent in settings such as estimating totals from sampled data, where the target involves dividing random quantities.
3 Computing estimator variance
3.1 Using the variance operator rules
For scalar random variables, variance rules include:
- \(\mathrm{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}[X])^2\)
- \(\mathrm{Var}(X+Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)+2\mathrm{Cov}(X,Y)\)
- \(\mathrm{Var}(aX)=a^2\mathrm{Var}(X)\)
These identities allow derivations once the joint behavior of the components is characterized.
3.2 Independence and covariance decomposition
When random variables are independent, their covariance is zero, simplifying variance calculations. In many practical derivations, observations are independent within a sampling scheme, but dependence may arise through shared estimators (e.g., using \(\bar{X}\) inside \(s^2\)). Recognizing where dependence exists prevents incorrect simplifications.
3.3 Conditioning and the law of total variance
Conditioning can simplify computations. The law of total variance states: \[ \mathrm{Var}(X)=\mathbb{E}[\mathrm{Var}(X\mid Z)]+\mathrm{Var}(\mathbb{E}[X\mid Z]). \] This decomposition is useful when variance arises from two sources: randomness within groups or conditional on some latent quantity, plus randomness across those conditions.
3.4 Using moments (moment-based variance calculation)
Because variance depends on the first two moments, many calculations proceed by deriving \(\mathbb{E}[\hat{\theta}]\) and \(\mathbb{E}[\hat{\theta}^2]\). For sums and products, moment methods may require combinatorial expansions or known formulas for distributions. Moment-based approaches can be exact but often become algebraically intensive.
3.5 Matrix form for vector-valued estimators
For a vector estimator \(\hat{\boldsymbol{\theta}}\), variance generalizes to covariance matrices: \[ \mathrm{Cov}(\hat{\boldsymbol{\theta}})=\mathbb{E}\big[(\hat{\boldsymbol{\theta}}-\mathbb{E}[\hat{\boldsymbol{\theta}}])(\hat{\boldsymbol{\theta}}-\mathbb{E}[\hat{\boldsymbol{\theta}}])^\top\big]. \] These matrices describe variance along every direction of the parameter space, not just the marginal variances.
3.5.1 Covariance matrices and quadratic forms
Covariance matrices often enter through quadratic forms. For a fixed vector \(\boldsymbol{a}\), the variance of \(\boldsymbol{a}^\top \hat{\boldsymbol{\theta}}\) is \[ \mathrm{Var}(\boldsymbol{a}^\top \hat{\boldsymbol{\theta}})=\boldsymbol{a}^\top \mathrm{Cov}(\hat{\boldsymbol{\theta}})\,\boldsymbol{a}. \] This form is central in constructing test statistics and deriving uncertainty for linear combinations of parameters.
4 Variance, bias, and mean squared error
4.1 Bias–variance decomposition
If the estimator has bias \(\mathrm{Bias}(\hat{\theta})=\mathbb{E}[\hat{\theta}]-\theta\), then \[ \mathbb{E}[(\hat{\theta}-\theta)^2]=\mathrm{Var}(\hat{\theta})+\mathrm{Bias}(\hat{\theta})^2. \] This decomposition separates randomness (variance) from systematic error (bias).
4.2 Mean squared error (MSE) as total estimation error
The mean squared error (MSE) is the expected squared distance between estimator and truth: \[ \mathrm{MSE}(\hat{\theta})=\mathbb{E}[(\hat{\theta}-\theta)^2]. \] MSE is frequently used as a single-number summary that incorporates both variance and bias, enabling fair comparison between methods that may trade off these components differently.
4.3 Trade-offs between bias and variance
A common pattern is that variance-reduction methods can introduce bias. Conversely, unbiased estimators may have high variance. The decision of which estimator is preferable depends on which notion of error is most relevant for the problem, with MSE often serving as a pragmatic objective.
4.4 Efficiency and comparative variance
When comparing estimators, one goal is to identify an estimator with smaller variance among those in a given class, or to minimize MSE. In large samples, notions such as asymptotic efficiency relate to variance behavior scaled appropriately, often leading to comparisons based on limiting covariance.
5 Asymptotic variance and large-sample approximations
5.1 Asymptotic normality and variance interpretation
Many estimators become approximately normal as sample size grows: \[ \sqrt{n}(\hat{\theta}-\theta)\ \xrightarrow{d}\ \mathcal{N}(0,\,V). \] In this setting, \(V\) is the asymptotic variance and describes the limiting spread of the appropriately rescaled estimation error.
5.2 Consistency and stabilization of variance
Consistency ensures \(\hat{\theta}\to \theta\) in probability. Together with asymptotic results, this implies that although the variance may shrink with \(n\), the scaled variance (such as \(n\,\mathrm{Var}(\hat{\theta})\)) can converge to a stable limit.
5.3 Delta method for asymptotic variance
If \(\hat{\theta}\) is approximately normal and a smooth function \(g(\cdot)\) is applied, then for \(g(\hat{\theta})\), \[ \sqrt{n}(g(\hat{\theta})-g(\theta))\ \approx\ \mathcal{N}(0,\,[g'(\theta)]^2 V). \] The derivative \(g'(\theta)\) translates uncertainty from the original estimator to the transformed scale, making the delta method a central tool for variance of transformed estimators.
5.4 Asymptotic covariance for multiple parameters
For vector estimators, asymptotic covariance matrices capture how uncertainty in one parameter component relates to others. If \(\sqrt{n}(\hat{\boldsymbol{\theta}}-\boldsymbol{\theta})\) converges to a multivariate normal distribution with covariance matrix \(\boldsymbol{\Sigma}\), then any linear transformation \(\boldsymbol{A}\hat{\boldsymbol{\theta}}\) has limiting covariance \(\boldsymbol{A}\boldsymbol{\Sigma}\boldsymbol{A}^\top\).
6 Variance reduction techniques
6.1 Averaging and replication effects
Independent replications and averaging reduce variance because averaging aggregates multiple draws. If two independent unbiased estimators of the same quantity are averaged, the resulting variance decreases, often roughly in proportion to the number of replications.
6.2 Control variates (conceptual overview)
Control variates improve estimator precision by adding a term whose expectation is known (or easily estimated) and whose randomness is correlated with the target quantity. By calibrating the strength of the control, one can reduce variance without changing the expectation, lowering MSE when bias is kept minimal.
6.3 Stratification and resampling approaches (overview)
Stratification divides data into homogeneous subgroups and combines within-stratum results. When strata are constructed so that variability is lower inside each group, the overall estimator variance can drop. Resampling-based variance reduction can also occur when averaging over bootstrap replicates or using sampling designs that balance observations to reduce dispersion.
6.4 Regularization and shrinkage effects on variance
Regularization methods often introduce bias to gain stability. Shrinkage can reduce variance by constraining estimates, especially in high-dimensional or ill-posed problems. The net effect is typically assessed through MSE or risk rather than variance alone.
7 Estimating the variance in practice
7.1 Plug-in (plug-in variance estimation)
In many settings, variance formulas depend on unknown parameters. Plug-in methods replace unknown quantities with consistent estimates. For example, replacing \(p\) by \(\hat{p}\) in \(\frac{p(1-p)}{n}\) yields an estimated variance for a proportion.
7.2 Bootstrap variance estimation
Bootstrap procedures approximate the sampling distribution by repeatedly resampling from the observed data (with replacement) and recalculating the estimator. The variance of the bootstrap replicates estimates the variance of the original estimator. Bootstrap validity depends on conditions such as smoothness and sample size.
7.3 Jackknife variance estimation
The jackknife systematically omits one observation (or small blocks) at a time, producing a set of leave-one-out (or leave-group-out) estimates. The variability among these estimates is used to approximate the variance. Jackknife methods can be effective for certain smooth estimators and for bias correction.
7.4 Robust (heteroskedasticity-aware) variance concepts
In many real-world analyses, equal-variance assumptions fail. Robust variance concepts aim to provide valid standard errors under heteroskedasticity or other forms of misspecification of the error structure.
7.4.1 Sandwich/robust covariance estimator (overview)
A commonly used robust approach is the sandwich estimator, which combines estimates of a model’s variability and sensitivity to produce covariance estimates that remain valid under weaker assumptions. While the name highlights a generic structure, the implementation depends on the modeling framework.
8 Dependence, covariance, and multistage sampling
8.1 Variance with correlated observations
If data are correlated, variance of sums and averages includes covariance terms. For instance, the variance of \(\bar{X}\) becomes \[ \mathrm{Var}(\bar{X})=\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n \mathrm{Cov}(X_i,X_j), \] showing that dependence can inflate or reduce uncertainty depending on the sign and magnitude of correlations.
8.2 Covariance and cross-terms in estimator variance
For an estimator built from multiple components, covariance between components contributes cross-terms. Ignoring these cross-terms can lead to underestimation of uncertainty. Correct variance computation therefore requires knowledge of joint structure, at least through covariance.
8.3 Clustered or grouped data (high-level)
When observations are clustered (e.g., multiple measurements within subjects or units), within-cluster dependence tends to be stronger than between-cluster dependence. This structure affects effective sample size: the number of independent clusters may be the relevant scale for variance, rather than the total number of observations.
8.4 Multistage estimator variance decomposition
8.4.1 Between-stage and within-stage components
In multistage sampling or hierarchical measurement schemes, variability arises from different levels—for example, randomness within selected units (within-stage) and randomness due to which units were selected (between-stage). Decomposing variance into these components clarifies where most uncertainty is generated and guides design improvements.
9 Applications and interpretive guidance
9.1 When variance dominates error behavior
Variance is often the primary driver of error when estimators are nearly unbiased and sample sizes are moderate. In such cases, improving precision (more data, better sampling design, variance reduction methods) tends to yield the most noticeable gains.
9.2 Comparing estimators with similar bias
When two estimators have comparable bias, variance becomes a direct comparator through MSE. Reporting both variance estimates and an MSE summary helps distinguish whether one method’s advantage comes from reduced dispersion or from smaller systematic error.
9.3 Choosing sample size based on variance goals
Variance frequently determines how large a study must be to reach a desired precision level. Since many variance expressions scale approximately like \(1/n\) under ideal assumptions, sample size planning often uses these relationships to set targets for standard errors or confidence interval widths.
9.4 Communicating uncertainty to nontechnical audiences
Nontechnical audiences benefit from uncertainty descriptions that connect variance to practical consequences: how much estimates might vary across repeated experiments, and what range is plausible. Effective communication typically emphasizes the meaning of confidence intervals or prediction bands rather than algebraic variance formulas.
10 Worked examples (conceptual calculations)
10.1 Variance of the sample mean under i.i.d. assumptions
Assume \(X_1,\dots,X_n\) are i.i.d. with \(\mathbb{E}[X_i]=\mu\) and \(\mathrm{Var}(X_i)=\sigma^2\). The sample mean is \(\bar{X}=\frac{1}{n}\sum X_i\). Using independence, \[ \mathrm{Var}(\bar{X})=\mathrm{Var}\left(\frac{1}{n}\sum_{i=1}^n X_i\right)=\frac{1}{n^2}\sum_{i=1}^n \mathrm{Var}(X_i)=\frac{\sigma^2}{n}. \] This illustrates how averaging reduces fluctuation.
10.2 Variance using law of total variance
Let \(Z\) represent a latent or conditioning variable that affects the variability of \(X\). Applying the law of total variance, \[ \mathrm{Var}(X)=\mathbb{E}[\mathrm{Var}(X\mid Z)]+\mathrm{Var}(\mathbb{E}[X\mid Z]). \] The first term captures variability within each condition, while the second captures variability of the conditional mean across conditions. This decomposition is particularly helpful in hierarchical and grouped settings.
10.3 Computing asymptotic variance via delta method
Suppose \(\hat{\theta}\) estimates \(\theta\) with asymptotic behavior \[ \sqrt{n}(\hat{\theta}-\theta)\Rightarrow \mathcal{N}(0,V). \] For a smooth transformation \(g\), the delta method gives \[ \sqrt{n}(g(\hat{\theta})-g(\theta))\Rightarrow \mathcal{N}(0,[g'(\theta)]^2V). \] Thus the asymptotic variance of \(g(\hat{\theta})\) is obtained by scaling \(V\) by the squared local derivative.
10.4 Comparing two estimators by variance and MSE
Consider two estimators \(\hat{\theta}_1\) and \(\hat{\theta}_2\) with biases \(b_1,b_2\) and variances \(v_1,v_2\). Their MSE values are \[ \mathrm{MSE}(\hat{\theta}_k)=v_k+b_k^2,\quad k=1,2. \] If \(v_1<v_2\) but \(b_1^2\) is larger, either estimator may be preferable. A comparison therefore depends on the combined error measure, not variance alone.