Mean squared error - Ciallo Wiki

Overview

Mean squared error (MSE) is a fundamental measure in statistics and machine learning that quantifies the average squared difference between estimated values (e.g., predictions) and the true values. For a set of observations, MSE is calculated as the mean of the squares of the errors—where error is the difference between the predicted and actual value. MSE is non-negative, with values closer to zero indicating better fit. It is widely used as a loss function in regression tasks, as well as in estimation theory, signal processing, and quality assessment.

1 Definition and Notation

1.1 Formal definition

Let \(\hat{Y}\) denote an estimator or prediction of a true value \(Y\). For \(n\) observations, the MSE is defined as

\[ \text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2 . \]

For a continuous random variable, the MSE is the expected value of the squared error: \(\text{MSE} = \mathbb{E}\left[(Y - \hat{Y})^2\right]\). It is a scalar measure that averages the squared deviations between estimated and actual quantities.

1.2 Notation in statistics and machine learning

In statistics, MSE often denotes the mean squared error of an estimator of a parameter \(\theta\), written as \(\text{MSE}(\hat{\theta}) = \mathbb{E}[(\hat{\theta} - \theta)^2]\). In machine learning, MSE is commonly used as a regression loss function, sometimes written as \(L(y, \hat{y}) = (y - \hat{y})^2\) per sample, with the overall loss being the average over the dataset.

1.3 Relationship to variance and bias

1.3.1 Bias-variance decomposition

The MSE of an estimator can be decomposed into the sum of its variance and the square of its bias. Let \(\hat{\theta}\) be an estimator of \(\theta\). Then

\[ \text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + \left(\text{Bias}(\hat{\theta})\right)^2 , \]

where \(\text{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta\). This decomposition is fundamental in understanding the trade-off between model complexity and generalization error.

1.3.2 MSE = Var(estimator) + [Bias(estimator)]²

The formula above shows that even an unbiased estimator (bias = 0) can have high MSE if its variance is large. Conversely, a slightly biased estimator with much lower variance may achieve a smaller MSE. This concept is central to regularisation techniques such as ridge regression and shrinkage methods.

2 Properties of MSE

2.1 Non-negativity and scaling

MSE is always non-negative because it is an average of squared terms. It is not invariant under scaling of the variables: multiplying the true and predicted values by a constant \(c\) multiplies the MSE by \(c^2\). This property must be considered when comparing MSE across different units or scales.

2.2 Convexity and differentiability

As a function of predictions (or estimator parameters), MSE is convex and differentiable. The convexity ensures that gradient-based optimization (e.g., gradient descent) converges to a global minimum for linear models. The differentiability allows analytic solutions in ordinary least squares via the normal equations.

2.3 Sensitivity to outliers

Because errors are squared, large deviations (outliers) have a disproportionately large influence on MSE. A single extreme outlier can dominate the MSE value, making it less robust than measures such as mean absolute error (MAE). This sensitivity is both a strength (when large errors must be penalized heavily) and a weakness (in noisy real-world data).

3 Estimation of MSE

3.1 Sample MSE

3.1.1 Formula for finite samples

Given a sample of \(n\) observed pairs \((x_i, y_i)\) and predictions \(\hat{y}_i\), the sample MSE is

\[ \widehat{\text{MSE}} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 . \]

This is a consistent estimator of the true MSE as \(n\) grows large, assuming the predictions are independent of the errors.

3.2 Unbiased estimation of MSE

3.2.1 Correction for degrees of freedom

When the parameters of a model are estimated from the same data used to compute MSE, the sample MSE is biased downwards. An unbiased estimate of the error variance (often denoted \(\sigma^2\)) in a linear model is

\[ s^2 = \frac{1}{n-p} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 , \]

where \(p\) is the number of estimated parameters. However, this corrected version estimates the variance, not the MSE, which includes bias; for unbiased estimators, the two coincide on average.

3.3 Relationship to root mean squared error (RMSE)

The root mean squared error (RMSE) is the square root of MSE: \(\text{RMSE} = \sqrt{\text{MSE}}\). RMSE has the same units as the original variable, making it more interpretable. While MSE penalizes large errors quadratically, RMSE provides a normalized measure of average error magnitude.

4 MSE in Regression Analysis

4.1 MSE as a loss function

4.1.1 Ordinary least squares

In ordinary least squares (OLS) regression, the goal is to minimize the sum of squared errors (SSE), which is equivalent to minimizing MSE (up to a scaling factor of \(1/n\)). OLS yields the best linear unbiased estimator (BLUE) under the Gauss–Markov assumptions.

4.1.2 Gradient descent and convex optimization

For models without a closed-form solution (e.g., neural networks, nonlinear regression), MSE is minimized iteratively using gradient descent. Because MSE is convex in the parameters for linear models, gradient descent always converges to the global optimum. For nonlinear models, the MSE loss landscape may have local minima.

4.2 Comparison with other loss functions

4.2.1 Mean absolute error (MAE)

MAE uses absolute deviations: \(\frac{1}{n}\sum	y_i - \hat{y}_i	\). Compared to MSE, MAE is more robust to outliers but is not differentiable at zero. MSE penalizes large errors more heavily and is smoother, which can be advantageous for optimization.

4.2.2 Huber loss

Huber loss combines MSE and MAE: it behaves quadratically for small errors and linearly for large errors, controlled by a threshold parameter \(\delta\). This gives robustness to outliers while maintaining differentiability and convexity.

4.2.3 Quantile loss

Quantile loss (pinball loss) is used for quantile regression. It is asymmetric and penalizes over- and under-predictions differently. Unlike MSE, it does not minimize the conditional mean but a specified quantile.

4.3 Coefficient of determination (R²) and MSE

The coefficient of determination \(R^2\) is defined as \(1 - \frac{\text{MSE}_{\text{model}}}{\text{MSE}_{\text{baseline}}}\), where the baseline MSE is the variance of the observed data (using the mean as the predictor). Thus, MSE directly enters the computation of \(R^2\), which measures the proportion of variance explained by the model.

5 Applications

5.1 Signal processing

5.1.1 Wiener filter

The Wiener filter minimizes the MSE between the desired signal and the filtered output. It is optimal in the MSE sense under stationary noise assumptions and is widely used for noise reduction and system identification.

5.1.2 Mean squared error in denoising

In signal denoising, MSE measures the reconstruction quality. Lower MSE indicates closer approximation to the original clean signal. The method of wavelet thresholding, for example, often uses MSE as a criterion for selecting threshold levels.

5.2 Image and video quality assessment

5.2.1 Peak signal-to-noise ratio (PSNR) derived from MSE

PSNR is defined as \(10 \log_{10}\left(\frac{\text{MAX}^2}{\text{MSE}}\right)\), where MAX is the maximum pixel value. Thus PSNR is inversely related to MSE. Higher PSNR implies lower MSE and better perceived quality, though MSE alone does not perfectly correlate with human visual perception.

5.3 Forecasting and time series

In forecasting, MSE (or its square root) is a standard metric to evaluate prediction accuracy. It is used in model selection criteria such as AIC and BIC (as part of the likelihood). The MSE of out-of-sample forecasts is also used to assess model stability.

6 Limitations and Extensions

6.1 Sensitivity to scale and variance

MSE is scale-dependent: comparing MSE across datasets with different units or magnitudes is meaningless. It also does not account for heteroscedasticity—a constant MSE may hide different error variances across the range of predictions.

6.2 Weighted and normalized MSE

Weighted MSE assigns different importance to each observation: \(\frac{1}{n}\sum w_i (y_i - \hat{y}_i)^2\). Normalized MSE (e.g., NMSE = MSE / variance of observed values) allows comparison across datasets with different scales.

6.3 Mean squared logarithmic error (MSLE)

MSLE is defined as \(\frac{1}{n}\sum (\log(1+y_i) - \log(1+\hat{y}_i))^2\). It is used when the target has exponential growth or when relative errors are more important than absolute errors. MSLE penalizes underestimates more heavily than overestimates.

7.1 Root mean squared error (RMSE)

As the square root of MSE, RMSE is interpretable in original units. It is widely reported in weather forecasting, finance, and engineering. Because it squares errors, RMSE is more sensitive to outliers than MAE.

7.2 Mean absolute error (MAE)

MAE averages absolute errors. It is robust, easy to interpret, but not differentiable at zero. For normally distributed errors, MSE is more efficient than MAE.

7.3 Mean squared prediction error (MSPE)

MSPE evaluates the average squared difference between the predicted and actual values in a prediction context, often using cross-validation. It is essentially the same as MSE but emphasizes out-of-sample performance.

7.4 Mean integrated squared error (MISE)

MISE is used in nonparametric density estimation and curve smoothing. For a kernel density estimator \(\hat{f}_h\) of a true density \(f\), MISE is \(\int \mathbb{E}[(\hat{f}_h(x) - f(x))^2] dx\). It integrates MSE over the domain, providing a global measure of fit.