Weighted RMSE

1 Definition and Intuition

1.1 From squared error to RMSE

Root Mean Squared Error (RMSE) is a widely used accuracy metric for regression-style predictions. Given a set of observations \(y_i\) and corresponding predictions \(\hat{y}_i\), the error for each observation is \(e_i = y_i - \hat{y}_i\). Squaring these errors penalizes larger deviations more strongly than smaller ones. RMSE then averages the squared errors and applies a square root, producing a single value with the same physical units as the original target (under standard, unweighted RMSE conventions).

1.2 Introducing weights in the error aggregation

Weighted RMSE modifies the averaging step by multiplying each squared error by a weight \(w_i\). The intent is to change how much each observation contributes to the final score. Observations can be weighted more heavily if they are more important, more reliable, or more costly to get wrong. Conversely, less relevant or noisier observations can be down-weighted.

1.3 Interpretation of weighted error contributions

Because the metric is based on a weighted average of squared errors, each term’s influence grows with both its squared error magnitude and its associated weight. After summation, the square root rescales the result, so the final number can be interpreted as an error magnitude in the original units, though the interpretation depends on whether weights are normalized (see Section 2.3).

1.4 Relationship to unweighted RMSE

Unweighted RMSE is a special case of weighted RMSE when all weights are equal (and normalized in a compatible way). In that situation, each observation contributes identically to the mean squared error before square-rooting.

2 Mathematical Formulation

2.1 Weighted RMSE formula

A common form of weighted RMSE for \(n\) observations is \[ \mathrm{WRMSE}=\sqrt{\frac{\sum_{i=1}^{n} w_i (y_i-\hat{y}_i)^2}{\sum_{i=1}^{n} w_i}}. \] This expression weights squared errors and divides by the total weight, yielding a weighted mean squared error before taking the square root.

2.1.1 Vector/matrix notation for weighted errors

Let \(\mathbf{e}=\mathbf{y}-\hat{\mathbf{y}}\) be the vector of residuals and let \(\mathbf{W}\) be a diagonal matrix with entries \(w_i\). Then \[ \mathrm{WRMSE}=\sqrt{\frac{\mathbf{e}^\top \mathbf{W}\mathbf{e}}{\mathbf{1}^\top \mathbf{W}\mathbf{1}}}, \] where \(\mathbf{1}\) is the all-ones vector. This emphasizes that the calculation is a weighted quadratic form followed by normalization.

2.2 Choice of weighting scheme

Weights \(w_i\) are typically nonnegative. The selection depends on what the analyst wants to reflect: varying importance (task priorities), varying reliability (confidence in each measurement), or varying representativeness (sampling differences). The scheme should align with the meaning of the prediction targets and the evaluation goal.

2.3 Normalized vs. unnormalized weighted RMSE

Two closely related variants appear in practice:

* Normalized weighted RMSE (shown in Section 2.1): divides by \(\sum_i w_i\). This keeps the metric scale comparable across different weight totals. * Unnormalized weighted RMSE: uses \(\sqrt{\sum_i w_i (y_i-\hat{y}_i)^2}\) without dividing by total weight. This can change with the number of weighted terms or the magnitude of the weights, making cross-dataset or cross-experiment comparisons less direct.

For reporting and comparison, normalized weighted RMSE is often preferred unless the evaluation protocol explicitly justifies the unnormalized form.

2.4 Special cases and edge conditions

Several edge cases arise: * Zero-weight observations: If \(w_i=0\), the corresponding residual does not affect the score. * All weights zero: The denominator \(\sum_i w_i\) becomes zero; the metric is undefined and must be handled explicitly. * Negative weights: If negative weights are allowed, the “weighted mean squared error” interpretation breaks down and the square-rooted quantity may not behave as intended. Most evaluation settings constrain weights to be nonnegative.

3 Statistical Properties and Behavior

3.1 Sensitivity to outliers

Like RMSE, weighted RMSE inherits strong sensitivity to large residuals because squared errors magnify extreme deviations. Weighting can either increase or mitigate this sensitivity depending on whether outliers receive high or low weights.

3.2 Effect of weight magnitude on influence

The gradient of influence in the metric is not linear: doubling a weight doubles its contribution to the weighted squared-error sum, and because the metric is averaged then square-rooted, the final change is sublinear in simple cases. Still, large weights can dominate the score when paired with moderate residuals, effectively steering the evaluation toward particular subsets of the data.

3.3 Bias and variance considerations

Weighted RMSE is an evaluation criterion rather than an unbiased estimator of a single underlying error distribution in general. Its expectation depends on how weights relate to the data-generating process. If weights correlate with noise levels or difficulty, the metric may reflect a more “risk-aware” notion of error, but that comes with shifts in what is being averaged, affecting variance across samples.

3.4 Comparability across datasets

Comparing weighted RMSE values across datasets requires attention to: * whether the metric is normalized by \(\sum_i w_i\), * how weights are defined and scaled, * whether the evaluation sets contain different subsets or coverage patterns. Without consistent normalization and weight construction, differences in WRMSE may reflect changes in weighting rather than model performance.

4 Practical Use Cases

4.1 Heterogeneous observation importance

In many applications, some observations matter more because they are associated with higher business impact, safety relevance, or downstream decision cost. Weighted RMSE can prioritize these cases by assigning larger weights to their errors.

4.2 Time series with varying time-step relevance

For forecasting, certain horizons or periods may be more critical than others. Weights can emphasize near-term accuracy, penalize long-term drift more strongly, or reflect varying reliability of sensor readings across time steps.

4.3 Imbalanced data and class-dependent weighting

When evaluation involves targets that are not equally distributed (e.g., rare event categories), weights can correct for imbalance by giving more importance to underrepresented cases. In regression settings with categorical structure, weights may be assigned based on the target region or derived class label.

4.4 Measurement uncertainty and reliability weighting

If each observation has an associated uncertainty, weights can be chosen inversely proportional to that uncertainty. This makes the metric treat more reliable measurements as stronger contributors to the final score, aligning evaluation with measurement quality.

4.5 Aggregated or grouped predictions

For scenarios where predictions are assessed over groups (e.g., per region, per device, or per user segment), weights can be used to reflect group sizes or importance. The metric can then combine errors across groups while controlling for overrepresentation of large groups.

5 Implementation Considerations

5.1 Common pitfalls in coding weights

Common mistakes include: * forgetting to normalize by the sum of weights when using normalized weighted RMSE, * misaligning weights with residuals (indexing errors), * using weights computed on one dataset split but applied to another, * unintentionally broadcasting arrays in a way that pairs weights with the wrong dimension.

5.2 Handling missing or masked observations

In practical pipelines, some targets may be missing or invalid. A typical approach is to mask these entries and set their weights to zero (or exclude them from both numerator and denominator). Care must be taken so that the total weight in the denominator reflects only the included observations.

5.3 Ensuring consistent units and scaling

Because RMSE is unit-consistent under standard definitions, analysts should ensure that weighting and normalization preserve the intended interpretation. In normalized weighted RMSE, the square root of the weighted mean squared error typically maintains error units; unnormalized variants may scale with weight magnitude and thus lose direct unit comparability.

5.4 Numerical stability (large/small weights)

Very large or very small weights can cause numerical issues when summing products like \(w_i e_i^2\). Remedies include: * using floating-point types with sufficient precision, * rescaling weights by a constant factor (while preserving normalized behavior), * applying stable masking and denominator checks to avoid division by extremely small totals.

6 Weight Selection Strategies

6.1 Manually specified importance weights

Weights can be set based on domain knowledge: certain segments are deemed critical, some errors are more costly, and others can be tolerated. Manual weighting is transparent but depends on expert calibration.

6.2 Inverse-variance or uncertainty-based weights

If each observation has an estimated variance \(\sigma_i^2\), a common choice is \(w_i \propto 1/\sigma_i^2\). This strategy reduces the influence of noisy measurements under an assumption that uncertainty estimates are meaningful and comparable across observations.

6.3 Data-driven weighting heuristics

When uncertainty estimates are unavailable, heuristics may be used: * weights proportional to sample representativeness or coverage, * weights derived from observed residual patterns on a validation set, * weights adjusted to reduce the impact of systematic differences between groups. Such methods should be validated to ensure they do not inadvertently reward model failure in certain regions.

6.4 Tuning weights via validation

Weights can be tuned to maximize a downstream utility or improve evaluation alignment with target behavior. A model selection protocol may include a grid or optimization over a parameterized weighting scheme, evaluated on held-out data to avoid overfitting the metric itself.

6.5 Constraints for meaningful normalization

To keep WRMSE interpretable and comparable: * weights are often normalized to sum to 1 (or to a fixed total) as a preprocessing step, * nonnegativity constraints are applied, * denominators are checked to avoid zero totals, * consistent scaling is enforced across experimental runs.

7 Comparison with Related Metrics

7.1 Weighted MAE vs. weighted RMSE

Weighted Mean Absolute Error (weighted MAE) uses \(\sum_i w_i	e_i	\) in place of \(\sum_i w_i e_i^2\). Compared with weighted RMSE, MAE typically reduces sensitivity to outliers because absolute errors grow linearly. Choosing between them depends on whether large errors should be penalized sharply (RMSE) or more robustly (MAE).

7.2 Weighted MSE as the squared form

Weighted MSE is the quantity inside the square root of weighted RMSE: \[ \mathrm{WMSE}=\frac{\sum_i w_i (y_i-\hat{y}_i)^2}{\sum_i w_i}. \] Working with WMSE can be convenient for optimization and theoretical analysis, while WRMSE is often preferred for reporting due to unit interpretability.

7.3 Weighted R-squared and correlation-based measures

R-squared variants and correlation-based scores assess explained variance or linear association. They differ from WRMSE by focusing on relative fit structure rather than absolute error magnitude. Weighted R-squared attempts to incorporate weighting in variance decomposition, but its interpretation depends strongly on modeling assumptions and how weights relate to the variance.

7.4 Other loss functions: Huber and quantile losses

If squared errors are too sensitive to extremes, alternatives may be used: * Huber loss transitions between squared and absolute penalties, * quantile loss targets conditional quantiles rather than mean error. These can complement or replace weighted RMSE depending on the distributional goal.

7.5 When to prefer alternative metrics

Alternatives are often preferred when: * outliers dominate performance and should not be overly penalized, * the evaluation target is a quantile or distributional tail, * the interest is in ranking accuracy or correlation rather than metric magnitude.

8 Worked Example (Conceptual)

8.1 Single-output regression example

Assume three predictions with residuals \(e_1, e_2, e_3\). If all weights are equal, WRMSE reduces to the usual RMSE. If the second observation is considered twice as important as the others, set weights \(w_1=1, w_2=2, w_3=1\) and compute the normalized weighted mean of squared residuals before taking the square root. The resulting score will shift toward the magnitude of \(e_2\); if \(e_2\) is large, WRMSE increases relative to unweighted RMSE, whereas if \(e_2\) is small, WRMSE may decrease.

8.2 Multi-output or multi-horizon extension (overview)

For multi-output regression, weights can be applied per output dimension (e.g., different target variables) and/or per time horizon (e.g., different forecast steps). One approach is to flatten residuals across outputs and horizons into a single list and assign weights accordingly, then apply the same weighted RMSE formula. Another approach keeps a structured computation, such as averaging weighted errors within each horizon and then aggregating across horizons.

8.3 Interpreting changes in the metric

A change in WRMSE after introducing weights indicates that the evaluation is emphasizing a different subset or error regime. Analysts should examine which terms contribute most heavily by checking weighted squared residual contributions \(w_i e_i^2\). This provides clarity on whether the model improved where it matters most under the chosen weighting.

8.4 Demonstrating normalization effects

If the same residuals and weights are used but normalized versus unnormalized WRMSE is computed, only the normalized version remains stable under rescaling of weights. For instance, multiplying all weights by a constant factor leaves normalized weighted RMSE unchanged (assuming the standard normalized formula), but it scales unnormalized WRMSE by the square root of that constant.

9 Reporting and Visualization

9.1 How to report weighted RMSE in experiments

Reports should specify: * whether weighted RMSE is normalized, * the exact weighting formula used, * how weights were constructed (importance, uncertainty, masking), * whether results are aggregated over samples, time steps, outputs, or groups. Stating these details prevents confusion when comparing results across papers or experiments.

9.2 Confidence intervals and bootstrap ideas (overview)

To express uncertainty in WRMSE estimates, bootstrap resampling can be applied at the appropriate level (e.g., resampling sequences for time series, or resampling groups for grouped evaluations). Confidence intervals then reflect sampling variability of the weighted error metric under the evaluation design.

9.3 Residual plots with weight emphasis

Residual scatter plots can be augmented by visual cues such as point sizes proportional to weights. This helps readers see whether errors occur in heavily weighted regions or whether discrepancies concentrate in lightly weighted areas.

9.4 Per-group decomposition of weighted errors

For interpretability, WRMSE can be decomposed into contributions by group. For each group \(g\), compute the group’s weighted mean squared error (or the weighted sum of squared errors) and report how much it contributes to the overall numerator. Such breakdowns reveal where the model performs well or poorly under the weighting scheme.