1 Definition and basic concept
In statistics, a residual is the difference between an observed value and the corresponding value predicted by a model. It represents the portion of a data point that the model does not explain. Residuals are central to assessing model fit because they show how closely estimated values match the actual data.
1.1 Observed values and fitted values
An observed value is the recorded or measured outcome in a dataset. A fitted value is the value produced by a statistical model for that same observation. When a model captures the main structure of the data, fitted values tend to be close to observed values, and residuals are relatively small.
1.2 Residual formula
The most common residual is written as observed value minus fitted value:
residual = observed value - fitted value
This definition makes the sign of the residual meaningful. A positive result indicates that the model underpredicted the observation, while a negative result indicates overprediction.
1.3 Interpretation of positive and negative residuals
A positive residual means the actual value lies above the model’s prediction. A negative residual means it lies below. The size of the residual indicates the extent of the mismatch, and unusually large residuals may signal atypical observations or model inadequacy.
2 Residuals in regression analysis
Residuals are especially important in regression, where a response variable is modeled as a function of one or more predictors. They quantify the unexplained part of each response value after the regression equation has been applied.
2.1 Ordinary least squares residuals
In ordinary least squares regression, the fitted model is chosen to minimize the sum of squared residuals. This criterion favors a line or surface that keeps overall prediction errors as small as possible. The resulting residuals are used both in estimation and in later diagnostics.
2.2 Residuals in linear regression
In linear regression, each residual is the vertical distance between an observed point and the regression line or hyperplane. These distances show how far the data depart from the linear pattern assumed by the model.
2.2.1 Relationship to the regression line
Residuals are measured from the fitted regression line, not from the average of the data. Points that lie exactly on the fitted line have residuals of zero. Larger vertical gaps correspond to larger residual magnitudes.
2.2.2 Sum of residuals property
When a linear regression model includes an intercept and is fitted by ordinary least squares, the residuals sum to zero. This property reflects the balancing effect of positive and negative errors around the fitted line. It does not mean each residual is small, only that they cancel in aggregate.
2.3 Residuals in multiple regression
In multiple regression, residuals are defined in the same way, but the fitted values depend on several predictor variables rather than one. Residuals help determine whether the combined effect of the predictors adequately explains the response. They also reveal whether important structure remains in the data after accounting for all included variables.
3 Types of residuals
Different residual definitions are used for different purposes. Some adjust for scale, leverage, or model family, making them more suitable for comparison across observations or for specialized diagnostics.
3.1 Raw residuals
Raw residuals are the simple differences between observed and fitted values. They are easy to interpret and are often the starting point for diagnostic work. However, their scale may vary across observations, which can limit comparability.
3.2 Standardized residuals
Standardized residuals divide raw residuals by an estimate of their standard deviation. This produces a scale-free measure that is easier to compare across cases. Large standardized residuals often draw attention because they indicate observations that are far from what the model expects.
3.3 Studentized residuals
Studentized residuals are adjusted to account for the effect of each observation on the fitted model. They are commonly used to identify unusual points because they provide a more refined assessment than raw or standardized residuals. Their values are especially useful when some observations have high leverage.
3.4 Pearson residuals
Pearson residuals are used mainly in generalized linear models. They are formed by dividing the difference between observed and fitted values by an estimate of the standard deviation under the model. This makes them useful for comparing departures across observations with different expected variabilities.
3.5 Deviance residuals
Deviance residuals measure each observation’s contribution to the model’s deviance, a goodness-of-fit statistic common in likelihood-based modeling. They are often used in generalized linear models because they reflect how much each point contributes to lack of fit. Their sign indicates whether the model overestimates or underestimates the observation.
4 Residual analysis and diagnostics
Residual analysis examines patterns in the errors left after fitting a model. By studying these patterns, analysts can judge whether the model assumptions are plausible and whether the model should be revised.
4.1 Residual plots
Residual plots display residuals against fitted values, predictors, or other variables. They are among the most widely used tools in statistical diagnostics because they can reveal structure that summary measures may miss.
4.1.1 Residuals versus fitted values
A plot of residuals versus fitted values is often used to check whether errors are randomly scattered around zero. A random cloud suggests a reasonable fit, while curves, funnels, or clusters may indicate model problems. This plot is especially helpful for detecting nonlinearity or changing variance.
4.1.2 Residuals versus predictor variables
Plotting residuals against individual predictors can show whether a particular variable has been modeled appropriately. Systematic patterns may suggest missing terms, transformations, or interactions. These plots are useful when the relationship between a predictor and the response is more complex than the model assumes.
4.2 Detecting nonlinearity
Residual patterns can reveal when a linear form is too simple for the data. Curved trends in the residual plot often indicate that the model is missing nonlinear structure. In such cases, polynomial terms, transformations, or different model forms may improve fit.
4.3 Detecting unequal variance
A spread of residuals that widens or narrows across fitted values can indicate unequal variance, also known as heteroscedasticity. This means the error variability changes across the range of the data. Unequal variance can affect inference and may call for weighted methods or variance-stabilizing transformations.
4.4 Detecting outliers and influential observations
Residuals can identify outliers, which are observations far from the fitted pattern. Some unusual cases also have high influence, meaning they strongly affect the fitted model when included. Residual size alone does not determine influence, but large residuals are often a first signal that an observation deserves closer inspection.
5 Residuals in different statistical models
Residuals appear in many model families, though their exact form may vary with the assumptions and distributional structure of the model.
5.1 Residuals in generalized linear models
In generalized linear models, residuals are adapted to outcomes that may not follow a normal distribution. Because the response can be binary, count-based, or otherwise noncontinuous, several residual definitions are used. These residuals help assess whether the chosen link function and distribution are appropriate.
5.2 Residuals in time series models
In time series analysis, residuals are the differences between observed values and the model’s one-step-ahead predictions or fitted values. They are examined for autocorrelation because remaining serial dependence suggests the model has not captured all time-related structure. Ideally, residuals should resemble random noise.
5.3 Residuals in nonlinear models
Nonlinear models also produce residuals by comparing observed values with fitted values. Their diagnostic use is similar to that in linear models, but the interpretation may depend more strongly on the local shape of the fitted curve. Residual plots can help determine whether the nonlinear form is flexible enough.
5.4 Residuals in machine learning models
In machine learning, residuals are often called prediction errors, but the same idea applies: they are the differences between actual and predicted outcomes. They are used to evaluate training performance, compare models, and detect systematic mistakes. In ensemble and flexible models, residual analysis can still uncover bias, overfitting, or data issues.
6 Mathematical properties
Residuals have mathematical features that reflect how the model was estimated. These properties are important in understanding why residuals behave the way they do and how they relate to inferential quantities.
6.1 Orthogonality in least squares estimation
In least squares regression, residuals are orthogonal to the fitted values and, more generally, to the columns of the design matrix under standard conditions. This means the residual vector is geometrically perpendicular to the space spanned by the model predictors. The property underlies several useful identities in linear modeling.
6.2 Degrees of freedom
Residuals are connected to degrees of freedom because each estimated parameter reduces the amount of independent information left for error estimation. The number of residual degrees of freedom is typically the sample size minus the number of estimated parameters. This quantity is used when estimating variance and conducting tests.
6.3 Connection to error terms
The error term in a statistical model represents the unobserved deviation between the true data-generating process and the model’s systematic part. Residuals are the observable counterparts of those errors after fitting the model. They estimate, but do not exactly equal, the underlying error terms because the model parameters are themselves estimated from the data.
6.4 Residual variance and mean squared error
Residual variance summarizes the typical size of the unexplained deviations. Mean squared error is the average of squared residuals and is widely used as a measure of predictive accuracy. Squaring gives more weight to large discrepancies, making the measure sensitive to substantial misfits.
7 Practical applications
Residuals are used in both formal analysis and routine data work. They help determine whether a model is trustworthy, whether predictions are adequate, and whether the data contain unusual features.
7.1 Model checking
Residuals provide one of the most direct ways to check whether a model fits the data well. If they show random scatter with no obvious pattern, the model is often considered more plausible. Clear structure in the residuals usually suggests that the model should be revised.
7.2 Prediction assessment
For prediction, residuals measure how far forecasts miss the observed values. Smaller residuals indicate more accurate predictions on the data used for evaluation. Their distribution can also reveal whether prediction errors are stable across different regions of the data.
7.3 Hypothesis testing support
Residuals support hypothesis testing by helping verify the assumptions underlying test statistics. In regression, for example, they can indicate whether linearity, constant variance, or independence is reasonable. If these assumptions are violated, standard tests may become less reliable.
7.4 Data quality evaluation
Large or unusual residuals may point to recording errors, measurement problems, or data points that require review. Residual analysis can therefore serve as a practical quality-control tool. It helps distinguish ordinary variability from observations that merit closer examination.