Goodness of fit - Ciallo Wiki

1. Definition and purpose

1.1 Basic meaning

Goodness of fit describes the degree to which a proposed model or statistical distribution reproduces observed data. When the model’s outputs align closely with measurements—without systematic gaps or structure in the discrepancies—the fit is considered good. When differences are systematic or large relative to expected random variation, the fit is poor.

1.2 Role in scientific modeling

In scientific practice, goodness of fit helps evaluate whether an assumed form—such as a parametric distribution, a functional relationship, or a theoretical expectation—captures key features of empirical observations. It supports decision-making about whether to retain a model, revise assumptions, or seek a different specification.

1.3.1 Fit versus accuracy

Fit and accuracy are related but not identical. Fit focuses on agreement between a model and the data used to assess it, often in terms of how well predicted quantities match observed ones. Accuracy typically emphasizes overall correctness of predictions, which may depend on how the model performs on new or unseen data.

1.3.2 Fit versus predictive power

A model may show excellent goodness of fit on a training dataset yet generalize poorly. Predictive power refers to performance on future observations, often assessed through validation or testing. Thus, goodness of fit is an evidence of in-sample correspondence, while predictive power concerns out-of-sample behavior.

1.3.3 Fit versus overfitting

Overfitting occurs when a model becomes tailored to random noise rather than the underlying signal. Such models can produce high goodness-of-fit statistics while failing to remain reliable under resampling or new data. Methods that penalize complexity or evaluate fit across folds are often used to mitigate this risk.

2. Statistical foundations

2.1 Observed and expected values

At the core of goodness of fit is a comparison between observed quantities (data) and expected quantities (model-implied values). Expected values can refer to means in regression, category probabilities in discrete distributions, or cumulative probabilities in distributional checks. The strength of evidence against a model depends on how far these comparisons deviate beyond what random sampling would plausibly produce.

2.2 Residuals and error terms

Many goodness-of-fit approaches are built around residuals, which quantify discrepancies between what is observed and what the model predicts.

2.2.1 Types of residuals

Residuals appear in multiple forms: raw residuals (observed minus predicted), standardized residuals (scaled by an estimate of variability), and studentized or otherwise adjusted versions used to compare discrepancies across observations. In regression, residuals can also be defined for different components, such as errors in transformed response scales.

2.2.2 Interpretation of residual patterns

Random noise typically produces residuals that fluctuate without obvious structure. Systematic patterns—such as curved trends across fitted values, changing spread with level, or repeated runs of positive versus negative residuals—indicate that the model omits structure. Plot-based diagnostics often reveal these issues more readily than a single summary statistic.

2.3 Sampling variation

Goodness of fit is inherently probabilistic because observed data are realizations from random processes. Even when the model is correct, sample fluctuations cause imperfect agreement. Statistical tests and reference distributions describe how large a discrepancy would be expected under the model, distinguishing genuine misfit from normal variability.

2.4 Null and alternative hypotheses

Most formal goodness-of-fit tests are framed as hypothesis tests. The null hypothesis states that the data follow the proposed model (or distribution), possibly with specified parameters. The alternative hypothesis asserts that the proposed model does not adequately describe the data, typically capturing deviations such as incorrect functional form or mismatched distributional shape.

3. Common goodness-of-fit tests

3.1 Chi-squared test

The chi-squared test evaluates discrepancies between observed and expected counts, especially in contingency tables or binned data.

3.1.1 Applications for categorical data

For categorical variables, the method compares observed frequencies in each category to expected frequencies derived from a hypothesized probability model. It is also widely used when continuous data are grouped into intervals, with expected bin probabilities supplied by a candidate distribution.

3.1.2 Assumptions and limitations

The classical chi-squared approximation relies on sufficiently large expected counts and correct binning choices. If categories have very small expected frequencies, the approximation may be inaccurate. Additionally, binning continuous data introduces sensitivity: different bin widths or cut points can lead to different conclusions.

3.2 Kolmogorov–Smirnov test

The Kolmogorov–Smirnov (K–S) test compares empirical and theoretical distribution functions, focusing on the maximum deviation across the range.

3.2.1 One-sample test

In the one-sample version, the empirical distribution function from the data is compared with a specified reference distribution, such as a normal or exponential law, possibly after parameter estimation.

3.2.2 Two-sample test

In the two-sample version, the empirical cumulative distribution functions of two samples are compared directly to assess whether they plausibly arise from the same underlying distribution.

3.3 Anderson–Darling test

The Anderson–Darling test is a distributional goodness-of-fit procedure that emphasizes tail behavior by weighting discrepancies more heavily near the ends of the distribution. This makes it useful when tail mismatch is scientifically or practically important.

3.4 Shapiro–Wilk test

The Shapiro–Wilk test is commonly used to assess whether a dataset is consistent with a normal distribution. It is often applied as part of residual diagnostics in regression or to evaluate normality assumptions in statistical procedures.

3.5 Likelihood-based tests

Likelihood-based goodness-of-fit checks rely on comparing model likelihoods, sometimes within a framework such as chi-squared approximations derived from differences in log-likelihood. These methods can accommodate complex models but require careful attention to parameter estimation, regularity conditions, and sample-size adequacy.

4. Goodness of fit in regression

4.1 Coefficient of determination

In regression, the coefficient of determination, commonly denoted R-squared, summarizes how much variance in the response is explained by the model.

4.1.1 Interpretation of R-squared

R-squared represents the proportion of total variability in the observed response accounted for by the regression model. Larger values indicate that the model’s predictions track the observed outcomes more closely, though interpretation depends on context, model form, and whether the model includes an intercept.

4.1.2 Adjusted R-squared

Adjusted R-squared modifies R-squared by accounting for the number of predictors relative to sample size. It tends to discourage adding predictors that do not meaningfully improve fit, providing a more conservative measure for comparing models of differing complexity.

4.2 Residual standard error

Residual standard error estimates the typical magnitude of residuals in the response scale. It provides a sense of how close predictions are, on average, to observed values, assuming an error model and degrees of freedom consistent with the fitted regression.

4.3 Analysis of residual plots

Residual plots provide qualitative checks that complement numerical metrics.

4.3.1 Nonlinearity

If the mean relationship is misspecified—such as using a straight line when curvature exists—residuals plotted against fitted values may show systematic curvature or trends. This suggests that a different functional form, transformation, or feature engineering may be needed.

4.3.2 Heteroscedasticity

Heteroscedasticity occurs when residual spread changes with the level of predictors or fitted values. Residuals may form a funnel pattern, signaling that the constant-variance assumption underlying many regression methods is violated.

4.3.3 Outliers and influential points

Some observations may deviate strongly from the rest of the data, producing large residuals. Others can be influential by strongly affecting parameter estimates even if their residual is moderate. Diagnostics such as leverage and influence measures are often used alongside residual plots to identify these cases.

5. Goodness of fit for probability distributions

5.1 Parameter estimation

When model parameters are unknown, they are often estimated from the data before assessing fit. This introduces additional uncertainty: discrepancy statistics may have different reference distributions than they would under known parameters. Many procedures incorporate corrections or rely on asymptotic behavior to address this effect.

5.2 Distributional comparison

Distributional goodness-of-fit aims to determine whether data plausibly follow a candidate distribution. In addition to classical tests, one can compare predicted and empirical quantiles, cumulative probabilities, or moments. The choice of summary measure influences which types of deviations—central, skewness, or tail—are most readily detected.

5.3 Graphical diagnostics

5.3.1 Histograms and density overlays

Overlaying a fitted probability density or expected histogram onto observed data gives an intuitive view of where mismatch occurs. While useful, the result depends on histogram binning and bandwidth choices, and it can be less sensitive than formal tests for detecting subtle discrepancies.

5.3.2 Q-Q plots

Quantile–quantile (Q–Q) plots compare sample quantiles against theoretical quantiles from the fitted distribution. Deviations from a straight reference line suggest mismatched location, scale, skewness, or tail thickness.

5.3.3 P-P plots

Probability–probability (P–P) plots compare cumulative probabilities rather than quantiles. They help reveal differences in distribution shape by showing whether the empirical cumulative function tracks the theoretical one across the range.

6. Model assessment and selection

6.1 Information criteria

Information criteria provide a way to compare models by balancing goodness of fit with model complexity. They are typically derived from likelihood principles and include penalties for additional parameters.

6.1.1 Akaike information criterion

The Akaike information criterion (AIC) rewards better fit but imposes a penalty proportional to the number of estimated parameters. Lower AIC values generally indicate a preferable model among the candidates, particularly in contexts where the goal is predictive accuracy.

6.1.2 Bayesian information criterion

The Bayesian information criterion (BIC) also balances fit and complexity, often applying a stronger penalty than AIC for larger parameter counts. This can lead to selecting simpler models, depending on sample size and modeling assumptions.

6.2 Cross-validation

Cross-validation evaluates model performance by repeatedly training on subsets of data and assessing fit on held-out portions. This yields an estimate of how well the model reproduces new observations, reducing the risk that apparent in-sample fit results from overfitting.

6.3 Penalization and complexity control

Regularization techniques introduce penalties on model parameters or complexity, encouraging smoother or sparser solutions. These approaches can improve generalization, indirectly influencing goodness-of-fit behavior by discouraging overly flexible models that match noise.

6.4 Trade-off between fit and parsimony

Model selection often reflects a trade-off: increasing flexibility can improve fit but may harm reliability on new data. Parsimonious models aim for adequate representation with fewer assumptions or parameters, supported by validation or information criteria that incorporate complexity costs.

7. Graphical methods

7.1 Residual plots

Residual plots are a primary visual tool for diagnosing where a model fails. They can reveal structure not captured by aggregate statistics and help guide remedial steps such as transformation, alternative link functions, or adding interaction terms.

7.2 Fitted versus observed plots

A fitted versus observed plot displays how predicted values correspond to actual measurements. Clustering around the diagonal line indicates strong agreement, while systematic deviations can point to bias, scaling issues, or model mismatch.

7.3 Quantile-quantile plots

Q–Q plots are widely used to assess distributional assumptions, especially normality of residuals or other derived quantities. Their shape makes it clear whether discrepancies are concentrated in tails or affect the center.

7.4 Probability plots

Probability plots, including P–P plots and related cumulative probability graphics, compare empirical and theoretical cumulative behavior. These can be more sensitive to certain distributional differences than histogram-based displays, while still offering interpretable visual guidance.

8. Interpretation and limitations

8.1 Sensitivity to sample size

Many goodness-of-fit tests are sensitive to the number of observations. With very large samples, even small deviations from a model may yield statistically significant results. Conversely, with small samples, tests may lack power and fail to detect meaningful misfit.

8.2 Effect of violations of assumptions

Test statistics and regression diagnostics often rely on assumptions about error structure, independence, or distributional form. When assumptions are violated, p-values and fit metrics may not correspond to their nominal interpretation, making it necessary to use robust methods or alternative diagnostics.

8.3 Practical versus statistical significance

A statistically significant lack of fit does not automatically imply practical irrelevance, and a non-significant result does not guarantee adequacy. Evaluating the magnitude of discrepancies, their impact on decisions, and the consequences for downstream tasks helps distinguish meaningful from purely statistical departures.

8.4 Common misinterpretations

Common pitfalls include treating a single p-value as a measure of model quality, ignoring residual structure, and conflating in-sample fit with generalization. Another frequent error is selecting models solely by fit metrics without accounting for complexity, leading to overly optimistic conclusions.

9. Applications

9.1 Natural sciences

In physics, chemistry, and biology, goodness-of-fit assesses whether observed experimental measurements match theoretical predictions or assumed noise models. It supports selecting parametric forms for distributions of measurements and verifying hypotheses about underlying mechanisms.

In studies of survey responses, behavioral data, or event counts, goodness-of-fit checks whether assumed distributions or regression specifications align with observed patterns. It helps evaluate competing models, such as alternative ways to represent heterogeneity or discrete choice behavior.

9.3 Engineering and quality control

Manufacturing and systems engineering use goodness-of-fit to evaluate whether measurements follow expected distributions, whether sensor noise behaves as modeled, and whether processes remain stable. It is often used to detect deviations that could indicate calibration issues, drift, or process faults.

9.4 Machine learning and data science

In modern workflows, goodness-of-fit connects to model diagnostics, calibration, and validation. Although machine learning often emphasizes predictive performance rather than classical distributional assumptions, techniques such as calibration curves, residual checks, and validation-based selection all reflect the underlying goal: ensuring model predictions are consistent with observed data patterns.