1 Concept and purpose

1.1 Definition

A goodness-of-fit test is a statistical hypothesis test used to evaluate whether observed data are consistent with a specified expected distribution or model. The test compares what was actually observed with what would be expected if the model were correct. Large discrepancies suggest that the model may not describe the data well.

1.2 Hypotheses in goodness-of-fit testing

Goodness-of-fit testing is framed in terms of competing hypotheses about the data-generating process. The null hypothesis states that the sample follows the proposed distribution, while the alternative states that it does not. The test is designed to measure whether deviations from the expected pattern are larger than would be reasonable by chance alone.

1.2.1 Null hypothesis

The null hypothesis usually asserts that the data come from a particular probability distribution, such as a normal, binomial, or Poisson distribution. In model-based settings, it may state that the observed frequencies match the predicted frequencies from a fitted model.

1.2.2 Alternative hypothesis

The alternative hypothesis states that the observed data do not follow the specified distribution. This may mean the shape is different, the probabilities across categories differ, or the model leaves systematic patterns unexplained.

1.3 Applications

Goodness-of-fit tests are used in many areas of statistics and data analysis. They help determine whether a theoretical distribution is plausible, whether a model is adequate, and whether data behave as expected under a given assumption.

1.3.1 Model validation

In model validation, these tests assess how well a statistical model matches the observed data. They are often used after fitting regression, counting, or probabilistic models to check whether residual patterns or frequency differences indicate lack of fit.

1.3.2 Distribution checking

A common use is to compare a sample to a named distribution, such as testing whether measurements are approximately normal or whether counts follow a Poisson pattern. This helps guide later analyses and method selection.

1.3.3 Categorical data analysis

For categorical data, goodness-of-fit tests compare observed category counts with expected counts. This is useful in quality control, survey analysis, genetics, and experimental studies where category proportions are of interest.

2 Types of goodness-of-fit tests

2.1 Chi-squared goodness-of-fit test

The chi-squared goodness-of-fit test is one of the most familiar methods for categorical data. It measures how far observed counts differ from expected counts across categories.

2.1.1 Observed and expected frequencies

Observed frequencies are the counts actually recorded in each category. Expected frequencies are the counts predicted by the null model, usually computed from hypothesized probabilities and the total sample size.

2.1.2 Test statistic

The chi-squared statistic is formed by summing the squared differences between observed and expected counts, divided by the expected counts. Larger values indicate greater disagreement between the sample and the model.

2.1.3 Degrees of freedom

Degrees of freedom depend on the number of categories and on how many parameters were estimated from the data. Estimating parameters reduces the degrees of freedom because it constrains the comparison between observed and expected values.

2.2 Kolmogorov–Smirnov test

The Kolmogorov–Smirnov test compares an empirical distribution with a theoretical one by examining the largest difference between their cumulative distribution functions. It is especially useful for continuous data.

2.2.1 One-sample version

In the one-sample version, the sample is compared with a fully specified theoretical distribution. The test focuses on the maximum vertical distance between the empirical cumulative distribution and the reference curve.

2.2.2 Comparison with theoretical distributions

This method is commonly used to assess whether data are compatible with a hypothesized continuous distribution. It is sensitive to overall shape differences, though its power can vary depending on where discrepancies occur.

2.3 Anderson–Darling test

The Anderson–Darling test is a distributional test that places more emphasis on differences in the tails than some alternatives. It is often used when tail behavior is especially important.

2.3.1 Tail sensitivity

Because it gives extra weight to extreme values, the Anderson–Darling test can detect departures from the expected distribution that might be missed by methods focused mainly on central values.

2.4 Cramér–von Mises test

The Cramér–von Mises test compares the empirical and theoretical cumulative distributions by averaging squared differences across the full range of the data. It provides a balanced measure of overall discrepancy.

2.5 Other specialized tests

Many specialized goodness-of-fit tests exist for particular data types or models. These include tests for discrete distributions, regression residuals, circular data, survival models, and other contexts where standard procedures are less suitable.

3 Test procedure

3.1 Formulating the hypotheses

The first step is to state the null and alternative hypotheses clearly. The null specifies the target distribution or model, and the alternative states that the observed data deviate in a meaningful way.

3.2 Choosing the significance level

A significance level is selected before computing the test, often set at 0.05 or another conventional value. This threshold determines how strong the evidence must be before rejecting the null hypothesis.

3.3 Calculating the test statistic

The appropriate test statistic is then calculated from the data. The formula depends on the chosen method and may use counts, cumulative differences, residuals, or other summaries of the discrepancy between observed and expected values.

3.4 Determining the p-value

The p-value represents the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null hypothesis is true. A small p-value suggests that the observed data are unlikely under the proposed model.

3.5 Making the decision

If the p-value is below the chosen significance level, the null hypothesis is rejected. If it is not below the threshold, the data are considered compatible with the model, though this does not prove that the model is correct.

4 Assumptions and conditions

4.1 Random sampling

Goodness-of-fit tests generally assume that the data arise from a random sample. Random sampling helps ensure that the observed discrepancies reflect the underlying population rather than selection bias.

4.2 Independence of observations

Many procedures require independent observations. Dependence between values can distort the test statistic and make the reference distribution inaccurate.

4.3 Sample size considerations

Sample size affects both reliability and sensitivity. Very small samples may not reveal meaningful departures, while very large samples can make even trivial differences appear statistically important.

4.4 Expected frequency requirements

For categorical tests such as the chi-squared procedure, expected counts should be sufficiently large in each category. When expected frequencies are too small, approximation accuracy can decline.

4.5 Distribution-specific requirements

Some tests require the theoretical distribution to be fully specified in advance, while others allow estimated parameters under certain conditions. It is important to follow the assumptions attached to each method.

5 Interpretation of results

5.1 Statistical significance

Statistical significance indicates that the observed difference from the expected distribution is unlikely to be due to chance alone under the null hypothesis. It does not by itself measure the importance of the deviation.

5.2 Practical significance

Practical significance considers whether the detected mismatch matters in a real-world context. A result can be statistically significant without being substantively important for decision-making or scientific interpretation.

5.3 Residual analysis

Residual analysis examines where the model fits poorly by comparing observed and expected values within categories or ranges. This can reveal patterns, such as particular categories contributing most to the overall lack of fit.

5.4 Effect of sample size on sensitivity

As sample size increases, tests become more capable of detecting small departures from the hypothesized model. This increased sensitivity can be useful, but it also means that minor and possibly unimportant differences may trigger rejection.

6 Limitations and pitfalls

6.1 Overfitting and model misspecification

If a model is too closely tailored to the sample, its apparent fit may be misleading. Conversely, if important features are omitted, the test may indicate poor fit even when the model seems reasonable at first glance.

6.2 Multiple testing issues

Running many goodness-of-fit tests raises the chance of finding at least one significant result by chance. When multiple comparisons are made, adjustments or cautious interpretation may be needed.

6.3 Small-sample limitations

With limited data, goodness-of-fit tests may have low power and unstable p-values. In such cases, graphical methods and subject-matter knowledge can be useful complements to formal testing.

6.4 Dependence and autocorrelation

Autocorrelated or otherwise dependent data violate common assumptions and can invalidate standard reference distributions. Time series and clustered observations often require modified methods.

6.5 Misuse of p-values

A p-value should not be treated as a direct measure of model quality. It only addresses compatibility with the null hypothesis, not the probability that the hypothesis is true or the magnitude of the discrepancy.

7.1 Tests for discrete distributions

Discrete goodness-of-fit methods are designed for count or categorical outcomes, such as binomial or multinomial data. They often rely on expected frequencies or discrete probability masses rather than continuous curves.

7.2 Tests for continuous distributions

Continuous distribution tests compare sample values with a hypothesized density or cumulative distribution. Common examples include methods based on empirical cumulative functions and tail-weighted discrepancies.

7.3 Comparing nested models

When one model is a special case of another, tests can compare their relative fit. These procedures are especially common in regression and likelihood-based modeling, where added parameters are evaluated against improved fit.

7.4 Likelihood-based alternatives

Likelihood ratio methods provide another approach to assessing model adequacy. They compare how well different models explain the observed data through their likelihood values.

7.5 Information criteria and model selection

Information criteria such as Akaike’s and Bayesian information criteria are not hypothesis tests, but they are closely related tools for judging model adequacy. They balance fit against complexity and are often used alongside goodness-of-fit procedures.