1 Foundations of generalized linear models

Generalized linear models form a broad class of regression methods designed to relate an outcome to one or more predictors. They preserve the familiar structure of linear modeling while allowing the response to be non-normal and the connection between predictors and the outcome to be nonlinear on the response scale.

1.1 Historical development

GLMs emerged from efforts to unify several regression methods under a single theoretical framework. Earlier work in linear regression and categorical data analysis showed that many models could be expressed through a linear predictor combined with a transformation that matched the response distribution. This synthesis became especially influential in modern statistical modeling because it offered a common language for binary, count, and continuous positive outcomes.

1.2 Motivation for extending linear regression

Ordinary linear regression is most suitable when errors are approximately normal, variances are roughly constant, and predictions may reasonably extend across the real line. Many practical data sets violate these conditions. Counts cannot be negative, binary responses are limited to two values, and some measurements are skewed or have variances that change with the mean. GLMs address these cases by adapting the distributional assumption and the scale on which predictors act.

1.3 Basic components of a GLM

A generalized linear model is built from three linked elements: a probability model for the response, a linear predictor for the explanatory variables, and a link function that connects the two. Together, these components provide a flexible yet structured approach to regression.

1.3.1 Random component

The random component specifies the probability distribution of the response variable. In a GLM, this distribution is chosen from the exponential family, which includes the normal, binomial, Poisson, and gamma distributions. The selected distribution reflects the nature of the data being modeled.

1.3.2 Systematic component

The systematic component is the linear predictor, usually written as a combination of coefficients and covariates. It represents the deterministic part of the model and summarizes how predictors contribute additively on a transformed scale.

The link function connects the expected value of the response to the linear predictor. It determines how changes in predictors affect the mean of the response. Different links are used depending on the response type and modeling goals.

1.4 Relationship to ordinary least squares

Ordinary least squares can be viewed as a special case of a generalized linear model with a normal response distribution and an identity link. In that setting, the mean of the response equals the linear predictor, and estimation reduces to minimizing squared error. GLMs therefore extend rather than replace standard linear regression.

2 Mathematical formulation

The mathematical structure of a GLM is organized around a probability distribution, a mean function, and a linear predictor. This formulation makes it possible to derive estimation procedures and interpret coefficients systematically.

2.1 Exponential family distributions

The exponential family is a large class of distributions that share a convenient algebraic form. Many common statistical models belong to this family, which is one reason GLMs are so widely applicable.

2.1.1 Canonical form

In canonical form, the distribution is written so that the response enters through a sufficient statistic and the parameter appears in the exponent. This representation simplifies derivations and often leads to efficient estimation.

2.1.2 Dispersion parameter

Some exponential family models include a dispersion parameter that controls the spread of the response around its mean. In certain cases, such as the Poisson and binomial distributions, this parameter is fixed by the model; in others, such as the normal or gamma distributions, it may be estimated from the data.

2.2 Linear predictor

The linear predictor is a weighted sum of explanatory variables, typically including an intercept term. It can also incorporate transformed variables, indicator variables, and interaction terms. The predictor captures the combined influence of covariates before the link function is applied.

2.3 Mean-variance relationship

A defining feature of many GLMs is that the variance depends on the mean. This relationship differs from ordinary linear regression, where variance is often assumed constant. For example, Poisson models have variance that increases with the mean, while binomial variance is largest at intermediate probabilities.

A canonical link is the link function naturally associated with a given exponential family distribution. It often leads to simpler formulas and desirable mathematical properties.

The identity link leaves the mean unchanged, so the linear predictor equals the expected response. It is standard in normal linear regression and useful when the response can vary over the real line.

The logit link maps probabilities to the real line through the log odds. It is the standard link for binary outcomes and for many models of proportions.

The log link relates the logarithm of the mean to the linear predictor. It is commonly used for counts and positive responses because it ensures predicted means remain positive.

3 Common model types

Several familiar regression methods are special cases of the generalized linear model framework. These models differ mainly in their response distribution and link function.

3.1 Linear regression as a special case

Linear regression fits naturally within the GLM framework when the response is normally distributed with constant variance and the identity link is used. It remains the benchmark model for continuous outcomes with approximately symmetric errors.

3.2 Logistic regression

Logistic regression models binary outcomes by expressing the log odds of success as a linear function of predictors. It is widely used for classification, risk estimation, and studies where the response represents the presence or absence of an event.

3.3 Poisson regression

Poisson regression is suited to count data, especially when counts represent the number of events occurring in a fixed interval or area. The model uses a log link and is often applied when the response takes nonnegative integer values.

3.4 Binomial regression

Binomial regression is used for aggregated binary data or proportion outcomes based on a known number of trials. It generalizes logistic regression to settings where the response summarizes multiple Bernoulli observations.

3.5 Gamma regression

Gamma regression is appropriate for positive, right-skewed continuous variables, such as waiting times or costs. It typically uses a log link and accommodates variance that grows with the mean.

3.6 Negative binomial regression

Negative binomial regression is often used for overdispersed count data, where the variability exceeds what a Poisson model can capture. It introduces an additional dispersion parameter that allows extra heterogeneity in the data.

4 Estimation and inference

Fitting a GLM involves estimating coefficients and assessing their uncertainty. The methods used are closely related to likelihood theory and asymptotic approximation.

4.1 Maximum likelihood estimation

Maximum likelihood estimation chooses parameter values that make the observed data most probable under the model. In GLMs, the likelihood is constructed from the chosen response distribution, and the resulting estimates are commonly obtained by numerical methods.

4.2 Iteratively reweighted least squares

Iteratively reweighted least squares is a standard algorithm for fitting many GLMs. It updates coefficient estimates by solving a sequence of weighted least-squares problems, with weights that depend on the current fitted values.

4.3 Deviance

Deviance measures the discrepancy between the fitted model and a saturated model that fits the data perfectly. Smaller deviance indicates closer fit, and differences in deviance are often used for comparing nested models.

4.4 Likelihood ratio tests

Likelihood ratio tests compare two nested models by examining how much the likelihood improves when additional parameters are included. The test statistic is commonly based on the difference in deviance and is used to evaluate whether the extra complexity is justified.

4.5 Wald tests

Wald tests assess whether individual coefficients or sets of coefficients differ significantly from specified values, often zero. They rely on estimated standard errors and are widely used because of their simplicity and direct connection to coefficient estimates.

4.6 Confidence intervals

Confidence intervals provide a range of plausible values for model parameters or transformed quantities such as odds ratios. They reflect estimation uncertainty and are usually computed from standard errors or likelihood-based methods.

5 Model assessment

Assessing a GLM involves checking whether the model describes the data adequately and whether its predictions are useful. Diagnostic tools help identify lack of fit, unusual observations, and violations of assumptions.

5.1 Residual analysis

Residuals compare observed outcomes with model-based fitted values. Examining their patterns can reveal systematic departures from the assumed model structure.

5.1.1 Pearson residuals

Pearson residuals scale raw residuals by the estimated standard deviation implied by the model. They are useful for spotting observations that deviate more than expected under the fitted distribution.

5.1.2 Deviance residuals

Deviance residuals express each observation’s contribution to the overall deviance. They are often preferred for diagnostic plots because they are linked directly to model fit.

5.2 Goodness-of-fit measures

Goodness-of-fit measures summarize how well the model matches the observed data. Common summaries include deviance, residual statistics, and measures based on predictive accuracy, though no single index captures every aspect of fit.

5.3 Overdispersion

Overdispersion occurs when the observed variability is greater than the model expects. It is especially common in count and binary data and may indicate omitted structure, unmodeled heterogeneity, or an inappropriate distributional choice.

5.4 Model comparison

Model comparison evaluates whether one specification is preferable to another. Analysts may compare nested models using likelihood-based tests or assess non-nested alternatives through information criteria and predictive performance.

5.5 Predictive performance

Predictive performance reflects how accurately the model forecasts new observations. It can be examined through cross-validation, prediction error, calibration, and classification measures, depending on the response type.

6 Interpretation

Interpreting a GLM requires attention to the link function and the scale on which coefficients operate. The raw coefficient values are often most meaningful after transformation.

6.1 Coefficients and effect sizes

Coefficients represent changes in the linear predictor associated with a one-unit change in a covariate, holding other variables fixed. Their practical meaning depends on the link function and may be more intuitive after exponentiation or other transformation.

6.2 Odds ratios

Odds ratios are commonly used in logistic regression. They describe the multiplicative change in odds associated with a one-unit increase in a predictor or with a comparison between categories.

6.3 Rate ratios

Rate ratios are frequent in Poisson and related count models. They express how the expected count changes multiplicatively across predictor values, making them useful for event-rate interpretation.

6.4 Marginal effects

Marginal effects quantify the change in the expected response associated with a change in a predictor on the response scale. They can be especially helpful when nonlinear links make raw coefficients difficult to interpret directly.

6.5 Predicted values and fitted means

Predicted values are the model’s estimated responses for given covariate patterns. Fitted means are central to GLM interpretation because the model is structured around the conditional expectation of the response.

7 Practical applications

GLMs are used across many disciplines because they adapt well to different data types and research questions. Their versatility makes them a standard tool in applied statistical analysis.

7.1 Biology and medicine

In biology and medicine, GLMs are used for disease risk modeling, laboratory assay analysis, survival-related covariates, and experiments involving counts or proportions. Logistic and Poisson models are especially common in these fields.

7.2 Social and behavioral sciences

Researchers in social and behavioral sciences use GLMs to study survey responses, binary choices, event counts, and categorical outcomes. The framework supports both explanation and prediction in observational and experimental data.

7.3 Economics and finance

In economics and finance, GLMs are applied to default modeling, claim counts, transaction frequencies, and other non-normal outcomes. They are also useful when interest centers on rates, incidences, or positive skewed variables.

7.4 Engineering and quality control

Engineering applications include defect counts, failure rates, process monitoring, and reliability analysis. GLMs help model outcomes that are discrete, rare, or constrained to positive values.

7.5 Ecology and environmental science

Ecology and environmental science often involve abundance counts, presence-absence data, and measurements that vary with exposure or habitat conditions. GLMs provide a natural framework for analyzing these response types.

Several important methods build on the generalized linear model framework while addressing correlated data, nonlinearity, or more complex outcome structures. These extensions broaden the scope of GLM-based analysis.

8.1 Generalized estimating equations

Generalized estimating equations extend GLMs to correlated observations, such as repeated measurements within the same subject or cluster. They focus on estimating population-averaged effects while accounting for within-group dependence.

8.2 Mixed-effects generalized linear models

Mixed-effects generalized linear models include both fixed effects and random effects. They are useful when data are grouped or hierarchical, allowing variation across clusters, subjects, or other units.

8.3 Quasi-likelihood methods

Quasi-likelihood methods relax strict assumptions about the full response distribution while retaining a mean-variance relationship. They are valuable when the exact distribution is uncertain but the mean structure is still of interest.

8.4 Generalized additive models

Generalized additive models extend GLMs by replacing some linear predictor terms with smooth functions of the covariates. This allows flexible nonlinear relationships while preserving the GLM framework for the response distribution and link.

8.5 Multinomial and ordinal extensions

Multinomial and ordinal models generalize GLMs to outcomes with more than two categories. Multinomial models handle unordered categories, while ordinal models are used when categories have a natural order.

9 Assumptions and limitations

Like all statistical models, GLMs rely on assumptions that should be examined in context. Violations do not always invalidate analysis, but they can affect estimation, interpretation, and prediction.

9.1 Independence of observations

Standard GLMs assume that observations are independent given the predictors. When data are clustered, repeated, or otherwise correlated, specialized methods are needed to avoid misleading uncertainty estimates.

The link function should be appropriate for the outcome and the scientific question. A poorly chosen link may distort the relationship between predictors and the response, reducing interpretability or fit.

9.3 Distributional assumptions

The assumed response distribution should match the data-generating process reasonably well. If the distribution is badly misspecified, standard errors, tests, and predictions may become unreliable.

9.4 Multicollinearity

Multicollinearity arises when predictors are highly correlated with one another. It can inflate standard errors and make coefficient estimates unstable, even when the model fits the data reasonably well overall.

9.5 Influential observations

Influential observations can disproportionately affect parameter estimates and fitted values. Identifying them is important because a small number of unusual cases may alter conclusions or signal data quality issues.

10 Software and implementation

GLMs are available in most statistical software environments and are typically straightforward to fit. Practical implementation often involves specifying the response distribution, selecting a link, checking diagnostics, and reporting results clearly.

10.1 Common statistical software

Many software systems support GLMs through built-in procedures or packages. These tools usually provide estimation, hypothesis testing, diagnostic plots, and prediction functions.

10.2 Model fitting workflows

A typical workflow begins with exploratory analysis, followed by model specification, estimation, diagnostic checking, and refinement. Analysts may compare alternative links or distributions if the initial model does not fit well.

10.3 Diagnostics in practice

Practical diagnostics combine residual plots, leverage measures, and influence checks with substantive knowledge of the application. Because no single diagnostic is sufficient, multiple views of the fitted model are often needed.

10.4 Reproducibility and reporting

Reproducible GLM analysis requires clear documentation of the response definition, predictors, link function, estimation method, and diagnostic results. Reports are most useful when they include effect sizes, uncertainty measures, and an explanation of the model’s scale.

</INTERNAL_LINK_CANDIDATES> Ordinary least squares (method for estimating linear regression coefficients) Exponential family (class of probability distributions used in GLMs) Link function (transformation connecting mean response and linear predictor) Linear predictor (weighted sum of covariates in the model) Maximum likelihood estimation (method for fitting model parameters) Iteratively reweighted least squares (algorithm for GLM fitting) Deviance (measure of discrepancy between fitted and saturated models) Likelihood ratio test (test comparing nested statistical models) Wald test (test based on estimated coefficients and standard errors) Residual analysis (diagnostic examination of model errors) Pearson residual (residual standardized by model-based variance) Deviance residual (residual based on contribution to deviance) Overdispersion (variance exceeding that expected under the model) Odds ratio (multiplicative effect on odds in logistic models) Rate ratio (multiplicative effect on event rates or counts) Marginal effect (change in expected response for a predictor change) Generalized estimating equations (method for correlated response data) Mixed-effects model (model with fixed and random effects) Quasi-likelihood (estimation approach without full distributional specification) Generalized additive model (model with smooth predictor effects)