1 Foundations of regression
Regression analysis is a family of statistical techniques used to describe and estimate relationships between a response variable and one or more explanatory variables. Its main value lies in summarizing patterns in data in a form that can be interpreted, compared, and often used for prediction. In applied research, regression also provides a framework for testing whether observed associations are statistically distinguishable from random variation.
1.1 Core idea and purpose
The core idea of regression is to model how the expected value of one variable changes when another variable changes. In the simplest case, this relationship is represented as a line, but in broader use it may involve curved, categorical, or multivariable patterns. Regression is used to explain variation, estimate effects, and forecast outcomes under specified conditions.
1.2 Variables in a regression model
A regression model typically distinguishes between the outcome being studied and the factors thought to influence it. The outcome is modeled as a function of explanatory terms, each of which contributes information about the pattern in the data. Additional variables may be included to account for background differences or reduce bias in estimates.
1.2.1 Dependent variable
The dependent variable, also called the response or outcome variable, is the quantity the model seeks to explain or predict. It may be continuous, binary, ordinal, or count-based, depending on the type of regression used. The choice of dependent variable strongly shapes the model form and estimation method.
1.2.2 Independent variables
Independent variables, also called predictors or explanatory variables, are the inputs used to explain variation in the dependent variable. They may be numerical or categorical and can represent measured characteristics, exposures, or conditions. Their coefficients describe how the model associates each predictor with changes in the outcome, holding other terms constant.
1.2.3 Control variables
Control variables are included to account for other influences that might otherwise distort the estimated relationship of interest. They are not always central to the research question, but they can improve the precision and interpretability of the model. In social research, controls are often used to adjust for demographic, contextual, or temporal differences.
1.3 Regression coefficients
Regression coefficients quantify the direction and magnitude of association between predictors and the outcome. In linear models, a coefficient indicates the expected change in the dependent variable associated with a one-unit change in a predictor, assuming other variables are unchanged. In other regression forms, coefficients may be expressed on transformed scales, such as log-odds or log counts.
1.4 Error term and residuals
The error term represents the part of the outcome not explained by the model, including random fluctuation and unmeasured factors. Residuals are the observed differences between actual values and fitted values after estimation. Examining residuals helps assess model adequacy, identify unusual observations, and diagnose departures from assumptions.
2 Types of regression models
Regression models differ according to the scale of the outcome, the shape of the relationship, and the assumptions made about the underlying data. Some are designed for continuous outcomes, while others are better suited to categorical or count data. Additional forms are used when analysts want to stabilize estimates or reduce the influence of many correlated predictors.
2.1 Linear regression
Linear regression models the mean of a continuous outcome as a linear combination of predictors. It is among the most widely used regression methods because of its simplicity, interpretability, and flexibility in empirical work. Despite the name, linear regression can include transformed predictors and interaction terms.
2.1.1 Simple linear regression
Simple linear regression uses one predictor to explain variation in one outcome. The model estimates a straight-line relationship, making it useful for basic trend analysis and introductory data exploration. It is often the starting point for understanding more complex regression structures.
2.1.2 Multiple linear regression
Multiple linear regression includes two or more predictors in the same model. This allows the analyst to estimate the association of each variable with the outcome while holding the others constant. It is especially useful when outcomes are shaped by several factors that may overlap in their influence.
2.2 Nonlinear regression
Nonlinear regression is used when the relationship between predictors and the outcome cannot be adequately described by a straight line. The functional form may involve exponential growth, saturation, curves, or other patterns determined by theory or data. These models are common in fields where change occurs at different rates across the range of a predictor.
2.3 Logistic regression
Logistic regression is designed for categorical outcomes, especially when the response is modeled through probabilities. Rather than predicting the raw outcome directly, it estimates the likelihood that an event occurs. The model uses a link function that keeps predicted values within a valid probability range.
2.3.1 Binary logistic regression
Binary logistic regression is used when the outcome has two categories, such as yes or no, success or failure. It models the log-odds of the event and is widely used in classification and decision research. Coefficients are commonly interpreted through odds ratios.
2.3.2 Multinomial logistic regression
Multinomial logistic regression handles outcomes with more than two unordered categories. It compares each category with a reference category to estimate how predictors affect category membership. This approach is useful when the possible outcomes are distinct but not naturally ranked.
2.3.3 Ordinal logistic regression
Ordinal logistic regression applies when outcome categories have a meaningful order, such as low, medium, and high. It incorporates the ranking information while avoiding the assumption that category distances are equal. The method is frequently used for survey responses and rating scales.
2.4 Count regression models
Count regression models are used when the dependent variable represents nonnegative integers, such as the number of visits, events, or items. These models account for the fact that count data often have distributions unlike those assumed by ordinary linear regression. They are especially helpful when the data show skewness or variance patterns tied to the mean.
2.4.1 Poisson regression
Poisson regression models count outcomes under the assumption that the mean and variance are closely related. It is often used for rare events or exposure-based counts. When the data fit its assumptions reasonably well, it provides a compact way to study event rates.
2.4.2 Negative binomial regression
Negative binomial regression is a count model that allows greater variability than the Poisson model. It is especially useful when the observed data are overdispersed, meaning the variance exceeds the mean by a substantial margin. This added flexibility often improves fit in applied settings.
2.5 Regularized regression
Regularized regression methods add a penalty to the fitting process to discourage overly complex models. They are useful when many predictors are available or when predictors are strongly correlated. By shrinking some coefficients, these methods can improve predictive performance and reduce instability.
2.5.1 Ridge regression
Ridge regression applies a penalty that shrinks coefficients toward zero without setting them exactly to zero. It is often used when many predictors contribute modestly and collinearity is a concern. The method helps produce more stable estimates in high-dimensional settings.
2.5.2 Lasso regression
Lasso regression uses a penalty that can shrink some coefficients to exactly zero. This makes it useful for variable selection as well as prediction. It is often chosen when researchers want a more compact model with fewer active predictors.
2.5.3 Elastic net regression
Elastic net regression combines features of ridge and lasso methods. It can shrink coefficients while also selecting variables, which is especially helpful when predictors are numerous and correlated. The approach is often used in modern predictive modeling.
3 Model specification
Model specification refers to how the analyst translates a substantive question into a statistical formula. Choices about included variables, transformations, and structural terms affect both interpretability and validity. A well-specified model balances theoretical relevance, data quality, and parsimony.
3.1 Selecting variables
Selecting variables involves deciding which predictors belong in the model and which do not. The choice may be guided by theory, prior research, measurement quality, and the intended purpose of the analysis. Good selection aims to capture the important structure in the data without adding unnecessary complexity.
3.2 Functional form
Functional form describes the mathematical shape used to represent relationships among variables. A linear form is often convenient, but some phenomena are better represented by logarithmic, exponential, or other transformations. Choosing an appropriate form can improve fit and make the model more faithful to the observed pattern.
3.3 Interaction effects
Interaction effects occur when the effect of one predictor depends on the level of another predictor. They allow models to represent conditional relationships rather than assuming a single uniform effect. Interactions are common in social science research where contexts and group differences matter.
3.4 Polynomial terms
Polynomial terms add powers of a predictor, such as a squared or cubic term, to capture curvature. They are useful when the outcome increases at one rate at lower values and another rate at higher values. Such terms provide a simple way to represent bends in an otherwise smooth relationship.
3.5 Fixed effects and random effects
Fixed effects and random effects are strategies for handling grouped or repeated data. Fixed effects control for unobserved differences across units that do not change over time or across observations, while random effects treat those differences as drawn from a broader population. The choice depends on the structure of the data and the inferential goal.
4 Estimation and fitting
Estimation is the process of finding coefficient values that best align the model with the observed data. Fitting procedures differ according to the model type and the underlying statistical assumptions. The quality of the fit influences both the credibility of the estimates and the usefulness of the model for prediction.
4.1 Ordinary least squares
Ordinary least squares is the standard estimation method for linear regression. It chooses coefficient values that minimize the sum of squared differences between observed and predicted outcomes. Because of its mathematical convenience, it remains a foundational technique in statistical analysis.
4.2 Maximum likelihood estimation
Maximum likelihood estimation selects parameter values that make the observed data most probable under the model. It is widely used in logistic, Poisson, and many other regression frameworks. The method offers a general and flexible approach to fitting models with nonnormal outcomes.
4.3 Bayesian estimation
Bayesian estimation combines prior information with observed data to produce a posterior distribution for the coefficients. Rather than yielding a single point estimate alone, it characterizes uncertainty in a probabilistic way. This approach is useful when prior knowledge is available or when complex models require richer uncertainty summaries.
4.4 Model convergence
Model convergence refers to whether the fitting algorithm successfully reaches a stable solution. Some models, especially those with many predictors or sparse data, can be difficult to estimate numerically. Convergence problems may signal weak data support, poor scaling, or an overly complicated specification.
4.5 Software implementation
Regression models are implemented through statistical software and programming environments that automate estimation, diagnostics, and visualization. These tools vary in syntax, defaults, and available options, so results can depend on how the analysis is configured. Careful implementation includes checking settings, reference categories, and output conventions.
5 Assumptions and diagnostics
Regression methods rely on assumptions that help justify estimation and inference. Diagnostics are used to assess whether those assumptions are approximately satisfied and to identify potential problems. No model is perfect, but understanding departures from assumptions helps interpret results responsibly.
5.1 Linearity
Linearity means that the expected outcome changes in a straight-line fashion with respect to a predictor, at least within the modeled scale. When this assumption is violated, coefficient estimates may misrepresent the true relationship. Plots and residual checks are often used to assess whether a linear approximation is adequate.
5.2 Independence of errors
Independence of errors means that one observation’s residual should not systematically depend on another’s. This assumption can fail when data are clustered, repeated over time, or otherwise linked. Dependence can lead to underestimated uncertainty and overly confident conclusions.
5.3 Homoscedasticity
Homoscedasticity means that the variance of the residuals is roughly constant across levels of the predictors or fitted values. When variance changes markedly, standard errors and tests may become unreliable. Heteroscedasticity is common in real data and often requires robust methods or model revision.
5.4 Normality of residuals
Normality of residuals is an assumption often associated with linear regression inference, especially in smaller samples. It concerns the distribution of the unexplained part of the outcome rather than the predictors themselves. Mild departures are often tolerated, but strong nonnormality can affect confidence intervals and tests.
5.5 Multicollinearity
Multicollinearity arises when predictors are highly correlated with one another. This can make it difficult to isolate the unique contribution of each variable and may inflate standard errors. The model may still predict well, but individual coefficients can become unstable and hard to interpret.
5.6 Outliers and influential observations
Outliers are observations that differ markedly from the rest of the data, while influential observations are points that have a strong effect on the fitted model. Both deserve attention because they can distort slopes, change significance levels, or reveal data issues. Diagnostic tools help determine whether such points reflect error, rarity, or meaningful structure.
5.6.1 Leverage
Leverage measures how unusual an observation is in terms of its predictor values. High-leverage points lie far from the center of the explanatory-variable space and can exert substantial influence on the fitted line or surface. Not all high-leverage cases are problematic, but they merit careful inspection.
5.6.2 Cook's distance
Cook's distance is a diagnostic that summarizes how much a regression fit would change if a particular observation were removed. Large values suggest that the point has notable influence on the model. It is commonly used alongside residual and leverage checks to identify sensitive cases.
6 Model evaluation
Model evaluation assesses how well a regression model describes the observed data and how reliably it may perform on new data. Different criteria are useful for different purposes, including explanation, inference, and prediction. Good evaluation usually combines several measures rather than relying on one statistic alone.
6.1 Goodness of fit
Goodness of fit refers to how closely the model reproduces the observed data. A strong fit indicates that the model captures much of the variation in the outcome, though it does not by itself prove that the specification is substantively correct. Fit statistics should be interpreted in light of model type and research goals.
6.2 R-squared and adjusted R-squared
R-squared indicates the proportion of variance in the dependent variable explained by the model. Adjusted R-squared modifies this measure to account for the number of predictors, helping compare models of different sizes. Both are most naturally interpreted in linear regression and less directly in other regression families.
6.3 Hypothesis testing
Hypothesis testing evaluates whether estimated effects differ meaningfully from zero or from a specified reference value. In regression, this is commonly used to assess whether predictors contribute explanatory power beyond random noise. The results depend on the model, sample size, and assumption set.
6.3.1 t-tests for coefficients
t-tests for coefficients examine whether an individual regression coefficient is statistically different from zero. They are commonly reported in linear models and serve as a basic tool for evaluating predictor importance. A significant test suggests that the associated effect is unlikely to be due solely to sampling variation.
6.3.2 F-tests and likelihood ratio tests
F-tests compare nested models or assess whether groups of coefficients jointly improve fit in linear regression. Likelihood ratio tests serve a similar purpose in likelihood-based models by comparing the relative fit of two nested specifications. These tests are useful when interest lies in a set of variables rather than a single coefficient.
6.4 Prediction error
Prediction error measures how far model predictions are from observed values. Smaller errors indicate better predictive performance, especially when evaluated on data not used for fitting. Common metrics include mean squared error, absolute error, and classification accuracy, depending on the type of outcome.
6.5 Cross-validation
Cross-validation divides data into training and validation subsets to estimate how well a model generalizes. It reduces the risk of overestimating performance based only on the sample used to fit the model. This technique is especially valuable when comparing competing specifications or tuning regularized models.
7 Interpretation of results
Interpreting regression results requires attention to the model scale, the outcome type, and the included terms. Coefficients, intervals, and predicted values each provide a different view of the same fitted relationship. Clear interpretation depends on matching the statistical output to the substantive question.
7.1 Coefficient interpretation
Coefficient interpretation translates model estimates into substantive meaning. In linear regression, a coefficient is usually read as the expected change in the outcome for a one-unit change in the predictor. In other regression models, the meaning may be less direct and often requires transformation or auxiliary measures.
7.2 Odds ratios and marginal effects
Odds ratios summarize effects in logistic regression by comparing the odds of an outcome across different predictor values. Marginal effects describe how predicted probabilities or expected values change as predictors change. These quantities are often easier to communicate than raw coefficients on the log scale.
7.3 Standardized coefficients
Standardized coefficients express effects in units of standard deviations rather than original measurement units. They help compare the relative size of predictors measured on different scales. However, they do not remove all interpretive issues and may be less intuitive than coefficients in original units.
7.4 Confidence intervals
Confidence intervals provide a range of plausible values for a parameter estimate. They indicate both the estimated effect and the precision of that estimate. Wider intervals suggest greater uncertainty, while narrower intervals indicate more precise estimation.
7.5 Predicted values
Predicted values are the model’s fitted outcomes for specific combinations of predictors. They are useful for presenting results in concrete terms and for comparing scenarios. In applied work, predicted values often communicate model implications more clearly than coefficients alone.
8 Applications in the social sciences
Regression is one of the most common tools in social science because it helps connect theory with observed data. Researchers use it to describe relationships, compare groups, examine trends, and evaluate hypotheses. Its flexibility allows it to be adapted to many kinds of social data.
8.1 Economics
In economics, regression is used to study wages, consumption, employment, inflation, and many other measurable outcomes. It helps estimate the association between economic variables while accounting for other factors that shape behavior. Economists also use regression extensively in forecasting and policy evaluation.
8.2 Sociology
Sociology uses regression to examine patterns in family structure, inequality, mobility, education, and social behavior. The method is valuable for analyzing how individual characteristics and social contexts jointly relate to outcomes. It is often paired with grouped or longitudinal data to reflect social structure.
8.3 Psychology
Psychology employs regression to relate behavior, cognition, and emotion to measured traits or conditions. It is used in studies of personality, mental health, performance, and experimental outcomes. Regression can help identify which variables are most strongly associated with a psychological response.
8.4 Political science
Political science applies regression to voting behavior, public opinion, institutions, and policy outcomes. The method supports analysis of survey data, electoral trends, and comparative political systems. It is also widely used to model binary and ordinal outcomes common in political research.
8.5 Education research
Education research uses regression to investigate student achievement, attendance, progression, and school-level influences. It is helpful for evaluating how instructional, demographic, and institutional factors relate to learning outcomes. Researchers often use multilevel or fixed-effects approaches when students are nested within schools or classes.
9 Limitations and common pitfalls
Regression is powerful, but its results depend on careful design, appropriate assumptions, and restrained interpretation. Missteps in specification or interpretation can produce misleading conclusions even when the calculations are correct. Awareness of common pitfalls improves both analytical rigor and communication.
9.1 Omitted variable bias
Omitted variable bias occurs when an important factor is left out of the model and is related to both the predictor and the outcome. This can distort coefficient estimates and create misleading associations. Including relevant controls can reduce, though not always eliminate, the problem.
9.2 Endogeneity
Endogeneity arises when a predictor is correlated with the error term, often because of reverse causation, measurement error, or unobserved confounding. It undermines simple interpretations of coefficients as isolated effects. Addressing it may require specialized methods or stronger research designs.
9.3 Overfitting
Overfitting happens when a model is too closely tailored to the sample data and captures noise rather than general structure. Such a model may perform well in-sample but poorly on new data. Cross-validation, regularization, and simpler specifications can help reduce this risk.
9.4 Causal inference limitations
Regression can support causal claims only under additional assumptions and research design features. Observational associations alone do not establish cause and effect. Without careful attention to confounding, timing, and identification, regression results should be treated as descriptive or associative.
9.5 Misinterpretation of correlation and causation
A common error is to assume that a statistically significant association implies causation. Regression can reveal patterns, but those patterns may reflect shared causes, selection effects, or coincidence. Clear interpretation requires distinguishing predictive association from causal explanation.