1 Introduction
1.1 Definition and scope
Regression algorithms are a class of supervised learning methods used to model the relationship between one or more independent variables (features) and a continuous dependent variable (target). They are fundamental to predictive analytics, time series forecasting, and scientific modeling across many domains. Common examples include linear regression, decision tree regression, and support vector regression, each with distinct assumptions and use cases. The goal of regression is to learn a function that minimizes the error between predicted and actual values, often quantified by loss functions such as mean squared error.
1.2 Historical context
The origins of regression trace back to the early 19th century with the work of Carl Friedrich Gauss and Adrien-Marie Legendre on the method of least squares. The term “regression” was coined by Francis Galton in the 1880s while studying the inheritance of traits, observing that extreme values tended to “regress” toward the mean. Throughout the 20th century, regression evolved from simple linear models into a broad family of algorithms incorporating regularization, non-linear transformations, and tree-based methods, driven by advances in statistics and computational power.
2 Core Algorithms
2.1 Linear Regression
Linear regression assumes a linear relationship between the independent variables and the target variable. It is the simplest and most interpretable regression model.
2.1.1 Simple Linear Regression
Simple linear regression models the relationship between a single feature \( x \) and the target \( y \) as \( y = \beta_0 + \beta_1 x + \epsilon \), where \( \beta_0 \) is the intercept, \( \beta_1 \) is the slope, and \( \epsilon \) is the error term. The parameters are estimated by minimizing the sum of squared residuals.
2.1.2 Multiple Linear Regression
Multiple linear regression extends the simple case to multiple features: \( y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p + \epsilon \). It allows modeling of complex relationships but may suffer from multicollinearity and overfitting when many features are present.
2.2 Regularized Regression
Regularized regression adds a penalty term to the loss function to reduce overfitting and improve generalization.
2.2.1 Ridge Regression (L2)
Ridge regression adds an L2 penalty proportional to the square of the magnitude of coefficients: \( L = \text{MSE} + \lambda \sum \beta_i^2 \). It shrinks coefficients toward zero but does not force any to be exactly zero, making it suitable when all features are relevant.
2.2.2 Lasso Regression (L1)
Lasso regression uses an L1 penalty (sum of absolute coefficient values): \( L = \text{MSE} + \lambda \sum |\beta_i| \). It can drive some coefficients to zero, performing automatic feature selection. This is beneficial when only a subset of features is believed to be important.
2.3 Polynomial Regression
Polynomial regression extends linear regression by adding polynomial terms of the original features (e.g., \( x^2, x^3 \)). It can model non-linear relationships while still using linear estimation techniques. However, high-degree polynomials are prone to overfitting, especially near the boundaries of the data.
2.4 Support Vector Regression
Support Vector Regression (SVR) applies the principles of Support Vector Machines to regression. It finds a hyperplane that fits the data within a specified margin (epsilon-tube), minimizing prediction error outside this tube. SVR can handle non-linearity through kernel functions (e.g., radial basis function), making it effective for complex datasets.
2.5 Tree-based Regression
Tree-based regression models partition the feature space into regions and assign a constant value (often the mean) to each region.
2.5.1 Decision Tree Regression
A decision tree recursively splits the data based on feature values to minimize variance within each leaf. The tree structure is interpretable but can overfit without pruning or depth constraints. It handles non-linear interactions naturally.
2.5.2 Random Forest Regression
Random forest builds many decision trees on bootstrap samples of the data and averages their predictions. It reduces variance compared to a single tree and is robust to overfitting. Random forest also provides feature importance scores.
2.6 Neural Network Regression
Neural network regression uses feedforward neural networks with one or more hidden layers to model complex, non-linear relationships. The output layer typically has a single neuron with a linear activation. These models are highly flexible but require large amounts of data and careful tuning of architecture and hyperparameters to avoid overfitting.
3 Loss Functions and Optimization
3.1 Common Loss Functions
Loss functions quantify the discrepancy between predicted and actual values, guiding the optimization process.
3.1.1 Mean Squared Error
Mean Squared Error (MSE) is defined as \( \frac{1}{n} \sum (y_i - \hat{y}_i)^2 \). It penalizes larger errors more heavily and is differentiable, making it convenient for gradient-based optimization. However, it is sensitive to outliers.
3.1.2 Mean Absolute Error
Mean Absolute Error (MAE) is \( \frac{1}{n} \sum |y_i - \hat{y}_i| \). It treats all errors equally and is more robust to outliers than MSE. Its non-differentiability at zero can be handled by subgradient methods.
3.1.3 Huber Loss
Huber loss combines MSE and MAE: it is quadratic for small errors and linear for large ones, using a threshold parameter \( \delta \). This provides robustness to outliers while retaining differentiability.
3.2 Optimization Techniques
Optimization techniques find the model parameters that minimize the loss function.
3.2.1 Ordinary Least Squares
Ordinary Least Squares (OLS) provides a closed-form solution for linear regression: \( \beta = (X^T X)^{-1} X^T y \). It is computationally efficient for small to moderate datasets but can be unstable when \( X^T X \) is ill-conditioned.
3.2.2 Gradient Descent
Gradient descent iteratively updates parameters in the opposite direction of the loss gradient. Variants include batch, stochastic (SGD), and mini-batch gradient descent. It scales to large datasets and is used for models without closed-form solutions, such as neural networks.
4 Model Evaluation
4.1 Performance Metrics
Performance metrics assess how well a regression model predicts unseen data.
4.1.1 R-squared
R-squared (coefficient of determination) measures the proportion of variance in the target explained by the model: \( R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}} \). It ranges from 0 to 1, with higher values indicating better fit. It can be misleading when adding irrelevant features.
4.1.2 Adjusted R-squared
Adjusted R-squared penalizes the addition of unnecessary features: \( R^2_{\text{adj}} = 1 - \frac{(1-R^2)(n-1)}{n-p-1} \). It increases only if a new feature improves the model more than expected by chance.
4.1.3 Root Mean Squared Error
Root Mean Squared Error (RMSE) is the square root of MSE, expressed in the same units as the target. It is widely used for interpretability and penalizes large errors.
4.1.4 Mean Absolute Percentage Error
Mean Absolute Percentage Error (MAPE) is \( \frac{100}{n} \sum |\frac{y_i - \hat{y}_i}{y_i}| \). It provides a scale-independent percentage error but is undefined when actual values are zero and can be biased toward low forecasts.
4.2 Validation Strategies
Validation strategies estimate model performance on unseen data without leaking information from the test set.
4.2.1 Train-Test Split
The dataset is split into a training set (e.g., 70–80%) and a test set (e.g., 20–30%). The model is trained on the training set and evaluated on the test set. This simple method can have high variance depending on the split.
4.2.2 Cross-Validation
Cross-validation (e.g., k-fold) partitions the data into k subsets, training on k-1 folds and testing on the held-out fold, repeating k times. The average performance over folds provides a more stable estimate. Common choices are 5-fold or 10-fold cross-validation.
5 Practical Considerations
5.1 Data Preprocessing
Data preprocessing improves model performance and stability.
5.1.1 Feature Scaling
Scaling features to a similar range (e.g., standardization to zero mean and unit variance, or min-max scaling to [0,1]) is essential for algorithms that rely on distances or gradients, such as SVR, neural networks, and gradient descent–based linear regression.
5.1.2 Handling Missing Values
Missing values can be handled by deletion (listwise or pairwise) or imputation (e.g., mean, median, or model-based imputation). The choice depends on the amount and pattern of missingness. Imputation can introduce bias but preserves sample size.
5.1.3 Encoding Categorical Variables
Categorical variables must be converted to numerical form. Common methods include one-hot encoding (creating binary columns for each category) and label encoding (assigning integers). One-hot encoding is preferred for models that assume linear relationships, while label encoding can imply ordinality.
5.2 Hyperparameter Tuning
Hyperparameters control model behavior (e.g., regularization strength \( \lambda \), tree depth, learning rate). Tuning is performed using grid search, random search, or Bayesian optimization over a validation set or cross-validation results. Proper tuning prevents underfitting or overfitting.
5.3 Overfitting and Underfitting
Overfitting occurs when a model learns noise in the training data, performing poorly on new data. Underfitting occurs when a model is too simple to capture underlying patterns.
5.3.1 Regularization Techniques
Regularization (L1, L2, elastic net) adds a penalty to the loss function to constrain model complexity. In tree-based models, pruning, limiting maximum depth, and minimum samples per leaf serve similar purposes.
5.3.2 Early Stopping
Early stopping monitors performance on a validation set during iterative training (e.g., gradient boosting, neural networks) and halts when validation error stops improving, preventing overfitting.
6 Advanced Topics
6.1 Bayesian Regression
Bayesian regression treats model parameters as random variables with prior distributions. After observing data, posterior distributions are computed, providing uncertainty estimates for predictions. Common variants include Bayesian linear regression and Gaussian process regression.
6.2 Quantile Regression
Quantile regression models conditional quantiles (e.g., median, 0.25 quantile) of the target rather than the mean. It is robust to outliers and provides a fuller picture of the target distribution, useful for risk assessment and heteroscedastic data.
6.3 Ensemble Methods
Ensemble methods combine multiple base models to improve predictive accuracy and robustness.
6.3.1 Gradient Boosting Regression
Gradient boosting builds an ensemble of weak learners (typically shallow trees) sequentially, each correcting the errors of the previous ensemble. The model minimizes a loss function using gradient descent in function space. Popular implementations include Gradient Boosting Machine (GBM).
6.3.2 XGBoost and LightGBM
XGBoost is an optimized gradient boosting library that includes regularization, parallel processing, and handling of missing values. LightGBM uses a histogram-based algorithm and leaf-wise tree growth for faster training on large datasets. Both are widely used in competitions and industry.
7 Applications
7.1 Economics and Finance
Regression is used for forecasting stock prices, predicting GDP growth, estimating demand elasticities, and modeling risk. Linear regression is common in econometrics, while ensemble methods are applied to high-frequency trading signals.
7.2 Engineering and Physical Sciences
In engineering, regression models predict material properties, optimize process parameters, and calibrate sensors. In physics, they fit experimental data to theoretical curves, such as in spectroscopy or particle physics.
7.3 Healthcare and Life Sciences
Regression predicts patient outcomes (e.g., blood pressure, disease progression), drug efficacy, and biomarker levels. Regularized and tree-based models handle high-dimensional genomic data.
7.4 Marketing and Sales Forecasting
Businesses use regression to forecast sales, estimate customer lifetime value, and assess the impact of advertising spend. Time series regression models account for seasonality and trends.
8 Future Directions
8.1 Automated Machine Learning for Regression
Automated Machine Learning (AutoML) aims to automate model selection, hyperparameter tuning, and feature engineering for regression tasks. Tools such as Auto-sklearn and H2O AutoML are becoming mainstream, reducing the need for manual intervention.
8.2 Deep Learning-based Regression Models
Deep learning continues to expand into regression domains, especially for complex data types (images, text, time series). Architectures like convolutional neural networks (CNNs) for image-based regression and recurrent neural networks (RNNs) for sequence prediction are increasingly used. Interpretability and data efficiency remain active research areas.