Regularization refers to a family of techniques in machine learning and statistics that aim to prevent overfitting by discouraging overly complex models. It works by adding a penalty to the loss function or by modifying the training procedure (e.g., early stopping, dropout), thereby improving the model's ability to generalize to unseen data. Regularization is widely applied in regression, neural networks, and other predictive models, and plays a fundamental role in controlling the bias–variance tradeoff.
1 Types of Regularization
1.1 Norm-Based Regularization
1.1.1 L1 Regularization (Lasso)
| L1 regularization, also known as Lasso (Least Absolute Shrinkage and Selection Operator), adds a penalty proportional to the absolute value of the model weights. The regularized loss becomes \( \mathcal{L}(\theta) + \lambda \|\theta\|_1 \). It encourages sparse solutions where many weights become exactly zero. |
|---|
1.1.1.1 Sparsity Property
The non‑differentiability of the L1 norm at zero leads to exact zero coefficients for sufficiently small weights. This property makes Lasso useful for feature selection, as irrelevant features are automatically removed from the model.
1.1.1.2 Geometric Interpretation
In the parameter space, the L1 penalty corresponds to a diamond‑shaped constraint region. The optimal solution occurs where the elliptical contours of the loss function first touch a corner of the diamond, often at axes, resulting in sparsity.
1.1.2 L2 Regularization (Ridge)
| L2 regularization, commonly called Ridge regression, adds a penalty proportional to the squared magnitude of the weights: \( \mathcal{L}(\theta) + \lambda \|\theta\|_2^2 \). It shrinks weights toward zero but does not force them to be exactly zero. |
|---|
1.1.2.1 Weight Decay Effect
The penalty term causes the weights to decay exponentially during gradient‑based optimization. In neural networks, this is known as weight decay and helps prevent large weights that lead to overfitting.
1.1.2.2 Rotational Invariance
The L2 penalty is isotropic—it does not favor any direction in parameter space. This rotational invariance means that Ridge regression treats all features equally and produces a closed‑form solution that remains well‑posed even under multicollinearity.
1.1.3 Elastic Net
| Elastic Net combines both L1 and L2 penalties: \( \lambda_1 \|\theta\|_1 + \lambda_2 \|\theta\|_2^2 \). It retains the sparsity property of Lasso while incorporating the stabilising benefits of Ridge. |
|---|
1.1.3.1 Combination of L1 and L2
By mixing the two norms, Elastic Net can select groups of correlated features jointly, whereas Lasso tends to pick only one from a correlated group. The mixing parameter, often denoted α, controls the balance between L1 and L2 regularization.
1.1.3.2 Grouped Variable Selection
When predictors are highly correlated, Elastic Net encourages the inclusion or exclusion of entire groups. This is especially useful in genomics and other high‑dimensional settings where correlated variables are common.
1.2 Data-Level Regularization
1.2.1 Data Augmentation
Data augmentation expands the training set by applying transformations to existing samples, thereby increasing effective data diversity and reducing overfitting.
1.2.1.1 Geometric Transformations
Common geometric augmentations include rotations, translations, flips, scaling, and cropping. These are widely used in image classification tasks where object pose and position should not affect the label.
1.2.1.2 Noise Injection
Adding small random noise to inputs (e.g., Gaussian noise in images or perturbations in tabular data) forces the model to become robust to minor variations, functioning as a form of regularization.
1.2.2 Label Smoothing
Label smoothing replaces hard targets (0 or 1) with softened versions, e.g., using a uniform distribution over classes mixed with the true label. This reduces overconfidence in predictions and improves generalization, especially in deep neural networks.
1.3 Model-Level Regularization
1.3.1 Dropout
Dropout randomly drops a fraction of neurons during training, forcing the network to learn redundant representations and reducing co‑adaptation.
1.3.1.1 Bernoulli Dropout
Each neuron is kept with probability \( p \) (and dropped with probability \( 1-p \)) independently per training step, following a Bernoulli distribution. During inference, all neurons are used but their outputs are scaled by \( p \).
1.3.1.2 Inverted Dropout
In inverted dropout, the scaling is applied during training (multiplying the kept neurons by \( 1/p \)) so that during inference no scaling is needed. This is the most common implementation in modern deep learning frameworks.
1.3.2 Early Stopping
Early stopping monitors the model’s performance on a validation set and halts training when performance stops improving, preventing overfitting.
1.3.2.1 Validation-Based Stopping Criterion
Training is often stopped when the validation loss has not decreased for a fixed number of epochs (the patience parameter). The model with the best validation performance is retained.
1.3.2.2 Relationship to L2 Regularization
Early stopping can be viewed as an efficient form of L2 regularization. In linear models, early stopping of gradient descent corresponds to limiting the norm of the weights, analogous to Ridge regression.
1.3.3 Batch Normalization
Batch normalization normalizes the activations of each layer to have zero mean and unit variance across a mini‑batch, then applies learnable scale and shift parameters.
1.3.3.1 Internal Covariate Shift Reduction
By stabilizing the distribution of layer inputs, batch normalization allows higher learning rates and reduces sensitivity to initialization. This mitigation of internal covariate shift accelerates training.
1.3.3.2 Regularizing Effect
Batch normalization introduces a slight noise because the mean and variance are estimated from the mini‑batch rather than the whole dataset. This stochasticity acts as a regularizer, often reducing the need for dropout.
2 Mathematical Formulation
2.1 Regularized Loss Function
2.1.1 General Form: L(θ) + λΩ(θ)
The regularized objective is a sum of the original loss function \( \mathcal{L}(\theta) \) and a penalty term \( \lambda \Omega(\theta) \), where \( \lambda \geq 0 \) controls the strength of regularization. The goal is to minimize this combined function over the parameters \( \theta \).
2.1.2 Common Penalty Functions
The most common penalty functions are:
| - L2: \( \Omega(\theta) = \|\theta\|_2^2 \) |
|---|
| - L1: \( \Omega(\theta) = \|\theta\|_1 \) |
| - Elastic Net: \( \Omega(\theta) = \alpha \|\theta\|_1 + (1-\alpha) \|\theta\|_2^2 \) |
Other penalties include the L0 norm (subset selection, NP‑hard) and various mixed norms for group regularization.
2.2 Hyperparameter Tuning
2.2.1 Cross-Validation for λ
The regularization strength λ is a hyperparameter typically selected via k‑fold cross‑validation. For each candidate λ, the model is trained on k‑1 folds and evaluated on the held‑out fold; the λ that minimizes average validation error is chosen.
2.2.2 Grid Search and Random Search
In practice, λ values are explored using grid search (e.g., logarithmically spaced values) or random search (sampling from a distribution). Random search is more efficient when only a few hyperparameters are tuned.
3 Theoretical Foundations
3.1 Bias–Variance Tradeoff
3.1.1 Decomposition of Expected Error
The expected test error of a model can be decomposed into the sum of bias², variance, and irreducible error. Regularization primarily increases bias (by shrinking parameters) but reduces variance, leading to lower total error when the reduction outweighs the bias increase.
3.1.2 Effect of Regularization on Variance
Penalizing large coefficients dampens the model’s sensitivity to random fluctuations in the training data, thereby lowering variance. The optimal regularization level balances the two components.
3.2 Statistical Properties
3.2.1 Consistency Under Regularization
Under appropriate conditions (e.g., increasing sample size, fixed number of parameters), regularized estimators such as Ridge and Lasso are consistent: they converge to the true parameter values. The rate of convergence may be slower than that of unregularized estimators but with better finite‑sample performance.
3.2.2 Oracle Property of Lasso
Under certain sparsity assumptions and conditions on the design matrix (e.g., the irrepresentable condition), Lasso achieves the oracle property: it identifies the correct support set (non‑zero coefficients) consistently and estimates the non‑zero parameters as efficiently as if the true model were known.
4 Applications in Applied Sciences
4.1 Regression Analysis
4.1.1 Ridge Regression in Multicollinearity
When predictor variables are highly correlated (multicollinearity), ordinary least squares estimates become unstable and have high variance. Ridge regression adds an L2 penalty that stabilises estimates by shrinking coefficients, making the solution unique and more interpretable.
4.1.2 Lasso for Feature Selection
Lasso’s sparsity property makes it a popular tool for simultaneous regression and variable selection. It is widely used in fields like genomics, economics, and signal processing to identify a small subset of relevant features among many.
4.2 Neural Networks
4.2.1 Dropout in Deep Learning
Dropout is a cornerstone regularization technique in deep neural networks. It is particularly effective in preventing overfitting when the number of parameters greatly exceeds the sample size, as often occurs in image and language processing tasks.
4.2.2 Weight Decay in Training
Weight decay (L2 regularization) is routinely added to the loss function when training neural networks with optimizers like SGD or Adam. It helps control the magnitude of weights and is often used together with dropout and batch normalization.
4.3 Support Vector Machines
4.3.1 L2 Regularization in Soft-Margin SVM
Support vector machines use an L2 penalty on the weight vector in the soft‑margin formulation. The parameter \( C \) (the inverse of λ) controls the trade‑off between maximizing the margin and minimizing training errors. The regularization term improves generalization.
4.4 Sparse Signal Processing
4.4.1 Compressed Sensing
Compressed sensing reconstructs a sparse signal from a small number of linear measurements by solving an L1‑regularized inverse problem. This approach, closely related to Lasso, is used in MRI, radar, and image compression to recover signals from incomplete data.