Bagging (short for Bootstrap Aggregating) is an ensemble machine learning technique designed to improve the stability and accuracy of base learning algorithms. It works by generating multiple bootstrap samples (random subsets with replacement) from the original training dataset, training a separate model on each sample, and then combining their predictions through averaging (for regression) or majority voting (for classification). Bagging is particularly effective for high-variance models such as decision trees, as it reduces overfitting and enhances generalization.
1 Historical Background
Bagging was introduced by Leo Breiman in 1994 in his paper "Bagging Predictors." The method emerged from earlier work on bootstrap techniques, which had been developed by Bradley Efron in the 1970s for statistical inference. Breiman's key insight was that aggregating multiple models trained on perturbed versions of the data could dramatically reduce the variance of unstable learners, thereby improving predictive performance. The technique quickly gained traction in the machine learning community and laid the foundation for later ensemble methods such as Random Forests.
2 Algorithm
The bagging algorithm proceeds in three main steps: bootstrap sampling, base model training, and aggregation. The central idea is to introduce diversity among the base models by exposing each to a slightly different training set, then combine their outputs to produce a more robust final prediction.
2.1 Bootstrap Sampling
Bootstrap sampling involves drawing random samples from the original dataset with replacement. Each bootstrap sample has the same size as the original dataset (or a specified fraction thereof), and because sampling is with replacement, some data points appear multiple times while others are omitted.
2.1.1 Sample Size and Replacement
By default, each bootstrap sample contains *n* observations, where *n* is the size of the original training set. Sampling with replacement ensures that approximately 63.2% of the original instances are included in a given sample (the rest being duplicates), while about 36.8% are left out. These omitted instances are called out-of-bag (OOB) samples.
2.1.2 Number of Bootstrap Replicates
The number of bootstrap replicates (i.e., the number of base models) is typically denoted by *B*. Common choices range from 50 to 1000, with larger values providing diminishing returns in error reduction. The optimal *B* depends on the dataset size and the base learner's variance; a typical default is 100 or 500.
2.2 Base Model Training
A separate base model (also called a base learner) is trained independently on each bootstrap sample. The base learner can be any machine learning algorithm, though bagging is most beneficial for high-variance, unstable learners such as decision trees.
2.2.1 Independence of Learners
Training is performed independently for each base model: the parameter estimation of one model does not influence another. This independence is the key source of the variance reduction effect, as the errors of the individual models are ideally uncorrelated.
2.2.2 Parallelizability
Because base models are trained independently, bagging is naturally parallelizable. In practice, each model can be trained on a separate CPU core or distributed across multiple machines, making bagging computationally efficient for large-scale problems.
2.3 Aggregation Methods
Once all base models are trained, their predictions are combined to produce the final ensemble output. The aggregation method depends on the task type.
2.3.1 Averaging for Regression
For regression tasks, bagging averages the predictions of all base models. If the base models have unbiased errors, the average reduces variance by a factor of *B* (assuming uncorrelated errors), leading to a more stable estimate.
2.3.2 Majority Voting for Classification
For classification tasks, bagging uses majority voting (or plurality voting if more than two classes). Each base model casts a "vote" for a class label, and the class with the most votes becomes the ensemble's prediction. This approach can be extended to weighted voting if base models have different confidence levels.
3 Properties
Bagging exhibits several important statistical and practical properties that explain its effectiveness and limitations.
3.1 Variance Reduction
The primary benefit of bagging is reducing the variance of the prediction. By averaging many models trained on perturbed data, the ensemble smooths out the high variability that characterizes unstable learners.
3.1.1 Mathematical Intuition
Let each base model's prediction have variance *σ*². If the *B* models are independent, the variance of the averaged prediction is *σ*²/*B*. While models trained on bootstrap samples are not fully independent (due to overlapping data), the correlation is generally low, so variance decreases substantially.
3.1.2 Effect on Overfitting
Because overfitting is often a symptom of high variance, bagging reduces overfitting. The ensemble is less likely to fit noise in the training data, leading to better generalization on unseen data. However, bagging does not necessarily eliminate bias (see §3.2).
3.2 Bias Considerations
Bagging's effect on bias is more nuanced than its effect on variance. For unstable learners the bias remains largely unchanged, while for stable learners a small increase may occur.
3.2.1 Unchanged Bias for Unstable Learners
For high-variance, low-bias learners such as deep decision trees, bagging averages models that individually have low bias. The ensemble's expected bias is approximately the average of the base models' biases, which remains low. Thus, bagging improves the bias-variance trade-off in this case.
3.2.2 Slight Bias Increase for Stable Learners
When applied to low-variance, high-bias learners (e.g., linear models), bagging can slightly increase bias. Because bootstrap samples are not perfect replicates of the original data distribution, the averaged model may shift slightly away from the optimal predictor. The increase is usually negligible compared to the gains from variance reduction, but it explains why bagging is not recommended for stable learners.
3.3 Out-of-Bag (OOB) Error
An important by‑product of bootstrap sampling is the out-of-bag error, which provides an internal validation mechanism without requiring a separate hold‑out set.
3.3.1 OOB Samples Definition
For each base model, approximately 36.8% of the original training instances are not included in its bootstrap sample. These omitted instances are called out‑of‑bag (OOB) samples. Because each instance is left out of roughly one‑third of the models, a prediction for that instance can be obtained by aggregating only those base models for which it was OOB.
3.3.2 Error Estimation without Validation Set
The OOB error is computed by averaging the misclassification rate (for classification) or mean squared error (for regression) over all training instances, using the ensemble predictions constructed solely from OOB votes. This estimate is nearly unbiased and correlates well with test set error, eliminating the need for a separate validation set in many applications.
4 Variants
Several extensions and modifications of bagging have been developed to address specific shortcomings or to improve performance.
4.1 Random Forest
Random Forest is one of the most popular bagging variants, specifically designed for decision tree base learners.
4.1.1 Additional Feature Randomization
In addition to bootstrap sampling of data, Random Forest introduces randomness at the feature level: at each node of each tree, only a random subset of features (typically the square root of the total number) is considered for splitting. This further decorrelates the trees, leading to additional variance reduction.
4.1.2 Relationship to Bagging
Random Forest is essentially bagging applied to decision trees with an extra layer of randomization. The OOB error estimation and aggregation methods (averaging for regression, majority vote for classification) are identical to those of standard bagging. Random Forest typically outperforms plain bagged trees on most tasks.
4.2 Pasted Bagging
Pasted bagging is a variant suitable for very large datasets where computational resources are limited.
4.2.1 Applicable to Large Datasets
Pasted bagging trains each base model on a small subsample of the full data. This reduces training time and memory usage. The subsamples are drawn without replacement to maximize diversity. When the dataset is massive, pasted bagging can achieve competitive performance with standard bagging using a fraction of the data.
4.2.2 Subsampling without Replacement
Unlike standard bootstrap sampling (with replacement), pasted bagging draws subsets of size *m* (much smaller than *n*) without replacement. This guarantees that all observations in a given subsample are distinct, further reducing correlation among base models.
4.3 Bagging with Other Learners
While decision trees are the most common base learner for bagging, the technique can be applied to any machine learning algorithm.
4.3.1 Neural Networks
Bagging neural networks (sometimes called "ensemble neural networks") reduces the variance typically associated with backpropagation training, especially when the network architecture is large. The independence assumption is justified if each network is initialized with different random weights. However, the computational cost is significant because training many deep networks is expensive.
4.3.2 Support Vector Machines
Bagging Support Vector Machines (SVMs) can improve generalisation when the SVM is unstable (e.g., with a linear kernel on noisy data). However, SVMs are often inherently stable, so the gains are marginal. Combined with the high training cost of SVMs, bagging is rarely used in practice for this learner.
5 Applications
Bagging has been applied to a wide range of supervised learning tasks. Its versatility and robustness make it a standard tool in many machine learning pipelines.
5.1 Classification Tasks
Bagging is widely used for classification problems such as spam detection, medical diagnosis, and image recognition. The majority‑voting aggregation is straightforward, and the ensemble often achieves higher accuracy than any single model. Random Forest, a bagging variant, is a top performer on many tabular classification benchmarks.
5.2 Regression Tasks
In regression problems (e.g., house price prediction, weather forecasting, stock market trend estimation), bagging averages the outputs of base models. The technique is especially useful when the base learner is prone to overfitting, such as deep regression trees or high‑order polynomial models.
5.3 Anomaly Detection
Bagging can be adapted for anomaly (outlier) detection by training detectors (e.g., isolation forests or one‑class SVMs) on bootstrap samples and aggregating their anomaly scores. The ensemble approach reduces the false positive rate caused by a single detector overfitting to the training data. OOB errors also provide a principled way to compute anomaly scores without a separate validation set.
6 Limitations
Despite its strengths, bagging has several limitations that practitioners must consider.
6.1 Computational Cost
Bagging requires training *B* separate base models. For large *B* and complex learners (e.g., deep neural networks), the total training time can become prohibitive. Memory usage also scales linearly with the number of models stored. Parallelisation helps but is not a panacea.
6.2 Diminishing Returns with Many Models
As the number of bootstrap replicates increases, the marginal benefit of adding more models decreases. After a certain point (typically a few hundred), adding extra trees or other learners yields negligible improvement in test error while increasing computational overhead.
6.3 Ineffectiveness on Very Low-Variance Learners
Bagging provides little benefit for learners that already have low variance, such as naive Bayes, linear regression (with few features), or *k*-nearest neighbors with large *k*. Because the variance reduction is minimal, the extra computational cost is rarely justified.
7 Comparison to Other Ensemble Methods
Bagging is one of several ensemble techniques. Its key distinction from other methods lies in how it introduces diversity and aggregates predictions.
7.1 Bagging vs. Boosting
Boosting (e.g., AdaBoost, Gradient Boosting) trains models sequentially, with each new model focusing on the mistakes of the previous ones. In contrast, bagging trains models independently in parallel. Boosting generally reduces both bias and variance, while bagging primarily reduces variance. Boosting can achieve lower error on some problems but is more prone to overfitting, especially if the base learners are complex.
7.2 Bagging vs. Stacking
Stacking (stacked generalization) combines multiple base models using a meta‑learner that is trained on the outputs of the base models. Unlike bagging, which uses a fixed aggregation rule (average or vote), stacking learns the optimal combination from data. Bagging is simpler and less prone to overfitting on small datasets; stacking can yield better performance when the base models have complementary strengths, but requires careful design to avoid overfitting.