1.1 What ensemble learning is
Ensemble learning is one of the central paradigms in machine learning, and its core idea comes from an old proverb: three cobblers working together are a match for Zhuge Liang. Statistics and machine learning formalize that idea as follows: combine several individual learners working on the same problem, and you get predictions that are more accurate and more stable than any single model can produce.
Formally, take a training set $\mathcal{D} = \{(x_1, y_1), (x_2, y_2), \ldots, (x_N, y_N)\}$, where $x_i \in \mathcal{X}$ is the input and $y_i \in \mathcal{Y}$ is the target label. A conventional single-model approach tries to learn a mapping $h: \mathcal{X} \to \mathcal{Y}$ that minimizes some loss function. Ensemble learning instead builds a set of $M$ base learners $\{h_1, h_2, \ldots, h_M\}$ and combines them through some strategy to produce the final prediction:
$$H(x) = \Phi\left(h_1(x), h_2(x), \ldots, h_M(x)\right)$$Here $\Phi(\cdot)$ is the combination function. It can be a plain average, a weighted average, or something more elaborate such as a meta-learner.
The theory behind ensembling goes back to Condorcet’s jury theorem (1785): if every member judges correctly with probability $p > 0.5$, and the members are independent, then as the number of members grows the probability that a majority vote is correct approaches 1. For a simple majority vote over $M$ members ($M$ odd), the probability that the group is correct is
$$P_{\text{maj}} = \sum_{k=(M+1)/2}^{M} \binom{M}{k} p^k (1-p)^{M-k}$$With $p = 0.6$, for instance, $M = 11$ gives $P_{\text{maj}} \approx 0.753$ and $M = 101$ gives $P_{\text{maj}} \approx 0.979$.
Put in the language of classifiers (Zhou Zhihua, Machine Learning, Ch. 8): let $T$ binary base learners each have error rate $\epsilon < 0.5$ and be independent of one another. By Hoeffding’s inequality, the error rate of a majority vote satisfies
$$P\left(H(x) \neq f(x)\right) = \sum_{k=0}^{\lfloor T/2 \rfloor} \binom{T}{k} (1-\epsilon)^k \epsilon^{T-k} \leq \exp\left(-\frac{1}{2} T (1-2\epsilon)^2\right)$$so the error rate falls off exponentially in $T$. Independence is an assumption that basically never holds in machine learning — base learners all train on the same data for the same task — but it gives an intuitive reason to expect that combining weak models can yield a strong one, and it shows why diversity among individual learners matters so much (see Section 1.5.3).
1.2 Why use ensemble learning
In practice a single model runs into several problems:
(1) The bias-variance dilemma
Under squared loss, suppose the data are generated by $y = f(x) + \varepsilon$ with $\mathbb{E}[\varepsilon] = 0$ and $\operatorname{Var}(\varepsilon) = \sigma^2$, and write $h_{\mathcal{D}}$ for the model learned from training set $\mathcal{D}$. At a fixed input $x$, the expected generalization error of the learned model splits into three parts:
$$\mathbb{E}_{\mathcal{D},\varepsilon}\left[\left(y - h_{\mathcal{D}}(x)\right)^2\right] = \underbrace{\left(\mathbb{E}_{\mathcal{D}}\left[h_{\mathcal{D}}(x)\right] - f(x)\right)^2}_{\text{bias}^2} + \underbrace{\mathbb{E}_{\mathcal{D}}\left[\left(h_{\mathcal{D}}(x) - \mathbb{E}_{\mathcal{D}}\left[h_{\mathcal{D}}(x)\right]\right)^2\right]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{noise}}$$Here $f(x)$ is the true target function and $\sigma^2$ is irreducible noise. The expectation $\mathbb{E}_{\mathcal{D}}$ is taken over randomly drawn training sets, so the variance measures how sensitive the model is to perturbations of the training data. Low-bias models such as deep neural networks tend to have high variance and to overfit; low-variance models such as linear regression tend to have high bias and to underfit. Ensembles offer a way to reduce both at once: bagging cuts variance, boosting cuts bias, and stacking pools the strengths of several models.
(2) Stability of model selection
Different learning algorithms make different assumptions about the data distribution. Nudge the distribution or perturb the training set slightly and a single model’s predictions can swing wildly. Averaging several models smooths out that instability and makes predictions more reliable.
(3) A larger hypothesis space
A single model’s hypothesis space may be limited — a linear model, for example, can only fit linear relationships. By combining models, an ensemble can approximate more complex functions and so extend what the model can express.
(4) Strong track record in practice
In competitions and in industry, ensembles are almost always the foundation of the best results. From Kaggle contests to recommender systems and credit risk, they’re hard to do without. Hastie, Tibshirani and Friedman devote several chapters of The Elements of Statistical Learning (2nd ed., 2009) to bagging (§8.7), boosting (Ch. 10), random forests (Ch. 15) and ensemble learning (Ch. 16). The price of ensembling is mostly more compute and less interpretability.
1.3 The three families of ensemble methods
By how base learners are trained and combined, ensembles fall into three families: bagging, boosting and stacking. Each reflects a different philosophy. Bagging wants agreement among models that differ — it trains in parallel and independently to reduce uncertainty. Boosting wants steady refinement — it corrects errors in sequence and raises accuracy step by step. Stacking wants each model to play to its strengths — a meta-learner works out how best to use them.
1.3.1 Bagging: bootstrap aggregating
Breiman proposed bagging in 1996. The idea: draw several different training subsets by bootstrap sampling, train a homogeneous base learner independently on each subset, then aggregate the predictions by averaging (regression) or majority vote (classification).
Training. Given an original training set of $N$ samples, draw $B$ bootstrap samples with replacement, each also of size $N$. The probability that a given sample is never drawn in $N$ draws with replacement is $\left(1 - \frac{1}{N}\right)^N \to \frac{1}{e} \approx 0.368$ as $N \to \infty$, so on average each bootstrap sample holds about $63.2\%$ of the distinct original samples. The other $\approx 36.8\%$ are out-of-bag (OOB) samples that can be used to evaluate the model.
Goal. Bagging aims to reduce variance. With a high-variance, low-bias base learner such as a deep decision tree, averaging several independently trained models cuts the variance of the prediction substantially and thereby reduces overfitting.
“Bagging reduces variance by averaging predictions from multiple models, making it ideal for high-variance algorithms like decision trees.” — GeeksforGeeks, 2024
Representative algorithm. Random Forest (Breiman, 2001) is the best-known example. It bootstraps over samples and also selects a random subset of features at each split, which raises the diversity of the base learners further and pushes variance down again.
1.3.2 Boosting
The core idea of boosting: train models in sequence, with each new model aimed at the samples the previous one got wrong, then combine all their predictions into a weighted output.
Unlike bagging’s parallel, independent training, boosting trains sequentially. In each round, samples misclassified by earlier models get larger weights, which forces later models to learn those hard cases. The weak learners are then combined with weights into a strong learner.
Goal. Boosting aims to reduce bias. By fitting residuals or hard samples step by step, it drives training error down and gives the model more expressive power. Note that bias reduction is the main goal, not the whole story: the large empirical study by Bauer & Kohavi (1999) showed that on unstable learners such as decision trees, AdaBoost cuts both bias and variance, whereas bagging’s gains come almost entirely from variance reduction (and on very stable learners such as naive Bayes, AdaBoost actually increases variance).
“Boosting trains models sequentially, with each model focusing on correcting the errors made by its predecessor. Boosting is designed to reduce bias by focusing on the hardest-to-predict data points.” — Interviewnode, 2024
Representative algorithms. AdaBoost (Adaptive Boosting, 1995/1997), Gradient Boosting Machine (GBM, 2001), XGBoost (2016), LightGBM (2017) and CatBoost (arXiv 2017, NeurIPS 2018), among others.
1.3.3 Stacking
Wolpert proposed stacking in 1992. It’s a more flexible strategy: train several heterogeneous base learners, then use another model — the meta-learner — to learn how best to combine their predictions.
Training. Stacking runs in two stages. In the first, several different kinds of base learner (a decision tree, an SVM, a neural network, and so on) are trained on the original training set. To keep the meta-learner from overfitting, the meta-learner’s training data usually comes from $k$-fold cross-validation: split the training set into $k$ folds, train the base learners on $k-1$ folds and predict the held-out fold, repeat $k$ times, then stack all the predictions as the meta-learner’s input. In the second stage the meta-learner (logistic or linear regression, say) is trained on the meta-features, which are the base learners’ out-of-fold predictions. At prediction time the base learners are usually retrained on the full training set and their predictions fed to the meta-learner; scikit-learn’s StackingClassifier/StackingRegressor work this way (cross_val_predict generates the meta-features). Feeding in the base learners’ fitted values on the training set instead is the most common stacking mistake: it biases the meta-learner toward whichever base learner overfits worst.
In “Stacked Regressions,” Breiman (1996b) went further and recommended non-negative least squares for the meta-learner on regression problems, which yields more stable combination weights. A simpler variant is blending, which generates meta-features from a fixed hold-out set instead of $k$-fold cross-validation; it’s easier to implement but leaves less data for training the meta-learner.
“Stacking mainly differs from bagging and boosting on two points: First, stacking often considers heterogeneous weak learners… Second, stacking learns to combine the base models using a meta-model.” — Duchesnay, PyStatsML
Goal. Stacking aims at higher overall accuracy by exploiting the complementarity of different algorithms.
Representative algorithm. Any combination of models plus a meta-learner. A common setup is decision tree + SVM + neural network, with logistic regression as the meta-learner.
1.4 How the three families differ
To make the differences clear, here’s a comparison across several dimensions:
| Dimension | Bagging | Boosting | Stacking |
|---|---|---|---|
| Training | Parallel and independent; base learners don’t affect each other | Sequential; each round depends on earlier results | Two stages: train base learners in parallel, then train the meta-learner |
| Main goal | Reduce variance (less overfitting) | Reduce bias (less underfitting) | Higher overall accuracy |
| Base learners | Homogeneous, usually low-bias, high-variance “strong” learners (e.g. unpruned deep trees) | Homogeneous weak learners (e.g. decision stumps, shallow trees) | Usually heterogeneous strong learners (different algorithms) |
| Data sampling | Bootstrap sampling | Adaptive reweighting or residual fitting | Original data or cross-validation |
| Combination | Simple average or majority vote | Weighted sum (AdaBoost weights by error rate; GBM accumulates by step size and learning rate) | Meta-learner learns the best combination |
| Correlation between base learners | Lowered by bootstrap sampling | High, since later models depend on earlier ones | Diversity comes naturally from mixing algorithms |
| Sensitivity to outliers | Low (averaging effect) | Higher (AdaBoost keeps raising the weight of misclassified samples); GBM can switch to a robust loss such as Huber | Depends on the base learners chosen |
| Representative algorithm | Random Forest | AdaBoost, GBM, XGBoost, LightGBM | Any models + a meta-learner |
| Parallelism | Highly parallel, scales well | Sequential, limited parallelism | First stage parallel, second stage sequential |
| Theoretical guarantees | Variance reduction is provable (see Section 1.5.1) | Comes from weak learnability in the PAC framework: training error falls exponentially under the weak learning assumption, plus margin-based generalization bounds | Mostly empirical and dependent on cross-validation; weaker theory |
1.5 Why bagging reduces variance and boosting reduces bias
1.5.1 The intuition behind bagging’s variance reduction
Take $B$ identically distributed base learners with predictions $\hat{f}_1(x), \hat{f}_2(x), \ldots, \hat{f}_B(x)$, each with expectation $f(x)$ and variance $\sigma^2$. Bagging predicts their average:
$$\hat{f}_{\text{bag}}(x) = \frac{1}{B} \sum_{b=1}^B \hat{f}_b(x)$$The variance of that average is
$$\operatorname{Var}\left[\hat{f}_{\text{bag}}(x)\right] = \frac{1}{B^2} \sum_{b=1}^B \operatorname{Var}\left[\hat{f}_b(x)\right] = \frac{\sigma^2}{B}$$(The first equality requires the base learners to be independent. In practice the bootstrap samples all come from the same training data, so the learners are pairwise correlated with coefficient $\rho > 0$; the variance is then $\rho \sigma^2 + \frac{1-\rho}{B}\sigma^2$ (ESL Eq. 15.1), which approaches $\rho\sigma^2$ as $B \to \infty$ — adding learners only removes the second term.)
Averaging leaves the expectation untouched: $\mathbb{E}\left[\hat{f}_{\text{bag}}(x)\right] = \mathbb{E}\left[\hat{f}_b(x)\right]$, so bagging’s bias equals that of a single base learner trained on a bootstrap sample. That’s exactly why bagging wants low-bias, high-variance base learners such as deep trees.
Intuition. Bagging is like polling several experts, each working from slightly different information (their bootstrap sample). When their opinions are independent or only weakly correlated, the average is steadier than any single opinion and fluctuates less. That’s what variance means here: how much the prediction moves when the training data are perturbed. Random Forest lowers the correlation $\rho$ between base learners further by randomizing features, which strengthens the variance reduction.
“Bagging (Bootstrap Aggregating): is the process of training several models using random subsets of the training data to minimize overfitting.” — PDfs.semanticscholar.org
1.5.2 The intuition behind boosting’s bias reduction
Each boosting round adds a new base learner on top of the current model, fitted to that model’s residuals or errors ($F_m$ here denotes the ensemble at round $m$, to keep it distinct from the true function $f(x)$ above):
$$F_m(x) = F_{m-1}(x) + \beta_m h_m(x)$$Under squared loss, $h_m(x)$ is a weak learner fitted to the residuals $r_i = y_i - F_{m-1}(x_i)$; for a general loss, the residual is replaced by the negative gradient of the loss (the pseudo-residual). Chapters 3 and 5 cover this in detail.
Intuition. Suppose you want to approximate a complicated function with a series of simple ones. In the first round a simple model (a decision stump, say) gives a rough fit but leaves a sizeable systematic error, the residual. The second round learns that residual and corrects more of the error. As rounds accumulate, the model fills in the patterns it missed and its predictions approach the true target. Because every round chips away at systematic error, the overall bias — the systematic gap between predictions and truth — keeps shrinking.
Model complexity grows with the rounds, though: too many rounds and variance rises and the model overfits. The number of rounds $M$ is therefore itself a regularization parameter, controlled by a validation set or early stopping (see Chapter 5).
1.5.3 Diversity: the error-ambiguity decomposition
The variance formula in Section 1.5.1 shows that the lower the correlation $\rho$ between base learners, the bigger the gain from ensembling. Krogh & Vedelsby (1995) gave an exact statement for regression. Let the ensemble be the weighted average $\bar{h}(x) = \sum_{t=1}^T w_t h_t(x)$ with $w_t \geq 0$ and $\sum_{t=1}^T w_t = 1$. Then for any sample $(x, y)$,
$$\left(\bar{h}(x) - y\right)^2 = \sum_{t=1}^T w_t \left(h_t(x) - y\right)^2 - \sum_{t=1}^T w_t \left(h_t(x) - \bar{h}(x)\right)^2$$Taking expectations over the data distribution and writing the result as $E = \bar{E} - \bar{A}$:
- $E$: the ensemble’s generalization error;
- $\bar{E}$: the weighted average generalization error of the individual learners;
- $\bar{A}$: the weighted “ambiguity” of the individual learners relative to the ensemble, always $\geq 0$.
This error-ambiguity decomposition says the ensemble’s error can never exceed the weighted average of the individual errors, and that the ensemble improves as the individual learners become more accurate (smaller $\bar{E}$) and more diverse (larger $\bar{A}$) — accuracy and diversity together, what Zhou Zhihua calls “good but different” (Machine Learning, §8.5). The usual ways to add diversity in practice are to perturb the training samples (bootstrap sampling in bagging, reweighting in boosting), perturb the input features (random subspaces, feature sampling in random forests), perturb algorithm parameters, or mix different algorithms (stacking).
The decomposition holds exactly for squared loss. For the 0-1 loss of classification there is no comparably clean form, and “diversity” still has no single agreed-upon measure.
1.5.4 Why ensembles work: Dietterich’s three reasons
Dietterich (2000) gave three reasons why an ensemble often beats a single model:
- Statistical. With limited training data, several hypotheses may fit the training set about equally well. Picking one of them risks picking the wrong one; averaging them reduces that risk.
- Computational. Many learning algorithms only find local optima (greedy splits in a decision tree, gradient descent in a neural network). Combining local solutions reached from different starting points often lands closer to the true function than any one of them.
- Representational. The true function may lie outside any single model’s hypothesis space, whereas a weighted combination of hypotheses can represent a wider range of functions (this is the “larger hypothesis space” point from Section 1.2).
1.6 Flowcharts for the three methods
Figure 1-1: How bagging works
flowchart TD
A["Original training set
N samples"] --> B1["Bootstrap sample
set 1"]
A --> B2["Bootstrap sample
set 2"]
A --> B3["Bootstrap sample
set B"]
B1 --> C1["Train base learner
h₁(x)"]
B2 --> C2["Train base learner
h₂(x)"]
B3 --> C3["Train base learner
h_B(x)"]
C1 --> D["Aggregate predictions
H(x) = avg/mvote"]
C2 --> D
C3 --> D
style A fill:#f9f,stroke:#333
style D fill:#bbf,stroke:#333
Figure 1-2: How boosting works
flowchart TD
A["Initialize sample weights
w_i = 1/N"] --> B1["Round 1: train weak learner
G₁(x)"]
B1 --> C1["Compute error rate &
learner weight α₁"]
C1 --> D1["Update sample weights
raise weight of misclassified samples"]
D1 --> B2["Round 2: train weak learner
G₂(x)"]
B2 --> C2["Compute error rate &
learner weight α₂"]
C2 --> D2["Update sample weights"]
D2 --> B3["..."]
B3 --> C3["..."]
C3 --> D3["..."]
D3 --> E["Round M: train weak learner
G_M(x)"]
E --> F["Weighted combination
G(x) = sign(∑αₘGₘ(x))"]
style A fill:#f9f,stroke:#333
style F fill:#bbf,stroke:#333
Figure 1-3: How stacking works
flowchart TD
A["Original training set"] --> B1["Base learner 1
decision tree"]
A --> B2["Base learner 2
SVM"]
A --> B3["Base learner 3
neural network"]
B1 --> C1["Predictions
p₁(x)"]
B2 --> C2["Predictions
p₂(x)"]
B3 --> C3["Predictions
p₃(x)"]
C1 --> D["Meta-feature matrix
p1, p2, p3"]
C2 --> D
C3 --> D
D --> E["Meta-learner
logistic regression"]
E --> F["Final prediction
H(x)"]
style A fill:#f9f,stroke:#333
style F fill:#bbf,stroke:#333
1.7 Chapter summary
This article covered the basic idea of ensemble learning, its three families, and the diversity principle of “good but different.” Bagging reduces variance through parallel independent training on bootstrap samples, with Random Forest as its flagship; boosting reduces bias by training sequentially and focusing on hard samples, as AdaBoost and GBM do; stacking combines heterogeneous models through a meta-learner in pursuit of the best overall accuracy. Understanding how these three differ is the groundwork for everything that follows on boosting. As Breiman (1996) showed in his original bagging paper, and as Friedman, Hastie & Tibshirani (2000) later drew out in their statistical reading of boosting, ensembling is not just a practical technique but a substantial part of statistical learning theory.
Key references:
Breiman, L. (1996). “Bagging Predictors.” Machine Learning, 24(2), 123-140.
Breiman, L. (1996b). “Stacked Regressions.” Machine Learning, 24(1), 49-64.
Breiman, L. (2001). “Random Forests.” Machine Learning, 45(1), 5-32.
Wolpert, D. H. (1992). “Stacked Generalization.” Neural Networks, 5(2), 241-259.
Krogh, A., & Vedelsby, J. (1995). “Neural Network Ensembles, Cross Validation, and Active Learning.” Advances in Neural Information Processing Systems 7, 231-238.
Bauer, E., & Kohavi, R. (1999). “An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants.” Machine Learning, 36(1-2), 105-139.
Dietterich, T. G. (2000). “Ensemble Methods in Machine Learning.” In Multiple Classifier Systems (MCS 2000), LNCS 1857, 1-15. Springer.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. Ch. 8, 10, 15, 16.
Zhou, Z.-H. (2016). Machine Learning. Tsinghua University Press. Ch. 8.
Tree-based boosting evolved from AdaBoost and gradient boosting to optimized implementations such as XGBoost, LightGBM, and CatBoost.
