Ensemble Methods in Machine Learning – Bagging, Boosting and Stacking Explained

Ensemble methods combine multiple machine learning models to produce predictions that are more accurate and robust than any individual model. They are the dominant approach in structured/tabular data competitions (Kaggle) and production ML systems — XGBoost and LightGBM ensemble methods won the majority of Kaggle competitions between 2015 and 2024, and stacked ensembles routinely top leaderboards. Understanding the theory, mechanics, and trade-offs of ensemble methods is essential for any data scientist and a frequent subject of machine learning interviews.

This guide covers all major ensemble strategies — bagging, boosting, stacking, blending, and voting — with intuition, mathematics, and practical guidance on when to use each. For the specific hyperparameters of gradient boosting frameworks, see our dedicated Gradient Boosting with XGBoost, LightGBM and CatBoost guide. For evaluating which ensemble performs best on your data, see our Model Evaluation and Hyperparameter Tuning Guide.

Why Ensembles Work — The Bias-Variance Decomposition

A single model’s prediction error decomposes into three components: Bias² (systematic error from wrong assumptions), Variance (sensitivity to training data fluctuations), and Irreducible Noise. Ensemble methods attack each source differently.

The wisdom of crowds principle: If individual models make independent errors, averaging their predictions cancels out those errors. If you have N models each with error variance σ², averaging their predictions reduces variance to σ²/N — a linear reduction. The key condition is independence: the models must make different errors, not the same errors. This is why diversity is the central design principle of all ensemble methods.

The three ensemble families differ in how they create diversity and which error component they target:

Bagging — Bootstrap Aggregating

Bagging (Breiman, 1996) creates diversity by training each model on a different bootstrap sample of the training data. A bootstrap sample is drawn with replacement — meaning some training examples appear multiple times while others are excluded (the excluded examples are the “out-of-bag” (OOB) sample, which serves as a free validation set).

How bagging reduces variance: A deep decision tree has very high variance — small changes in training data produce very different trees. Each bagged tree sees a different random subset of data and learns different patterns. Averaging N such trees smooths out the individual idiosyncrasies. The bias of each tree is roughly preserved (they are all deep trees), but the variance is divided by N (assuming independence).

Random Forest = Bagging + Feature Randomness: Random Forest adds a second source of randomness — at each split, only a random subset of features (sqrt(d) for classification, d/3 for regression) is considered. This further de-correlates the trees, producing better variance reduction than pure bagging. Random Forest is one of the most robust off-the-shelf algorithms: it handles missing values, mixed feature types, high dimensionality, and requires minimal hyperparameter tuning. It also provides free feature importance and OOB error estimates.

Key Random Forest hyperparameters:

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

rf = RandomForestClassifier(
    n_estimators=300,
    max_features='sqrt',
    min_samples_leaf=2,
    n_jobs=-1,
    oob_score=True,
    random_state=42
)
rf.fit(X_train, y_train)

print(f'OOB Score: {rf.oob_score_:.4f}')   # free validation estimate
# Feature importance
import pandas as pd
feat_imp = pd.Series(rf.feature_importances_, index=feature_names)
feat_imp.sort_values(ascending=False).head(15).plot(kind='barh')

Boosting — Sequential Error Correction

Where bagging trains models in parallel and combines them by averaging, boosting trains models sequentially — each new model focuses on the examples the previous models got wrong. Boosting reduces bias by iteratively correcting errors, whereas bagging reduces variance by averaging independent learners.

AdaBoost (Adaptive Boosting, 1996): The first practical boosting algorithm. After each training round, misclassified examples are given higher weight so the next model pays more attention to them. The final prediction is a weighted vote of all models, where more accurate models get higher weight. AdaBoost uses decision stumps (1-split trees) as weak learners. It is sensitive to outliers and noisy data — mislabelled examples get increasing weight and can dominate training.

Gradient Boosting (Friedman, 2001): Generalises AdaBoost by framing boosting as gradient descent in function space. Instead of reweighting examples, each new model is trained to predict the negative gradient of the loss function — the residuals for MSE loss. The ensemble prediction is updated in the direction that minimises the loss. This generalisation allows any differentiable loss function (MSE, cross-entropy, MAE, custom losses). The learning rate (shrinkage) scales each tree’s contribution, controlling the speed of gradient descent and providing regularisation — lower learning rate with more trees typically outperforms higher learning rate with fewer trees.

For in-depth coverage of XGBoost, LightGBM, and CatBoost — the three dominant gradient boosting frameworks — see our Gradient Boosting guide. For interview questions specifically on boosting algorithms, our ML Interview Q&A covers the most commonly asked questions including XGBoost’s regularisation terms and LightGBM’s GOSS and EFB techniques.

Comparison of gradient boosting frameworks:

Stacking — Learning to Combine

Stacking (Wolpert, 1992) takes the combination step further — instead of averaging base model predictions, it trains a meta-learner to discover the optimal combination. The meta-learner is trained on the out-of-fold predictions of the base models, learning which models to trust in which situations.

Stacking procedure:

1. Split training data into k folds (typically k=5). 2. For each fold: train all base models on the remaining k-1 folds and predict the held-out fold. 3. Concatenate out-of-fold predictions → meta-features matrix (same shape as training data). 4. Train meta-learner on meta-features (with the original target). 5. For test data: predict with each base model trained on full training data; average predictions (or use all k base model versions); feed to meta-learner.

The key insight: using out-of-fold predictions prevents the meta-learner from being trained on predictions the base models already “saw” — which would cause overfitting. This is why stacking uses cross-validation internally rather than a simple hold-out.

What to use as a meta-learner: Logistic Regression (for classification) or Ridge Regression (for regression) are the most common — they are interpretable, fast, and resistant to overfitting on the small meta-features dataset. Gradient boosting as a meta-learner can work but risks overfitting. Neural networks are rarely better than Ridge for this purpose.

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC

base_learners = [
    ('rf',  RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1)),
    ('gbm', GradientBoostingClassifier(n_estimators=200, learning_rate=0.05, random_state=42)),
    ('svm', SVC(probability=True, kernel='rbf', C=1.0)),
]
meta_learner = LogisticRegression(C=0.1, max_iter=500)

stack = StackingClassifier(
    estimators=base_learners,
    final_estimator=meta_learner,
    cv=5,           # k for out-of-fold generation
    stack_method='predict_proba',
    passthrough=False,  # True: also pass original features to meta-learner
    n_jobs=-1
)
stack.fit(X_train, y_train)

Blending — Simplified Stacking

Blending is a simpler alternative to stacking. Instead of generating meta-features via cross-validation, blending uses a single hold-out set. Train base models on 80% of training data. Generate predictions on the 20% hold-out set → meta-features. Train meta-learner on these meta-features. Generate predictions on the test set by averaging all base model test predictions.

Advantages over stacking: simpler, faster, less code. Disadvantages: wastes 20% of training data (not used by base models), higher variance in meta-features (only one fold), more likely to overfit. Use blending for quick ensembling in competitions; use proper stacking for production systems.

Voting Ensembles

Voting ensembles combine predictions from multiple different models with no meta-learning. Hard voting (classification): each model votes for a class; majority wins. Simple but ignores prediction confidence. Soft voting: average the predicted probabilities across models; take argmax. Almost always better than hard voting because it uses calibrated probability estimates. For regression: simply average the predictions. Weighted voting assigns different weights to different models — the weights can be tuned via cross-validation or set based on individual model performance.

Interview Q&A — Ensemble Methods

Q: Why do ensemble methods outperform individual models? Ensembles average out individual model errors. If errors are uncorrelated (diverse models), averaging N models reduces variance by a factor of N. Ensembles also reduce bias (boosting) by combining many weak learners into one strong learner.

Q: What is the out-of-bag (OOB) error in Random Forest? Each tree is trained on ~63.2% of the data (bootstrap sample). The remaining ~36.8% (OOB samples) are used to evaluate that tree. Averaging OOB predictions across all trees gives the OOB error — a free estimate of generalisation performance without needing a separate validation set. OOB error is typically close to 5-fold CV error.

Q: Why does Random Forest use sqrt(features) at each split? Using all features at each split would cause trees to all choose the same strong feature first (high correlation). Using sqrt(d) features forces trees to discover different pathways through the feature space, de-correlating them and maximising the variance reduction from averaging. Our Feature Engineering guide covers feature importance from Random Forest in detail.

Q: What is the difference between bagging and boosting for imbalanced classes? Both can struggle with severe class imbalance. For Random Forest: use class_weight=’balanced’ or balance the bootstrap samples. For boosting: use scale_pos_weight (XGBoost) or is_unbalance (LightGBM). Evaluate with PR-AUC rather than ROC-AUC for imbalanced problems — ROC-AUC can be misleadingly high when true negatives dominate.

Q: When would you NOT use an ensemble? When interpretability is required (a loan denial explanation needs a single model), when training time is severely constrained, when the individual models are already very strong (diminishing returns), or when deployment infrastructure cannot handle multiple model inference calls.

Q: What is the role of diversity in ensemble learning? If all base models make identical errors, averaging them does not reduce error at all. Diversity is created through: different training data subsets (bagging), different feature subsets (random subspaces), different algorithm families (stacking with RF + SVM + GBM), different hyperparameters, or different random seeds. The bias-variance-covariance decomposition of ensemble error shows that the benefit of averaging N models equals σ²(1-ρ)/N where ρ is the average pairwise correlation between model errors — minimising ρ maximises ensemble benefit.

Ensemble methods are a core topic in both applied ML and interviews. Our ML Interview Q&A covers 60 questions including ensemble-specific questions on variance reduction, tree pruning, and regularisation. For production deployment of ensemble models, see our MLOps Interview Q&A on model serving latency trade-offs when running multiple base models in production.