The wisdom of crowds applies to machine learning: combining multiple models often produces better predictions than any single model alone. This is the fundamental insight behind ensemble methods — techniques that train multiple “weak” learners and combine their predictions to create a stronger learner. Ensemble methods consistently top Kaggle competitions and dominate real-world prediction tasks. Understanding them is essential for any serious ML practitioner.
Bagging: Bootstrap Aggregating
Bagging trains multiple models on different bootstrap samples (random subsets with replacement) of the training data, then averages their predictions. Because each model sees a slightly different dataset, they make different errors — and those errors cancel out when averaged. The most famous bagging algorithm is the Random Forest:
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, random_state=42)
# Manual bagging with decision trees
bagging = BaggingClassifier(
estimator=DecisionTreeClassifier(max_depth=None),
n_estimators=100,
max_samples=0.8, # each tree sees 80% of the data
max_features=0.8, # each tree uses 80% of features
bootstrap=True,
random_state=42
)
# Random Forest = bagging + random feature selection at each split
rf = RandomForestClassifier(
n_estimators=200,
max_features='sqrt', # standard setting for classification
min_samples_leaf=2,
n_jobs=-1,
random_state=42
)
for name, model in [('Bagging', bagging), ('Random Forest', rf)]:
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
print(f"{name}: {scores.mean():.3f} (+/- {scores.std():.3f})")
Random Forests add an extra randomisation step: at each split, they only consider a random subset of features (typically sqrt(n_features) for classification). This decorrelates the trees even further, reducing variance. Feature importance from Random Forests is also a useful tool for understanding which variables drive your predictions.
Boosting: Learning from Mistakes
While bagging trains models in parallel on different data subsets, boosting trains models sequentially — each new model focuses on the examples the previous models got wrong. This iterative correction is powerful but requires careful tuning to avoid overfitting.
AdaBoost was the original boosting algorithm. Gradient Boosting is the modern standard. XGBoost and LightGBM are highly optimised implementations that dominate tabular data competitions:
from sklearn.ensemble import GradientBoostingClassifier, AdaBoostClassifier
import xgboost as xgb
import lightgbm as lgb
# Gradient Boosting (sklearn)
gbm = GradientBoostingClassifier(
n_estimators=200,
learning_rate=0.05, # smaller = better generalisation, more trees needed
max_depth=4,
subsample=0.8, # stochastic gradient boosting
min_samples_leaf=10,
random_state=42
)
# XGBoost — faster, better regularisation
xgboost = xgb.XGBClassifier(
n_estimators=500,
learning_rate=0.05,
max_depth=5,
subsample=0.8,
colsample_bytree=0.8, # feature subsampling per tree
reg_alpha=0.1, # L1 regularisation
reg_lambda=1.0, # L2 regularisation
eval_metric='logloss',
early_stopping_rounds=50,
random_state=42
)
# LightGBM — fastest for large datasets
lgbm = lgb.LGBMClassifier(
n_estimators=500,
learning_rate=0.05,
num_leaves=31,
feature_fraction=0.8,
bagging_fraction=0.8,
bagging_freq=5,
random_state=42
)
from sklearn.model_selection import train_test_split
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
xgboost.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
print(f"XGBoost accuracy: {xgboost.score(X_val, y_val):.3f}")
Stacking: Meta-Learning
Stacking (stacked generalisation) takes ensemble methods to another level. Multiple base models are trained, and their predictions become the input features for a second-level “meta-model” that learns how to best combine them. This meta-model learns which base models are reliable in which situations:
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
# Base models with diverse learning approaches
base_models = [
('rf', RandomForestClassifier(n_estimators=100, random_state=42)),
('xgb', xgb.XGBClassifier(n_estimators=200, random_state=42, eval_metric='logloss')),
('svm', SVC(probability=True, random_state=42)),
]
# Meta-model learns to combine base model predictions
stacking = StackingClassifier(
estimators=base_models,
final_estimator=LogisticRegression(C=1.0),
cv=5, # use 5-fold CV to generate out-of-fold predictions
stack_method='predict_proba',
passthrough=False # set True to also pass original features to meta-model
)
scores = cross_val_score(stacking, X, y, cv=5, scoring='accuracy')
print(f"Stacking: {scores.mean():.3f} (+/- {scores.std():.3f})")
Which Ensemble Method Should You Use?
For most tabular data problems, start with XGBoost or LightGBM — they’re the fastest path to strong results with minimal tuning. Random Forests are more robust when you have limited time to tune hyperparameters and your dataset is moderate-sized. Stacking is worth the complexity only in competitions or when you’ve already extracted most of the performance from individual models and need that last 0.5% improvement. For very large datasets (millions of rows), LightGBM is the practical choice due to its speed advantage.
Frequently Asked Questions
Do ensemble methods always outperform single models?
Usually on structured/tabular data, yes. But on highly structured problems where a single model already fits well, ensembles add complexity without proportional benefit. Deep learning on images and text often beats ensembles because neural networks already implicitly learn ensemble-like internal representations.
How many estimators (trees) should I use?
More trees are almost always better for bagging methods (until performance plateaus). For boosting, more trees with a smaller learning rate generally gives better generalisation than fewer trees with a large learning rate. Use early stopping to find the optimal count automatically.
Is feature importance reliable in ensemble methods?
Random Forest feature importance (based on impurity decrease) can be biased toward high-cardinality features. Permutation importance is more reliable — it measures the drop in performance when a feature’s values are shuffled. Use sklearn.inspection.permutation_importance for more trustworthy results.
