Gradient Boosting Explained – XGBoost, LightGBM & CatBoost Guide 2026

Gradient boosting algorithms dominate tabular data competitions and production ML systems. XGBoost, LightGBM, and CatBoost consistently outperform neural networks on structured data while being faster to train and easier to interpret. This guide explains how boosting works and shows you how to use all three libraries effectively.

How Gradient Boosting Works

Gradient boosting builds an ensemble of weak learners (shallow decision trees) sequentially. Each new tree is trained to correct the errors of the previous ensemble — specifically, it fits the negative gradient of the loss function (the residual errors). The final prediction is a weighted sum of all trees. More trees = lower training error, but at risk of overfitting, which is controlled by learning rate, tree depth, and regularisation parameters.

XGBoost

pip install xgboost scikit-learn optuna

import xgboost as xgb
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt

X, y = make_classification(n_samples=10_000, n_features=30,
                             n_informative=15, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
X_tr, X_va, y_tr, y_va = train_test_split(X_tr, y_tr, test_size=0.2, random_state=42)

model = xgb.XGBClassifier(
    n_estimators       = 1000,
    learning_rate      = 0.05,
    max_depth          = 6,
    min_child_weight   = 1,
    subsample          = 0.8,    # row subsampling per tree
    colsample_bytree   = 0.8,    # feature subsampling per tree
    gamma              = 0,      # min loss reduction for split
    reg_alpha          = 0,      # L1 regularisation
    reg_lambda         = 1,      # L2 regularisation
    scale_pos_weight   = 1,      # set to neg/pos ratio for imbalanced data
    use_label_encoder  = False,
    eval_metric        = 'auc',
    early_stopping_rounds = 50,
    random_state       = 42,
    n_jobs             = -1
)

model.fit(X_tr, y_tr,
          eval_set=[(X_va, y_va)],
          verbose=100)

y_pred = model.predict_proba(X_te)[:, 1]
print(f'XGBoost Test AUC: {roc_auc_score(y_te, y_pred):.4f}')
print(f'Best iteration: {model.best_iteration}')

# Feature importance
xgb.plot_importance(model, max_num_features=15, importance_type='gain')
plt.tight_layout()
plt.show()

LightGBM – Faster for Large Datasets

LightGBM uses histogram-based splitting and leaf-wise (best-first) tree growth instead of XGBoost’s level-wise approach. It is typically 5-10x faster than XGBoost on large datasets and uses less memory.

pip install lightgbm

import lightgbm as lgb

dtrain = lgb.Dataset(X_tr, label=y_tr)
dval   = lgb.Dataset(X_va, label=y_va, reference=dtrain)

params = {
    'objective':        'binary',
    'metric':           'auc',
    'learning_rate':    0.05,
    'num_leaves':       63,          # controls model complexity (< 2^max_depth)
    'max_depth':        -1,          # -1 = no limit
    'min_child_samples': 20,
    'subsample':        0.8,
    'subsample_freq':   1,
    'colsample_bytree': 0.8,
    'reg_alpha':        0.1,
    'reg_lambda':       0.1,
    'n_jobs':           -1,
    'verbose':          -1,
    'random_state':     42,
}

callbacks = [
    lgb.early_stopping(50, verbose=True),
    lgb.log_evaluation(100)
]

model_lgb = lgb.train(
    params, dtrain,
    num_boost_round=1000,
    valid_sets=[dval],
    callbacks=callbacks
)

y_pred_lgb = model_lgb.predict(X_te)
print(f'LightGBM Test AUC: {roc_auc_score(y_te, y_pred_lgb):.4f}')

# SHAP values
import shap
explainer   = shap.TreeExplainer(model_lgb)
shap_values = explainer.shap_values(X_te)
shap.summary_plot(shap_values[1], X_te)

CatBoost – Best for Categorical Features

CatBoost handles categorical features natively without one-hot encoding, using ordered target statistics. It is the best choice when you have many high-cardinality categoricals and want to avoid data leakage from naive target encoding.

pip install catboost

from catboost import CatBoostClassifier, Pool

# Specify categorical feature indices
cat_features = [0, 5, 10]   # column indices of categorical columns

train_pool = Pool(X_tr, y_tr, cat_features=cat_features)
val_pool   = Pool(X_va, y_va, cat_features=cat_features)

model_cb = CatBoostClassifier(
    iterations         = 1000,
    learning_rate      = 0.05,
    depth              = 6,
    l2_leaf_reg        = 3,
    border_count       = 128,    # number of splits per feature
    eval_metric        = 'AUC',
    early_stopping_rounds = 50,
    random_seed        = 42,
    task_type          = 'GPU',  # use 'CPU' if no GPU
    verbose            = 100
)

model_cb.fit(train_pool, eval_set=val_pool)

y_pred_cb = model_cb.predict_proba(X_te)[:, 1]
print(f'CatBoost Test AUC: {roc_auc_score(y_te, y_pred_cb):.4f}')

# Feature importance
feat_imp = model_cb.get_feature_importance()

Hyperparameter Tuning with Optuna

import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    params = {
        'n_estimators':    trial.suggest_int('n_estimators', 100, 1000),
        'learning_rate':   trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
        'max_depth':       trial.suggest_int('max_depth', 3, 9),
        'subsample':       trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
        'reg_alpha':       trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
        'reg_lambda':      trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),
        'use_label_encoder': False,
        'eval_metric': 'auc',
        'early_stopping_rounds': 50,
        'random_state': 42,
        'n_jobs': -1,
    }
    model = xgb.XGBClassifier(**params)
    model.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], verbose=False)
    preds = model.predict_proba(X_va)[:, 1]
    return roc_auc_score(y_va, preds)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50, show_progress_bar=True)

print(f'Best AUC: {study.best_value:.4f}')
print(f'Best params: {study.best_params}')

Stacking Ensemble

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression

base_models = [
    ('xgb', xgb.XGBClassifier(n_estimators=300, learning_rate=0.05,
                                use_label_encoder=False, eval_metric='logloss',
                                random_state=42, n_jobs=-1)),
    ('lgb', lgb.LGBMClassifier(n_estimators=300, learning_rate=0.05,
                                 num_leaves=63, random_state=42, n_jobs=-1,
                                 verbose=-1)),
]

stack = StackingClassifier(
    estimators=base_models,
    final_estimator=LogisticRegression(C=0.1),
    cv=5, passthrough=False, n_jobs=-1
)
stack.fit(X_tr, y_tr)
y_pred_stack = stack.predict_proba(X_te)[:, 1]
print(f'Stacked AUC: {roc_auc_score(y_te, y_pred_stack):.4f}')

Which Library to Choose?

Use XGBoost when you want the most battle-tested library with the largest community and best documentation. Use LightGBM when training speed matters — it is significantly faster on large datasets (1M+ rows) and uses less RAM. Use CatBoost when you have many categorical features with high cardinality and want to avoid the complexity of manual target encoding. In practice, try all three and ensemble them — the diversity between their predictions often improves over any single model.

Conclusion

Gradient boosting remains the dominant approach for tabular ML in 2026. XGBoost, LightGBM, and CatBoost each have distinct strengths. The key hyperparameters that matter most are learning rate (lower = more trees needed but better generalisation), number of leaves/max depth (controls model complexity), and subsampling rates (prevent overfitting). Always use early stopping — it prevents overfitting and eliminates the need to tune n_estimators manually.