XGBoost Tutorial: Gradient Boosting in Python Explained (2026)

XGBoost (eXtreme Gradient Boosting) wins Kaggle competitions. It is fast, accurate, and handles messy real-world data better than almost anything else. This guide takes you from zero to a working model with hyperparameter tuning.

What Is Gradient Boosting?

Gradient boosting builds an ensemble of weak learners one at a time. Each new tree corrects the errors of the previous ensemble by fitting the residuals. XGBoost adds L1/L2 regularisation, second-order gradients, parallel tree construction, built-in missing value handling, and depth-first pruning.

Installation

pip install xgboost scikit-learn pandas numpy matplotlib

Your First Model

import pandas as pd, numpy as np, xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
df = pd.read_csv(url)
df['Sex'] = (df['Sex'] == 'male').astype(int)
df['Age'].fillna(df['Age'].median(), inplace=True)
df['Embarked'].fillna('S', inplace=True)
df['Embarked'] = df['Embarked'].map({'S':0,'C':1,'Q':2})

X, y = df[['Pclass','Sex','Age','SibSp','Parch','Fare','Embarked']], df['Survived']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

model = xgb.XGBClassifier(n_estimators=200, max_depth=4, learning_rate=0.1,
    subsample=0.8, colsample_bytree=0.8, eval_metric='logloss', random_state=42)
model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)
print(f'Accuracy: {accuracy_score(y_test, model.predict(X_test)):.4f}')

Key Hyperparameters

n_estimators — number of trees; use early stopping to find the sweet spot.

learning_rate — contribution of each tree; lower (0.01-0.1) is more robust but needs more trees.

max_depth — tree complexity; typical range 3-8.

subsample / colsample_bytree — fraction of rows/features per tree (0.7-0.9 reduces overfitting).

reg_alpha / reg_lambda — L1 and L2 regularisation weights.

Early Stopping

model = xgb.XGBClassifier(n_estimators=1000, learning_rate=0.05,
    max_depth=5, early_stopping_rounds=50, eval_metric='logloss')
model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=100)
print(f'Best iteration: {model.best_iteration}')

Grid Search Tuning

from sklearn.model_selection import GridSearchCV
param_grid = {'max_depth':[3,4,5], 'learning_rate':[0.05,0.1,0.2], 'n_estimators':[100,200,300]}
gs = GridSearchCV(xgb.XGBClassifier(eval_metric='logloss'), param_grid, cv=5, n_jobs=-1)
gs.fit(X_train, y_train)
print('Best:', gs.best_params_, 'Score:', gs.best_score_)

Feature Importance

import matplotlib.pyplot as plt
xgb.plot_importance(model, max_num_features=10, importance_type='gain')
plt.tight_layout(); plt.show()

XGBoost for Regression

from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_squared_error
housing = fetch_california_housing(as_frame=True)
X, y = housing.data, housing.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
reg = xgb.XGBRegressor(n_estimators=500, learning_rate=0.05, max_depth=5,
    early_stopping_rounds=30, eval_metric='rmse')
reg.fit(X_tr, y_tr, eval_set=[(X_te, y_te)], verbose=False)
print(f'RMSE: {mean_squared_error(y_te, reg.predict(X_te), squared=False):.4f}')

Common Mistakes

Not using early stopping leads to overfitting. Ignoring class imbalance — use scale_pos_weight = negative_count / positive_count. Tuning learning rate without adjusting n_estimators — always tune them together.

Conclusion

XGBoost is the gold standard for tabular machine learning. Lower learning rate plus more trees plus regularisation almost always outperforms a high learning rate with fewer trees. Master early stopping and you will beat most baselines.