If you have ever watched someone make a decision by asking a series of questions — “Is it raining? If yes, take an umbrella. Is it cold? If yes, wear a jacket.” — you already understand the core idea behind a decision tree. This algorithm mimics exactly that kind of structured, hierarchical reasoning, and it does so entirely from data.
Decision trees are among the most widely used machine learning algorithms for one simple reason: you can read and explain every decision the model makes. In healthcare, finance, and legal settings where regulators demand explainability, a decision tree is often the only acceptable model. Even in contexts where accuracy matters more than transparency, decision trees form the backbone of powerful ensemble methods like Random Forest and XGBoost.
This guide explains how decision trees actually work under the hood, walks through a complete Python implementation, and addresses the questions that trip up most beginners — including the overfitting problem that every decision tree eventually runs into.
What Is a Decision Tree?
A decision tree is a predictive model that maps observations about a dataset to conclusions about a target value through a series of binary questions. Each internal node of the tree represents a question about a feature (“Is age greater than 35?”). Each branch coming from that node represents an answer. Each leaf node at the bottom of the tree represents a final prediction — a class label in classification problems, or a numeric value in regression problems.
Consider a bank trying to decide whether to approve a loan application. A human loan officer might mentally work through questions: Does the applicant have a credit score above 700? Have they defaulted on a previous loan? Is their debt-to-income ratio below 40%? A decision tree learns to ask the same questions — automatically, from historical data — and arranges them in the order that gives the most accurate predictions.
What makes this algorithm special is full transparency. After training, you can print the tree and read every rule it learned. You can show it to a compliance officer, a doctor, or a court. This is fundamentally different from neural networks or gradient boosting, which are effective but essentially opaque.
How Does a Decision Tree Split Data?
The algorithm builds the tree top-down by repeatedly choosing the feature and threshold that creates the “purest” split — the one that separates classes most cleanly. Purity is measured with one of two metrics.
Gini Impurity asks: if we randomly picked a sample from this node and randomly assigned it a class based on the distribution of classes here, how often would we be wrong? A node where 90% of samples belong to one class has low Gini impurity — it’s nearly pure. A node split 50/50 between two classes has high impurity — it’s completely uncertain. The algorithm picks the split that minimises Gini impurity across the resulting child nodes.
Information Gain approaches the same problem through entropy — a measure of uncertainty borrowed from information theory. It quantifies how much a given split reduces uncertainty about the target class. Both methods tend to produce very similar trees in practice. Gini is the default in scikit-learn and is slightly faster to compute.
This splitting process continues recursively at every node until one of the stopping criteria is met: maximum depth is reached, a node has too few samples to split further, or further splits would not improve purity meaningfully.
The Overfitting Problem
Left completely unconstrained, a decision tree will keep splitting until every leaf contains exactly one training sample. This achieves perfect training accuracy — and completely fails on new data. The tree has memorised noise rather than learned patterns.
Imagine a tree that learns: “If the customer is a 34-year-old female from Mumbai who applied on a Tuesday in March with exactly ₹47,200 in their account, approve the loan.” That rule applies to perhaps one person in the training set. It will never generalise.
The solution is controlled tree growth through hyperparameters. max_depth sets a hard limit on tree levels. min_samples_leaf ensures every leaf node contains at least N samples — preventing the tree from creating rules that apply to only a handful of people. min_impurity_decrease stops splits that don’t meaningfully reduce impurity. Using all three together gives a well-regularised tree.
Building a Decision Tree in Python
from sklearn.tree import DecisionTreeClassifier, plot_tree, export_text
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report
import matplotlib.pyplot as plt
import numpy as np
# Load dataset
data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names
class_names = data.target_names
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
# Train — with regularisation to prevent overfitting
tree = DecisionTreeClassifier(
max_depth=5,
min_samples_leaf=10,
criterion='gini',
random_state=42
)
tree.fit(X_train, y_train)
# Evaluate
print(classification_report(y_test, tree.predict(X_test), target_names=class_names))
print(f'Training accuracy: {tree.score(X_train, y_train):.4f}')
print(f'Test accuracy: {tree.score(X_test, y_test):.4f}')
After training, you can read every learned rule in plain text — invaluable for stakeholder explanations:
print(export_text(tree, feature_names=list(feature_names)))
And visualise the full tree:
plt.figure(figsize=(20, 10))
plot_tree(tree, feature_names=feature_names,
class_names=class_names,
filled=True, rounded=True, fontsize=8)
plt.title('Decision Tree — Breast Cancer Dataset', fontsize=14)
plt.tight_layout()
plt.show()
Finding the Optimal Depth
Rather than guessing the right depth, use cross-validation to let the data tell you:
depths = range(1, 20)
train_scores = []
cv_scores = []
for d in depths:
clf = DecisionTreeClassifier(max_depth=d, min_samples_leaf=5, random_state=42)
clf.fit(X_train, y_train)
train_scores.append(clf.score(X_train, y_train))
cv_scores.append(cross_val_score(clf, X_train, y_train, cv=5).mean())
best_depth = depths[np.argmax(cv_scores)]
print(f'Best depth: {best_depth} | CV accuracy: {max(cv_scores):.4f}')
plt.figure(figsize=(9, 5))
plt.plot(depths, train_scores, label='Training accuracy', linestyle='--')
plt.plot(depths, cv_scores, label='CV accuracy (5-fold)')
plt.axvline(best_depth, color='red', linestyle=':', label=f'Best depth = {best_depth}')
plt.xlabel('max_depth'); plt.ylabel('Accuracy')
plt.title('Overfitting Curve — Decision Tree Depth')
plt.legend()
plt.show()
The training accuracy will keep rising as depth increases. The cross-validation accuracy will rise, plateau, then often fall — that plateau is your sweet spot.
Feature Importance
One of the most useful outputs of a decision tree is feature importance — which features were actually used for splitting, and how much they contributed to reducing impurity:
import pandas as pd
importances = pd.Series(tree.feature_importances_, index=feature_names)
importances.sort_values(ascending=True).tail(10).plot(kind='barh', figsize=(8, 5))
plt.title('Top 10 Most Important Features')
plt.xlabel('Importance (Gini-based)')
plt.tight_layout()
plt.show()
When to Use a Decision Tree
Decision trees are the right choice when interpretability is not optional. If a doctor needs to explain why a diagnosis was made, a loan officer needs to justify a rejection, or an auditor needs to verify that no protected characteristics influenced a decision — only a decision tree gives you that level of transparency.
They are a poor choice when raw predictive accuracy is the primary goal. A single decision tree almost always loses to Random Forest or XGBoost on accuracy benchmarks because those methods address the tree’s core weakness: high variance. A small change in training data can produce a very different tree, making single trees unreliable in production. Use ensembles for accuracy, single trees for interpretability.
Frequently Asked Questions
Does a decision tree need feature scaling?
No, and this is one of its significant practical advantages. Decision trees split on thresholds (“Is feature X greater than value V?”), so the absolute scale of a feature has no effect on which splits are chosen. You can mix features measured in rupees, kilometres, percentages, and counts in the same tree without any preprocessing. This is fundamentally different from distance-based algorithms like SVM or K-Nearest Neighbours, which require all features to be on the same scale to work correctly.
How do I handle missing values in a decision tree?
Scikit-learn’s DecisionTreeClassifier does not natively handle missing values — you will get an error if your data contains NaN. The recommended approach is to impute missing values before fitting: use SimpleImputer(strategy='mean') for numeric features and SimpleImputer(strategy='most_frequent') for categorical ones. Alternatively, XGBoost and LightGBM implement internal missing value handling, making them better choices when your dataset has significant missing data.
Decision Tree vs Random Forest — when should I use each?
Use a decision tree when you need to explain your model to non-technical stakeholders, when you are in a regulated industry (finance, healthcare, insurance), or when you need a fast baseline to understand your data. Use Random Forest when accuracy matters more than interpretability — Random Forest fixes the high-variance problem of single trees by averaging hundreds of them, consistently outperforming a single tree on most datasets. As a rule of thumb: prototype with a decision tree to understand your data, then switch to Random Forest or XGBoost for production.
What is the difference between max_depth and min_samples_leaf?
They control overfitting in complementary ways. max_depth is a hard ceiling — the tree can never be more than N levels deep, regardless of how much further splitting might improve purity. min_samples_leaf works from the bottom up: it prevents any leaf node from being created with fewer than N samples. A leaf with 2 samples is almost certainly memorising noise rather than learning a pattern. In practice, use both together: start with max_depth=5 and min_samples_leaf=10 as defaults, then tune with cross-validation.
Summary
Decision trees learn by recursively splitting data on the features that most cleanly separate classes. They are fully interpretable, require no feature scaling, and form the building block of the most powerful ensemble methods in machine learning. Their main weakness is high variance and overfitting when unconstrained — always regularise with max_depth and min_samples_leaf, and use cross-validation to find the right settings for your data.
