Model Interpretability – SHAP, LIME and Feature Importance Explained

Machine learning models are increasingly used in high-stakes decisions — loan approvals, medical diagnoses, fraud detection, and hiring — where “the model said so” is not an acceptable justification. Regulators require explanations, product teams need to debug unexpected behaviour, and data scientists must verify that models are learning the right patterns rather than spurious correlations. Model interpretability is the set of techniques for understanding what a model has learned and why it makes specific predictions. This guide covers the most important methods: SHAP values (the gold standard), LIME (fast local approximations), partial dependence plots, and tree feature importance.

Feature Importance — Tree-Based Methods

Tree-based models (Random Forest, XGBoost, LightGBM) produce built-in feature importance scores. Three types: Split count: how many times a feature is used to split nodes across all trees. Gain (impurity-based): total reduction in impurity (Gini or entropy) attributable to the feature. Permutation importance: decrease in model performance when the feature values are randomly shuffled (model-agnostic, most reliable). Impurity-based importance is biased toward high-cardinality continuous features — permutation importance and SHAP are more trustworthy.

import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

# Load or create a dataset
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=5000, n_features=15, n_informative=8,
                            random_state=42)
feat_names = ['age','income','tenure','balance','transactions',
              'credit_score','missed_payments','products',
              'online_logins','branch_visits','complaints',
              'promo_clicks','referrals','region_code','account_type']
X_df = pd.DataFrame(X, columns=feat_names)
X_tr, X_te, y_tr, y_te = train_test_split(X_df, y, test_size=0.2, random_state=42)

rf = RandomForestClassifier(n_estimators=300, max_depth=8, random_state=42)
rf.fit(X_tr, y_tr)

# Built-in impurity-based importance
imp_df = pd.Series(rf.feature_importances_, index=feat_names).sort_values(ascending=False)
print('Impurity-based importance (top 5):')
print(imp_df.head())

# Permutation importance (more reliable, especially for correlated features)
perm_result = permutation_importance(rf, X_te, y_te, n_repeats=10,
                                     random_state=42, scoring='roc_auc')
perm_df = pd.DataFrame({'importance': perm_result.importances_mean,
                         'std': perm_result.importances_std},
                        index=feat_names).sort_values('importance', ascending=False)
print('
Permutation importance (top 5):')
print(perm_df.head())

# Plot
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
imp_df.plot.bar(ax=axes[0], color='steelblue'); axes[0].set_title('Impurity-based')
perm_df['importance'].plot.bar(ax=axes[1], color='coral'); axes[1].set_title('Permutation')
for ax in axes: ax.tick_params(axis='x', rotation=45)
plt.tight_layout(); plt.show()

SHAP Values — Theoretically Grounded Attributions

SHAP (SHapley Additive exPlanations) is based on Shapley values from cooperative game theory. For a prediction, the Shapley value for feature i is its average marginal contribution across all possible subsets of features — it is the only attribution method that satisfies axioms of Efficiency (attributions sum to prediction), Symmetry (equal contribution = equal attribution), Dummy (unused features get 0), and Additivity. TreeSHAP computes exact Shapley values for tree ensembles in polynomial time, making it practical for production use.

import shap

# TreeSHAP for tree models — fast and exact
explainer  = shap.TreeExplainer(rf)
shap_vals  = explainer.shap_values(X_te)   # shape: (n_samples, n_features, n_classes)
# For binary classification, shap_vals[1] = SHAP for class 1
sv_class1  = shap_vals[1]

# --- Global: Summary plot (beeswarm) ---
shap.summary_plot(sv_class1, X_te, plot_type='dot', show=True)
# Violin version — shows distribution
shap.summary_plot(sv_class1, X_te, plot_type='violin', show=True)

# --- Global: Bar plot of mean |SHAP| ---
shap.summary_plot(sv_class1, X_te, plot_type='bar', show=True)

# --- Local: Force plot for a single prediction ---
idx = 0
print('Prediction:', rf.predict_proba(X_te.iloc[[idx]])[0, 1].round(3))
shap.force_plot(explainer.expected_value[1],
                sv_class1[idx], X_te.iloc[idx],
                matplotlib=True)

# --- Local: Waterfall plot (cleaner single-prediction view) ---
shap_explanation = shap.Explanation(
    values=sv_class1[idx],
    base_values=explainer.expected_value[1],
    data=X_te.iloc[idx].values,
    feature_names=feat_names
)
shap.plots.waterfall(shap_explanation)

# --- Dependence plot: how does one feature affect predictions? ---
shap.dependence_plot('income', sv_class1, X_te,
                     interaction_index='credit_score', show=True)

LIME — Local Approximations

LIME (Local Interpretable Model-agnostic Explanations) works by perturbing a single instance, getting model predictions for the perturbed samples, weighting samples by proximity to the original instance, and fitting a simple linear model to the weighted samples. The linear model’s coefficients are the “explanation”. LIME is model-agnostic (works on any black-box model) and fast, but the explanations can be unstable across runs and depend heavily on hyperparameters (number of samples, kernel width). For tabular data, prefer SHAP when you have a tree model; use LIME for neural networks or when you need very fast local explanations.

from lime.lime_tabular import LimeTabularExplainer

explainer_lime = LimeTabularExplainer(
    training_data = X_tr.values,
    feature_names  = feat_names,
    class_names    = ['No Churn', 'Churn'],
    mode           = 'classification',
    random_state   = 42
)

# Explain a single prediction
instance = X_te.iloc[5].values
exp = explainer_lime.explain_instance(
    data_row    = instance,
    predict_fn  = rf.predict_proba,
    num_features = 10,
    num_samples  = 1000
)
exp.show_in_notebook()

# Get explanation as a list of (feature, weight) tuples
for feat, weight in exp.as_list():
    direction = 'increases' if weight > 0 else 'decreases'
    print(feat, direction, 'churn probability by', abs(round(weight, 4)))

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

For how interpretability connects to feature selection and model debugging, our Feature Selection guide covers using SHAP values to select features. The gradient boosting models where SHAP is most commonly applied are explained in our Gradient Boosting Deep Dive. Model evaluation beyond accuracy — bias detection, calibration — is in our Model Evaluation guide. ML system design interview questions on fairness and explainability are covered in our ML System Design guide.