Explainable AI (XAI) – SHAP, LIME & Model Interpretability Guide 2026

Black-box models achieve great accuracy, but accuracy alone is not enough in regulated industries like finance, healthcare, and insurance. Explainable AI (XAI) bridges the gap between model performance and human understanding. SHAP and LIME are the two most widely adopted explanation frameworks in production data science. This guide shows you how to use both.

Why Explainability Matters

Regulatory requirements (EU AI Act, GDPR Article 22) require explanations for automated decisions affecting people. Beyond compliance, explainability helps data scientists debug models, catch data leakage, build stakeholder trust, and identify when a model is making predictions for the wrong reasons. A model that achieves 95% accuracy because it accidentally learned a spurious correlation is worse than useless — it gives false confidence.

Setting Up

pip install shap lime scikit-learn xgboost pandas matplotlib

import pandas as pd
import numpy as np
import shap
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

# Load sample dataset
data    = load_breast_cancer(as_frame=True)
X, y    = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

# Train XGBoost classifier
model = xgb.XGBClassifier(n_estimators=100, max_depth=4,
                           use_label_encoder=False,
                           eval_metric='logloss', random_state=42)
model.fit(X_train, y_train)
print(f'Test accuracy: {model.score(X_test, y_test):.4f}')

SHAP – Global and Local Explanations

SHAP (SHapley Additive exPlanations) assigns each feature a contribution score for each prediction based on cooperative game theory. SHAP values are consistent, locally accurate, and model-agnostic. They are the gold standard for XAI in production.

import shap
import matplotlib.pyplot as plt

# Create SHAP explainer (TreeExplainer is fast for tree-based models)
explainer   = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# ── Global: which features matter most overall? ──────────────
shap.summary_plot(shap_values, X_test, plot_type='bar',
                  max_display=15, show=True)

# ── Global: direction of effect ──────────────────────────────
shap.summary_plot(shap_values, X_test, max_display=15, show=True)

# ── Local: explain a single prediction ───────────────────────
idx = 5   # explain prediction for test sample 5
shap.force_plot(explainer.expected_value,
                shap_values[idx],
                X_test.iloc[idx],
                matplotlib=True)

# ── Waterfall plot for one prediction ────────────────────────
shap.waterfall_plot(shap.Explanation(
    values=shap_values[idx],
    base_values=explainer.expected_value,
    data=X_test.iloc[idx],
    feature_names=X_test.columns.tolist()
))

# ── Dependence plot: how one feature interacts with another ──
shap.dependence_plot('worst radius', shap_values, X_test,
                     interaction_index='mean concave points')

LIME – Local Surrogate Explanations

LIME (Local Interpretable Model-agnostic Explanations) explains individual predictions by fitting a simple linear model around each data point. It works with any black-box model including neural networks and is useful when you need explanations for non-tree-based models.

from lime import lime_tabular

# Create LIME explainer
lime_explainer = lime_tabular.LimeTabularExplainer(
    training_data=X_train.values,
    feature_names=X_train.columns.tolist(),
    class_names=['malignant', 'benign'],
    mode='classification',
    discretize_continuous=True
)

# Explain a single test instance
idx         = 10
instance    = X_test.iloc[idx].values
explanation = lime_explainer.explain_instance(
    data_row=instance,
    predict_fn=model.predict_proba,
    num_features=10,
    num_samples=1000
)

explanation.show_in_notebook(show_table=True)
explanation.as_pyplot_figure()
plt.tight_layout()
plt.show()

# Get feature weights as a list
for feat, weight in explanation.as_list():
    print(f'{feat}: {weight:.4f}')

Partial Dependence Plots

from sklearn.inspection import PartialDependenceDisplay

fig, ax = plt.subplots(figsize=(12, 5))
PartialDependenceDisplay.from_estimator(
    model, X_test,
    features=['worst radius', 'worst perimeter',
              ('worst radius', 'mean concave points')],  # 2D interaction
    ax=ax,
    kind='average'
)
plt.tight_layout()
plt.show()

Explaining to Stakeholders

Technical SHAP plots are not enough for non-technical stakeholders. Convert SHAP values into plain-English summaries programmatically.

def explain_in_plain_english(shap_vals, feature_names, top_n=3):
    pairs = sorted(zip(feature_names, shap_vals),
                   key=lambda x: abs(x[1]), reverse=True)[:top_n]
    lines = []
    for feat, val in pairs:
        direction = 'increased' if val > 0 else 'decreased'
        lines.append(f'{feat} {direction} the prediction by {abs(val):.3f}')
    return 'Top factors: ' + '; '.join(lines)

idx = 5
print(explain_in_plain_english(shap_values[idx],
                               X_test.columns.tolist()))

SHAP vs LIME — When to Use Which

Use SHAP when you have tree-based models (XGBoost, LightGBM, Random Forest) — TreeExplainer makes it very fast. SHAP values are theoretically grounded and globally consistent. Use LIME when you have neural networks, deep learning models, or any model where SHAP is too slow. LIME is faster for single predictions on complex models. In practice, use SHAP as your primary tool and LIME as a sanity check or fallback.

Conclusion

Explainable AI is no longer optional — it is a professional requirement in most industries deploying ML. SHAP gives you mathematically rigorous, globally consistent explanations that scale from individual predictions to whole-model analysis. LIME complements it with fast, model-agnostic local explanations. Together they give you the full picture: what your model learned, why it makes specific predictions, and how to communicate both to people who need to trust your work.