ML Model Monitoring in Production – Drift Detection Guide 2026

A model that performs well at deployment can silently degrade over months as the world changes. Users’ behaviour shifts, economic conditions change, data pipelines break. Without monitoring, you won’t know until a business metric drops or someone files a bug report. This guide covers everything you need to monitor ML models reliably in production in 2026.

Why Models Degrade in Production

There are two root causes of model degradation. Data drift occurs when the distribution of input features changes — for example, a fraud detection model trained on 2024 transactions may underperform in 2026 as fraud patterns evolve. Concept drift occurs when the relationship between features and the target changes — the same inputs now map to different correct outputs. Both are invisible without monitoring.

What to Monitor

A complete monitoring strategy covers four layers. Data quality monitoring catches upstream problems — null rates, out-of-range values, schema changes. Feature drift monitoring detects when input distributions shift from the training baseline. Prediction drift monitoring watches the distribution of model outputs. Performance monitoring tracks business metrics and (if labels are available) accuracy metrics like AUC and F1.

Setting Up Evidently for Drift Detection

pip install evidently

import pandas as pd
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, DataQualityPreset
from evidently.metrics import ColumnDriftMetric

# Reference = training data, current = recent production data
reference_df = pd.read_parquet("data/train_features.parquet")
current_df   = pd.read_parquet("data/production_week_33.parquet")

# Full data drift report
report = Report(metrics=[
    DataDriftPreset(),
    DataQualityPreset(),
])
report.run(reference_data=reference_df, current_data=current_df)
report.save_html("drift_report_week33.html")

# Get drift results programmatically
result = report.as_dict()
drift_score = result['metrics'][0]['result']['dataset_drift']
print(f"Dataset drift detected: {drift_score}")

Statistical Tests for Drift

from scipy import stats
import numpy as np

def detect_drift(reference, current, threshold=0.05):
    '''Returns True if drift detected (KS test p-value < threshold).'''
    stat, p_value = stats.ks_2samp(reference, current)
    drift_detected = p_value < threshold
    return {
        'ks_statistic': round(stat, 4),
        'p_value': round(p_value, 6),
        'drift_detected': drift_detected
    }

for col in feature_cols:
    result = detect_drift(reference_df[col].dropna(), current_df[col].dropna())
    if result['drift_detected']:
        print(f"⚠️  DRIFT in {col}: KS={result['ks_statistic']}, p={result['p_value']}")

Monitoring Prediction Distribution

import matplotlib.pyplot as plt

# Compare prediction score distributions
ref_scores  = model.predict_proba(reference_df)[:, 1]
prod_scores = model.predict_proba(current_df)[:, 1]

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.hist(ref_scores,  bins=50, alpha=0.7, label='Training')
ax1.hist(prod_scores, bins=50, alpha=0.7, label='Production')
ax1.set_title("Prediction Score Distribution")
ax1.legend()

# Population Stability Index (PSI)
def compute_psi(expected, actual, bins=10):
    exp_pct = np.histogram(expected, bins=bins)[0] / len(expected) + 1e-6
    act_pct = np.histogram(actual,   bins=bins)[0] / len(actual)   + 1e-6
    psi = np.sum((act_pct - exp_pct) * np.log(act_pct / exp_pct))
    return psi

psi = compute_psi(ref_scores, prod_scores)
print(f"PSI: {psi:.4f}")  # <0.1: no drift, 0.1-0.25: moderate, >0.25: significant

Alerting on Drift

import requests

def send_slack_alert(message: str, webhook_url: str):
    requests.post(webhook_url, json={"text": message})

def weekly_drift_check(reference_df, current_df, model, webhook_url):
    report = Report(metrics=[DataDriftPreset()])
    report.run(reference_data=reference_df, current_data=current_df)
    result = report.as_dict()

    drifted_cols = [
        m['result']['column_name']
        for m in result['metrics']
        if m.get('result', {}).get('drift_detected')
    ]

    if drifted_cols:
        send_slack_alert(
            f"🚨 Model drift alert: {len(drifted_cols)} features drifted — {drifted_cols}",
            webhook_url)
    else:
        print("✅ No significant drift detected")

Shadow Mode Deployment

Before replacing a model in production, run the new model in shadow mode alongside the current model. Both receive all requests, but only the old model's predictions are served to users. Log both sets of predictions and compare them. When the new model's performance is confirmed superior (and safe), switch over. This pattern eliminates the risk of deploying a regression.

Building a Monitoring Dashboard

For a simple monitoring dashboard, combine Evidently reports with a scheduled Airflow DAG: run the drift check weekly, save the HTML report to S3, and send a Slack summary. For more sophisticated needs, Arize AI, WhyLabs, and Fiddler offer managed monitoring platforms with real-time alerting, automatic retraining triggers, and fairness monitoring. Most data teams start with the DIY Evidently approach and migrate to a managed platform as scale demands it.

Conclusion

Model monitoring is not optional in production ML systems — it's what separates a science project from a reliable product. Start with data quality checks (nulls, range violations), add a weekly drift check using Evidently or KS tests, and monitor your prediction distribution. Set up Slack alerts so you hear about problems before users do. The entire monitoring stack can be operational in an afternoon and will save you from the silent degradation that kills ML models in the wild.