Machine Learning Model Deployment – REST APIs, Docker and Kubernetes

The gap between a trained model in a Jupyter notebook and a model delivering value in production is where most ML projects stall. The modelling work — data exploration, feature engineering, training, evaluation — feels like the substance of the project. But a model that lives in a notebook is an interesting experiment, not a product. Getting from experiment to production requires understanding REST API design, containerisation with Docker, orchestration with Kubernetes, deployment strategies, monitoring, and rollback. These are engineering skills that data scientists increasingly need to own, not delegate. This guide covers the full deployment stack for ML models — from wrapping a model in an API to monitoring it in production — with architecture diagrams in table form and practical decision guidance at each stage.

The ML Serving Stack — Components and Responsibilities

A production ML serving system has five layers, each with distinct responsibilities. Understanding what each layer does and where failures can occur is the foundation of reliable deployment.

Layer 1 — The Model Artefact: The serialised, versioned representation of the trained model. For scikit-learn and LightGBM models, this is a pickle file or joblib file. For PyTorch models, it is a state_dict checkpoint or a TorchScript-compiled module. For TensorFlow/Keras, it is a SavedModel directory. The best practice is to export to ONNX format, which decouples the model from the training framework and enables serving with optimised runtimes. Alongside the model artefact, the serving system needs the preprocessing pipeline — the exact transformations (scaling, encoding, imputation) applied to training data — serialised with the same library version used during training.

Layer 2 — The Prediction Service: A web service that receives prediction requests, applies the preprocessing pipeline, runs model inference, post-processes outputs, and returns predictions. FastAPI (Python) is the most popular framework for building prediction services — it provides automatic request validation via Pydantic schemas, async support for concurrent requests, and auto-generated OpenAPI documentation. The prediction service should validate inputs (reject requests with missing required features or out-of-range values), log every prediction with its inputs and outputs (essential for monitoring and debugging), and return structured responses with prediction confidence alongside the primary prediction.

Layer 3 — The Container: Docker packages the prediction service and all its dependencies (Python version, library versions, system libraries) into a portable image that runs identically across development, staging, and production environments. The key practices are: use a minimal base image (python:3.11-slim rather than a full OS image) to reduce image size and attack surface; pin all dependency versions in requirements.txt to ensure reproducibility; and implement a multi-stage build that separates the build environment from the runtime image, further reducing the final image size.

Layer 4 — The Orchestrator: Kubernetes manages the lifecycle of containerised services at scale — scheduling containers onto cluster nodes, restarting failed pods, scaling the number of replicas in response to traffic, performing rolling updates, and routing traffic. For most ML serving use cases, a Deployment resource (managing a set of identical prediction service pods) behind a Service resource (providing a stable internal IP) behind an Ingress resource (routing external traffic) is the standard pattern. Horizontal Pod Autoscaler (HPA) automatically adjusts replica count based on CPU utilisation or custom metrics (requests per second, queue depth), enabling the serving infrastructure to absorb traffic spikes without manual intervention.

Layer 5 — The Monitoring System: Prometheus (metrics collection) + Grafana (visualisation) is the standard open-source monitoring stack for containerised services. Metrics to track divide into two categories: infrastructure metrics (CPU utilisation, memory, request latency, error rate, pod restarts) and model metrics (prediction distribution, feature distribution, output confidence distribution, and business-level KPIs). Model metric monitoring is the more critical and more neglected layer — a model can be serving requests with low latency while its prediction distribution has drifted significantly from the distribution seen during validation, silently delivering degraded business value.

Deployment Strategies — Reducing Risk for New Model Versions

Releasing a new model version to 100% of production traffic immediately is the highest-risk deployment strategy. A degraded model does not throw an exception — it continues serving predictions silently, making it possible for a bad model to run in production for hours or days before the business impact becomes visible. Safer strategies manage the rollout progressively.

Blue-Green Deployment: Two identical production environments are maintained — blue (the current live version) and green (the new version). The new model is deployed and validated on the green environment with synthetic or replayed traffic before any live traffic is switched. The load balancer then switches all traffic from blue to green in a single step. Rollback is instantaneous — switch back to blue. The cost is maintaining two full production environments simultaneously, which doubles infrastructure cost during the transition period.

Canary Deployment: The new model version receives a small percentage of live traffic (typically 1–5%) while the current version continues serving the remainder. The canary is monitored against the same business metrics as the current version. If the canary performs equivalently or better over a predefined evaluation window, the traffic percentage is gradually increased (5% → 20% → 50% → 100%). If the canary underperforms, traffic is rolled back to zero. Canary deployment requires a traffic splitting mechanism (feature flags, A/B routing in the load balancer, or a service mesh like Istio) and automated monitoring with alerting rules that trigger rollback when metrics fall below thresholds.

Shadow Mode: The new model receives a copy of all live traffic and generates predictions, but those predictions are not served to users — they are logged and compared to the current model’s predictions. Shadow mode is invaluable for validating that a new model’s prediction distribution matches expectations on live data before exposing it to users. It is also the safest way to validate a model that produces consequential outputs (medical diagnoses, financial decisions) where even a brief period of degraded performance would be unacceptable.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

The MLOps practices that surround deployment — CI/CD pipelines for models, feature stores, and experiment tracking — are covered in our MLOps Interview Q&A. ML system design patterns at the architecture level — online vs batch serving, feature pipelines, model registries — are in our ML System Design guide. Cloud infrastructure for running Kubernetes clusters (EKS, GKE, AKS) is covered in our Cloud for Data Scientists guide. Building lightweight prediction UIs on top of deployed model APIs is covered in our Streamlit Deployment guide.