Kubernetes for Data Scientists – Deploy ML Models at Scale 2026

Getting a model to 90% accuracy is the fun part. Keeping it running reliably under production traffic — that is where Kubernetes comes in. K8s is the industry standard for deploying, scaling, and managing containerized ML workloads. This guide gives data scientists the practical K8s knowledge needed to move models from notebook to production.

Why Kubernetes for ML?

ML models have unique deployment challenges: they are compute-heavy, they need GPU access, traffic is unpredictable, and model versions change frequently. Kubernetes solves all of these with auto-scaling, GPU node pools, rolling updates with zero downtime, and declarative configuration. Once you understand K8s basics, deploying a new model version is a single command.

Containerise Your Model First

# Dockerfile for a FastAPI ML model
FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY model/ ./model/
COPY app.py .

EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

# app.py — FastAPI serving a scikit-learn model
import joblib
from fastapi import FastAPI
from pydantic import BaseModel
import numpy as np

app   = FastAPI()
model = joblib.load('model/clf.pkl')

class Features(BaseModel):
    features: list[float]

@app.post('/predict')
def predict(data: Features):
    arr  = np.array(data.features).reshape(1, -1)
    pred = model.predict(arr)[0]
    prob = model.predict_proba(arr)[0].max()
    return {'prediction': int(pred), 'confidence': round(float(prob), 4)}

@app.get('/health')
def health():
    return {'status': 'ok'}

# Build and push to Docker Hub or GCR
docker build -t your-dockerhub/ml-model:v1 .
docker push your-dockerhub/ml-model:v1

Core Kubernetes Concepts

A Pod is the smallest deployable unit — one or more containers sharing network and storage. A Deployment manages a set of identical Pods, handling restarts and rolling updates. A Service exposes Pods to network traffic. A Namespace groups resources for isolation. ConfigMaps and Secrets store configuration and credentials separately from images.

Deployment YAML

# ml-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
  namespace: ml-prod
spec:
  replicas: 3                        # run 3 identical pods
  selector:
    matchLabels:
      app: ml-model
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1                    # one extra pod during update
      maxUnavailable: 0              # no downtime
  template:
    metadata:
      labels:
        app: ml-model
    spec:
      containers:
      - name: ml-model
        image: your-dockerhub/ml-model:v1
        ports:
        - containerPort: 8000
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        readinessProbe:              # don't send traffic until healthy
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:               # restart if unhealthy
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
        env:
        - name: MODEL_VERSION
          value: "v1"

# ml-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ml-model-service
  namespace: ml-prod
spec:
  selector:
    app: ml-model
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: LoadBalancer           # use ClusterIP for internal-only

# Apply both
kubectl apply -f ml-deployment.yaml
kubectl apply -f ml-service.yaml
kubectl get pods -n ml-prod
kubectl get service ml-model-service -n ml-prod

Auto-scaling with HPA

# Horizontal Pod Autoscaler — scale between 2 and 20 pods based on CPU
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
  namespace: ml-prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ml-model-deployment
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

kubectl apply -f hpa.yaml
kubectl get hpa -n ml-prod   # watch it scale under load

Rolling Model Updates

# Build and push new version
docker build -t your-dockerhub/ml-model:v2 .
docker push your-dockerhub/ml-model:v2

# Update deployment — zero downtime rolling update
kubectl set image deployment/ml-model-deployment   ml-model=your-dockerhub/ml-model:v2 -n ml-prod

# Watch the rollout
kubectl rollout status deployment/ml-model-deployment -n ml-prod

# Rollback if something breaks
kubectl rollout undo deployment/ml-model-deployment -n ml-prod

GPU Workloads

# Request GPU in your pod spec
resources:
  limits:
    nvidia.com/gpu: 1      # request 1 GPU

# Check GPU node availability
kubectl get nodes -l accelerator=nvidia-tesla-t4

Conclusion

Kubernetes transforms ML deployment from a fragile manual process into a reliable, scalable, auditable system. Start by containerising your model with Docker, write a simple Deployment and Service YAML, and use HPA for auto-scaling. The investment in learning K8s basics pays dividends across every model you deploy afterwards — update, rollback, and scale in seconds rather than hours.