📋 KEY INSIGHTS
- Streaming machine learning extends the standard batch ML paradigm to settings where data arrives continuously and model predictions or features must be available with low latency — seconds or milliseconds rather than hours or days. Real-time fraud detection, dynamic pricing, live recommendation updates, and industrial process control all require streaming ML infrastructure.
- The core challenge of streaming ML is feature engineering under time constraints: features that are trivial to compute in batch (rolling averages over the past 30 days, aggregates over user history) require stateful stream processing systems (Apache Flink, Apache Kafka Streams) to compute incrementally as each event arrives. A point-in-time correct rolling average must be updated with each new event and must reflect exactly the data available at the time of the prediction.
- Online learning algorithms — SGD with constant learning rate, FTRL, Vowpal Wabbit — update model weights with each arriving data point, enabling the model to adapt to distribution shifts in real time. The tradeoff is stability: batch retraining on a fixed window produces a stable, well-evaluated model; online learning produces a continuously adapting model that may drift in unexpected directions and is harder to monitor.
- The two-tower architecture for real-time recommendation systems separates query (user) and item encoding into two independent neural networks, enabling the item tower outputs to be pre-computed and cached in a vector database while only the user tower runs at query time. This architecture achieves sub-millisecond item retrieval for billions of items and is the standard architecture for real-time recommendation at YouTube, TikTok, and Pinterest.
- Feature stores (Feast, Tecton, Hopsworks) bridge the batch and streaming worlds: they provide a unified API for writing features from both batch pipelines (dbt transformations, Spark jobs) and streaming pipelines (Flink, Kafka Streams), and serving those features with consistent point-in-time semantics for both training and online inference. This eliminates training-serving skew — the most common production ML reliability failure.
- Monitoring streaming ML systems requires different tooling than batch monitoring: rather than evaluating on a held-out test set periodically, streaming systems monitor concept drift (changes in the relationship between features and labels), data drift (changes in the input feature distribution), and prediction quality (via delayed labels or proxy metrics) in real time, triggering automated retraining or human review when anomalies are detected.
Most ML production systems are batch systems: features are computed overnight, models are retrained weekly, and predictions are generated in bulk and served from a cache. Batch ML is straightforward to build, easy to debug, and sufficient for applications where decisions do not need to be updated faster than the batch cycle. But a significant and growing class of applications requires something faster: a credit card fraud decision must be made in 200ms at transaction time; a content recommendation must reflect the user’s behaviour from two minutes ago, not two days ago; a dynamic pricing system must respond to real-time demand signals, not yesterday’s averages. These applications require streaming ML — machine learning pipelines that ingest, process, and act on data streams with low latency. This guide covers the streaming ML stack: stream processing with Kafka and Flink, streaming feature engineering and feature stores, online learning algorithms, real-time recommendation architectures, and the monitoring infrastructure for streaming systems.
Stream Processing with Kafka and Flink
The streaming ML infrastructure stack has two layers: the message broker layer (Apache Kafka) that durably stores and distributes event streams, and the stream processing layer (Apache Flink, Apache Kafka Streams, Apache Spark Structured Streaming) that applies stateful computations to those streams. Kafka is the de facto standard message broker for ML event data: events (user clicks, transactions, sensor readings, API calls) are published to Kafka topics by producer services, and ML consumers (feature computation jobs, inference services) subscribe to those topics and process events with at-least-once or exactly-once delivery guarantees. Kafka’s log-based architecture means that events are retained for a configurable period (days to weeks) and can be re-processed from any point in history — enabling historical replay for feature backfilling and model training on the exact event stream that production saw. The connection to our Big Data Technologies guide is direct: Kafka is covered there alongside Spark and Hadoop in the context of distributed data infrastructure.
Apache Flink is the strongest open-source stream processor for stateful ML feature computation. Flink’s key features for streaming ML: (1) Exactly-once state semantics via distributed snapshots (Chandy-Lamport algorithm), ensuring that stateful aggregations are consistent even after failures and restarts; (2) Event-time processing with watermarks — Flink tracks the progress of event time (the timestamp embedded in events) separately from processing time (wall-clock time), correctly handling out-of-order events that arrive late due to network delays; (3) Rich stateful operators — tumbling windows, sliding windows, session windows, and arbitrary keyed state — for computing rolling aggregates, session features, and user-level statistics; (4) Backpressure handling that automatically throttles fast producers when downstream operators are slower, preventing memory overflow under load spikes. The connection to feature engineering is central: every feature that a batch ML pipeline computes in SQL or Spark must be re-implemented in Flink as a streaming operator for real-time serving, and the feature store is the layer that keeps these two implementations consistent.
| Layer | Tools | Role in Streaming ML | Key Concepts |
|---|---|---|---|
| Message broker | Apache Kafka, AWS Kinesis, Pulsar | Durable event log; decouple producers/consumers | Topics, partitions, consumer groups, offsets |
| Stream processor | Apache Flink, Kafka Streams, Spark Streaming | Stateful feature computation on streams | Windows, watermarks, exactly-once state |
| Feature store | Feast, Tecton, Hopsworks, Redis | Serve pre-computed features at inference time | Point-in-time correct, batch+stream unification |
| Inference serving | Seldon, BentoML, Triton, vLLM | Low-latency model predictions | gRPC, batching, model versioning |
| Online learning | River, Vowpal Wabbit, custom SGD | Adapt model to stream in real time | FTRL, concept drift, incremental update |
| Monitoring | Evidently AI, Arize, Grafana | Detect drift; alert on prediction degradation | PSI, KL divergence, delayed label evaluation |
Feature Stores — Bridging Batch and Streaming Features
Training-serving skew is the most pervasive production ML failure: the feature values used during model training differ from the feature values computed at inference time, because training uses a batch computation (dbt SQL, Spark) and serving uses a real-time computation (Python function, Flink operator), and small implementation differences accumulate into significant prediction quality degradation. Feature stores are the engineering solution: they provide a unified feature registry and serving layer where features are defined once, computed by both batch and streaming pipelines using consistent logic, and served to both training (point-in-time correct historical features) and inference (low-latency current features) through the same API.
The feature store architecture: an offline store (typically a data lakehouse table — Delta Lake or Iceberg in a Bronze-Silver-Gold architecture) that holds the full historical feature time series, used for training dataset generation with point-in-time correct joins; an online store (Redis, DynamoDB, or Cassandra) that holds only the latest feature value per entity (user, item, transaction), served with sub-millisecond latency to the inference service. The streaming pipeline writes to both stores simultaneously: each event updates the latest value in the online store and appends a historical record to the offline store. The connection to our ETL pipelines guide and feature engineering guide is direct: feature stores are the operational layer that makes the feature engineering work done by data scientists available to production ML systems reliably and at scale. The MLflow experiment tracking integration with feature stores allows each training run to record exactly which feature version and snapshot were used, creating a complete audit trail from raw events to deployed model.
Concept Drift Detection and Streaming Monitoring
Concept drift — changes in the statistical relationship between features and labels over time — is the primary failure mode for production ML systems. A fraud detection model trained on Q1 fraud patterns will degrade as fraudsters adapt their tactics in Q2. A demand forecasting model will fail when a macroeconomic shock changes purchasing behaviour. Streaming monitoring systems detect this degradation before it becomes a business problem. The three types of drift correspond to three monitoring signals: data drift (change in P(X)) is detected by comparing feature distributions between the current window and the training set using PSI (Population Stability Index) or KL divergence — covered in our statistics fundamentals guide; concept drift (change in P(Y|X)) is detected by tracking prediction quality on delayed labels (labels that arrive hours or days after the prediction) or proxy metrics that correlate with ground-truth quality; prediction drift (change in P(Ŷ)) is a leading indicator that can be detected without labels, by monitoring the distribution of predicted scores over time. The connection to data quality monitoring is direct: the same data contract validation approach used for batch data quality applies to streaming feature validation, and anomalies in feature distributions trigger the same alerting and remediation workflows.
✦ SUMMARIZE THIS ARTICLE WITH AI
ChatGPTKafka, Spark Streaming, and the distributed infrastructure underlying streaming ML are in our Big Data Technologies guide. Feature engineering — the batch features that streaming pipelines must replicate in real time — is in our Feature Engineering guide. Modern data architecture — the lakehouse offline store for feature history — is in our Modern Data Architecture guide. Online learning and multi-armed bandits for real-time model adaptation are in our Multi-Armed Bandits guide. MLflow experiment tracking for streaming ML model versioning is in our MLflow guide. Model deployment — inference serving infrastructure that streaming features feed into — is in our Model Deployment guide.
