Machine Learning System Design Interview – Framework, Examples and Common Questions

Machine learning system design interviews are the most differentiating component of senior data scientist and ML engineer hiring processes at top technology companies. Unlike coding interviews (where there is a correct answer) or ML theory questions (where knowledge can be crammed), system design requires integrating ML knowledge with engineering constraints, business requirements, and real-world tradeoffs under time pressure. This guide covers the complete framework, worked examples, and the evaluation criteria interviewers use at companies like Google, Meta, and Amazon.

ML system design draws on nearly every technical skill in this cluster: model training from our Machine Learning Interview Q&A, neural network choices from our Neural Network Architectures guide, model evaluation from our Model Evaluation guide, feature engineering from our Feature Engineering guide, data pipelines from our Data Engineering guide, and deployment from our MLOps Interview Q&A.

The RADIO Framework — How to Structure Your Answer

Experienced interviewers evaluate ML system design on five dimensions. A strong answer touches all five in roughly this order:

Always ask clarifying questions first: Spend 3–5 minutes asking about scale (how many users? QPS?), latency requirements (real-time or batch?), data availability (do we have labels?), existing infrastructure, and the primary metric. This demonstrates business sense and prevents spending 40 minutes on the wrong problem.

Worked Example — Feed Ranking System (Facebook / Instagram)

Requirements: Rank N candidate posts for each of 1B users; latency <100ms at p99; optimise for long-term engagement; content safety constraint (no misinformation/harmful content).

Metrics: Offline — AUC-ROC for engagement prediction, NDCG for ranking quality. Online — sessions per user, total time spent, post interaction rate. Business — 7-day and 30-day retention. Note: optimising purely for clicks creates sensationalist content — weight “meaningful social interactions” (comments, shares) higher than passive likes.

Feature families: User features (historical engagement patterns, preferences, device type); Post features (content type, creator quality score, post age, engagement velocity); User-post interaction features (past interactions with this creator, content similarity to previously engaged posts). Training labels: positive = user interacted; negative = post shown but skipped. Label imbalance: 99%+ negatives — use negative sampling.

Model iteration:

1. Baseline: Chronological feed — no ML. Establishes metrics baseline. Often surprisingly hard to beat.
2. Logistic regression on engineered features: Fast to train, interpretable. Often achieves 70–80% of the performance of complex models.
3. Gradient boosted trees (XGBoost/LightGBM): Handles non-linear relationships and feature interactions automatically.
4. Two-tower neural network: Separate user and item towers producing embeddings; scored by dot product. Enables ANN retrieval for candidate generation at scale.
5. Multi-task learning: Jointly predict multiple engagement signals (like, comment, share, hide probabilities) to address the engagement vs. quality tradeoff.

Serving architecture (two-stage): (1) Candidate Generation: retrieve 500–1000 posts from millions using ANN search on embeddings (Faiss, ScaNN) — must be fast (<10ms). (2) Ranking: score all 500–1000 candidates with the full model and return top 50 (100ms budget). Pre-computed user embeddings live in a low-latency feature store (Redis); real-time features computed inline.

Worked Example — Fraud Detection System

Key design differences from recommendation: Extreme class imbalance (0.1% fraud rate); adversarial users who actively evade the model; asymmetric costs (false positives block legitimate transactions, false negatives mean fraud loss); latency critical (<50ms during payment processing).

Metrics: Precision at a given recall level (e.g., precision at 90% recall). Not AUC-ROC — the fraudster can make the operating point irrelevant. Business metric: fraud loss rate ($/$ processed) at a fixed false positive rate (e.g., 0.5% of legitimate transactions blocked). The model outputs a probability score; the threshold is set by business policy based on the fraud loss vs. friction tradeoff.

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import precision_recall_curve
import numpy as np

model = GradientBoostingClassifier(
    n_estimators=500, learning_rate=0.05,
    max_depth=6, subsample=0.8
)
model.fit(X_train, y_train)

y_prob = model.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_val, y_prob)

# Find threshold achieving 90% recall
target_recall = 0.90
idx = np.argmin(np.abs(recall - target_recall))
optimal_threshold = thresholds[idx]

print('Threshold:', round(optimal_threshold, 4))
print('Precision at 90% recall:', round(precision[idx], 4))

# Business cost: weighted FP + FN
fraud_loss_per_fn   = 150   # avg fraud value
friction_cost_per_fp = 5   # customer friction cost
y_pred = (y_prob >= optimal_threshold).astype(int)
fn = int(((y_val == 1) & (y_pred == 0)).sum())
fp = int(((y_val == 0) & (y_pred == 1)).sum())
print('Business cost: $' + str(fn*fraud_loss_per_fn + fp*friction_cost_per_fp))

Common ML Design Problems and Key Decisions

What interviewers are actually evaluating: They are not looking for the “correct” architecture — there is none. They evaluate: whether you ask the right clarifying questions before proposing solutions; whether your metric choices align with business goals; whether you identify the hardest engineering and ML challenges; whether you articulate tradeoffs clearly; and whether you demonstrate awareness of production realities (data freshness, latency budgets, monitoring). The candidate who says “I’d use a transformer because it performs best” scores far lower than one who says “I’d start with logistic regression to establish a debuggable baseline, then move to a tree model, then a neural approach if we need the extra few percent — because reducing time-to-production matters more than raw accuracy at this stage.”

For further preparation, pair this framework with the interview Q&A clusters on Machine Learning, Deep Learning, and MLOps. The data engineering components are covered in our Data Engineering Interview Q&A. The statistical reasoning behind evaluation and A/B testing in ML systems is in our Statistics Interview Q&A and Model Evaluation guide. The ensemble and regularisation techniques used within these systems are covered in our Ensemble Methods guide and Regularisation Techniques guide.