Case study interviews are the most feared and least practised part of data science interviews. Unlike technical questions with definitive answers, case studies are open-ended — they test how you think through ambiguous problems, structure a solution, and communicate your reasoning to a non-technical audience. Companies like Google, Meta, Amazon, Netflix, Uber, and Airbnb use case studies to assess the skills that matter most in day-to-day data science work. This guide covers the major types of case study questions with frameworks and sample answers.
The Case Study Interview Framework
Why companies ask case study questions:
Technical skills can be assessed with coding and theory questions. Case studies assess the complementary skills that differentiate senior data scientists: Can you identify the right metric to measure success? Can you design an experiment that will give unambiguous answers? Can you spot the flaw in an analysis that looks correct on the surface? Can you communicate a complex finding clearly to a product manager? Case studies reveal how you think when given incomplete information — the normal state of affairs in real data science work.
The STAR+ framework for case study answers:
Structure your answer: (1) Clarify — ask 1-2 clarifying questions to scope the problem before diving in. (2) State your approach — summarise what you will do before doing it. (3) Define the metric — what does success look like, quantitatively? (4) Identify data sources — what data would you need? (5) Describe the analysis — walk through your reasoning step by step. (6) Anticipate issues — proactively mention caveats, edge cases, and potential problems. (7) Recommendation — end with a clear, concrete recommendation. Do not ramble — structure beats completeness. Interviewers prefer a crisp, structured answer over a technically exhaustive but disorganised one.
Product Metrics Case Studies
Case 1: “Instagram’s feed engagement has dropped 15% in the last two weeks. Walk me through how you’d diagnose this.”
Clarifying questions: Is this across all platforms or just iOS/Android? All markets or specific regions? Is it the feed specifically or all engagement? Is the metric definition unchanged?
Diagnosis framework — work from broad to specific:
Step 1: Is it real? Check if the metric definition or logging changed. A 15% drop could be a data pipeline issue. Check for NULL spikes, unusual timestamps, or recent code deploys to analytics.
Step 2: Segment the drop. Break by platform (iOS vs Android), OS version, user cohort (new vs retained), geography, device type, feed type (stories vs reels vs posts), and time of day. A bug affecting iOS 18.0.1 would appear here. If the drop is uniform across all segments, it is more likely to be product-wide.
Step 3: Check for external factors. Competitor launches (TikTok feature update), platform-wide news event, algorithm change, content supply change (did number of posts decrease?).
Step 4: Correlate with recent changes. What shipped in the last two weeks? Features, algorithm changes, UI changes, feed ranking model updates. Did any correlate temporally?
Step 5: Investigate the specific component. Is average session length down, or just feed scroll depth? Are likes down but comments up? Is the drop in content consumption (views) or interaction (likes, comments)?
Recommendation: Prioritise checking data pipeline integrity first (fast, low-cost). Then segment analysis to identify which population is affected. Then correlate with recent deploys. Never assume the metric is accurately measured before investigating the measurement itself.
Case 2: “How would you measure the success of Instagram Stories?”
Clarifying: What does “success” mean for the business — user engagement, retention, monetisation, advertiser value?
North Star Metric: Stories daily active users (Stories DAU) / total DAU — what fraction of users use Stories, trending over time.
Engagement metrics: Stories created per Stories DAU (supply); Stories watched per viewer per session (demand); completion rate (how many viewers watch the full story); reply rate; reactions rate. These measure whether Stories provide value to both creators and viewers.
Retention impact: Do users who use Stories have higher 30-day retention than those who do not? (This requires careful causal analysis — Stories users may already be more engaged.) Run a causal inference model or experiment.
Monetisation: Stories ad revenue, advertiser return on ad spend (ROAS), CPM trends.
Counter-metrics (guardrails): Does Stories adoption come at the cost of feed engagement? Is feed post creation declining as creators shift to Stories?
Key insight: Always define a hierarchy — one North Star, a few driver metrics that explain it, and guardrails that should not degrade.
A/B Test Design Case Studies
Case 3: “We want to test a new checkout flow. Design the A/B test.”
Clarifying: What is the hypothesis? What change was made to checkout? What is the current checkout conversion rate (we need this for power analysis).
Randomisation unit: User, not session or page view. A user should always see the same checkout variant — seeing different flows across sessions confuses users and violates the SUTVA (Stable Unit Treatment Value Assumption) independence assumption.
Primary metric: Checkout conversion rate (completed purchases / users who reached checkout). This directly measures the business outcome.
Secondary metrics: Average order value, purchase completion time, cart abandonment rate at each step, error rate, customer support contacts (guardrail).
Sample size calculation: Use power analysis — decide minimum detectable effect (e.g., 2% relative lift in conversion), set α=0.05 and power=0.80. With a baseline conversion of 3% and MDE of 2% relative (=0.06% absolute), you need ~200,000 users per variant. At current traffic, that might be two weeks.
Risks: Novelty effect — users may engage more with the new flow just because it is new. Run for at least two weeks to let novelty fade. Network effects — if checkout has social sharing features, isolating users is not sufficient. Cookie churn — if users clear cookies, they may see both variants. Use login-based randomisation.
Decision criteria: Reject H₀ if p-value < 0.05 AND conversion rate increase is above the minimum threshold that justifies the development cost. Report both.
Case 4: “An A/B test shows higher revenue in treatment, but customer satisfaction scores decreased. What do you do?”
This is a metric trade-off question — very common at product companies. First, verify both results are statistically significant and practically meaningful. A 0.5% revenue lift with a 2% CSAT drop should be treated differently than a 10% revenue lift with a 0.1% CSAT drop.
Investigate the mechanism: Why did CSAT drop? Was the new flow confusing? Did it pressure users into purchases they later regretted (leading to returns and chargebacks)? Check: return rate, refund rate, churn in the following 30 days for treatment users, net revenue after returns.
Long-term vs short-term revenue: Higher short-term revenue from pushy flows often reduces long-term revenue through lower repeat purchase rates. Calculate 60-day revenue per user, not just immediate purchase value.
Recommendation framework: If the revenue lift is large and genuine (not returns) and the CSAT drop is small and temporary → ship with monitoring. If long-term metrics favour control → do not ship. If ambiguous → iterate on the design to find a version that improves revenue without hurting CSAT. Always present the full picture, including the risk of each choice, and let the product team make the final call with full information.
Root Cause Analysis Case Studies
Case 5: “Uber’s daily rides have dropped 10% this month. How do you investigate?”
Segment by: Geography (one city vs global drop — global = platform issue, local = market-specific). Time of day (surge pricing glitch often appears at peak hours). User type (riders vs drivers — is supply constrained?). Trip type (UberX vs Uber Black vs UberEats). Device (app crash on specific OS version?). Payment method (payment processor outage?).
External factors: Competitor promotion (Lyft coupon campaign), public transit strikes or improvements, regulatory changes (city banned surge pricing), weather anomalies, economic conditions.
Supply-side check: Are driver hours declining (economic conditions, competitor)? Has Uber’s driver incentive structure changed? Is driver cancellation rate up?
Demand-side check: Is the search-to-request conversion declining? Are users requesting but not completing? Check price sensitivity — did average trip price increase?
Funnel analysis: App open → search → request → match → trip completion → payment. Where in the funnel does the 10% loss occur? A drop at “match” suggests driver supply issue; a drop at “request” suggests demand/price issue; a drop at “payment” suggests payment system issue.
Estimation and Fermi Case Studies
Case 6: “Estimate the number of Uber rides per day in Mumbai.”
Framework — break into solvable components:
Mumbai population: ~20 million people. Working adults who might use Uber (age 18-60, non-poor): ~30% = 6 million potential users. Smartphones + disposable income for Uber: ~50% of those = 3 million addressable users. Uber penetration among addressable: ~20% use Uber (competitor alternatives: Ola, autos, metro) = 600,000 Uber users in Mumbai. Frequency: power users (5+ rides/week) ~15% = 90,000 users × 1.5 daily rides. Casual users (1-2 rides/week) ~45% = 270,000 users × 0.25 daily rides. Occasional users (~40%) = 240,000 × 0.05 daily rides. Total: 90K×1.5 + 270K×0.25 + 240K×0.05 = 135,000 + 67,500 + 12,000 ≈ 215,000 rides/day. Sanity check: Uber has stated it does ~6 million rides daily across India. With Mumbai being ~15% of rideshare activity, ~900,000 rides seems high. Revisit penetration or frequency assumptions. This is the right process — estimate, sanity check, revise. Show your reasoning clearly.
Metric Selection and Evaluation Case Studies
Case 7: “Netflix wants to increase average session length. Is that the right metric?”
The answer is: it depends, and you should challenge the metric. Session length can increase for bad reasons: content is less satisfying so users scroll longer looking for something to watch (the “decision paralysis” effect). Netflix actually uses completion rate and thumbs up/down ratings alongside session length, precisely because session length alone is ambiguous. Better metrics to discuss: Intentional play rate — did users find and play content immediately, or browse for 20 minutes? Completion rate — did users finish what they started? Content satisfaction (thumbs up proportion). Multi-month retention — are users renewing subscriptions? Content diversity — are users exploring new genres or stuck in a filter bubble? The interviewer is testing whether you can critically evaluate a metric before optimising for it. Blindly optimising engagement metrics has been publicly criticised for producing addictive rather than valuable experiences — showing awareness of this is impressive.
Common pitfalls to avoid in case study interviews:
Jumping to a solution without clarifying. Proposing an experiment without doing power analysis. Ignoring guardrail metrics. Not distinguishing statistical significance from practical significance. Saying “I would build an ML model” without first asking whether a simple rule or A/B test would answer the question faster. Not structuring your answer — rambling is the #1 killer of otherwise good case study answers. Not asking for feedback mid-answer: “Does this level of detail match what you’re looking for?” is always acceptable and shows self-awareness.
Conclusion
Case study interviews reward structured thinking and clear communication over encyclopaedic knowledge. Practice with a partner using real prompts (“diagnose a drop in metric X”, “design an A/B test for feature Y”, “estimate the size of market Z”). Record yourself answering — most people are surprised by how unstructured their answers sound when heard back. The goal is not to find the single correct answer — it is to demonstrate rigorous, transparent reasoning that a real team could build on. Companies hire people they can work with, and case study interviews show whether your thinking process is one they want in the room.
