Experiment Design for Data Scientists – Power Analysis, A/B Testing and Sequential Testing

Every A/B test, clinical trial, and product experiment is an instance of experiment design — the branch of statistics concerned with how to collect data efficiently to answer causal questions. Yet the majority of data scientists running experiments have never formally studied experiment design, and the gaps show: underpowered tests that declare “no effect” when effects are real, multiple testing that inflates false discovery rates, novelty effects that produce spurious short-term wins, network interference that violates the independence assumption, and sequential peeking that invalidates p-values. These are not subtle issues — they represent systematic errors that cause organisations to make bad product and business decisions at scale. This guide covers the core concepts of experiment design that every data scientist needs: power analysis, A/B test mechanics and common pitfalls, multi-armed bandits, sequential testing, factorial designs, and the analysis of observational data when randomisation is not feasible.

Statistical Power and Sample Size Calculation

A hypothesis test has four parameters that are related by a fixed constraint: the significance level α (probability of a false positive — rejecting H₀ when it is true), the power 1−β (probability of a true positive — correctly rejecting H₀ when the alternative is true), the effect size δ (the minimum difference worth detecting), and the sample size N. Given any three, the fourth is determined. In practice, α is set by convention (0.05 for most business experiments), power is set at 0.80 or 0.90 (meaning 20% or 10% false negative rate), and the effect size is the minimum business-relevant difference — leading to a specific required sample size. Running the experiment without this calculation first, then analysing results with whatever sample accumulated, is the most common experiment design error.

For a two-sample proportion test (the most common case in product A/B testing — comparing conversion rates), the required sample size per group is approximately N = 2(z_{α/2} + z_β)²p(1−p)/δ², where p is the baseline conversion rate, δ is the minimum detectable effect (MDE), z_{α/2} is the z-score for the significance level, and z_β is the z-score for the power level. For α = 0.05 (two-tailed) and power = 0.80: z_{α/2} = 1.96 and z_β = 0.84. The key insight from this formula is that required sample size scales as 1/δ²: halving the MDE quadruples the required sample. This is why long-running experiments are needed to detect small effects — and why declaring “no effect” from a short experiment that was never powered to detect the relevant effect size is a serious methodological error. Our hypothesis testing guide covers the z-test and t-test mechanics, and our statistics fundamentals guide covers the CLT approximations that justify the normal distribution assumption in these formulas.

A/B Testing — Pitfalls and Best Practices

Standard A/B testing — randomly assigning subjects to control (A) and treatment (B), measuring the outcome of interest, and applying a statistical test to determine whether the observed difference is consistent with chance — is conceptually simple but operationally complex. The A/B testing guide covers the full mechanics; here we focus on the design pitfalls most commonly encountered in industry practice.

Novelty and primacy effects are systematic biases that affect almost every product experiment. Novelty effect: users interact more with any new feature simply because it is new, producing an inflated short-term treatment effect that decays to a lower long-term steady-state. Primacy effect (the opposite): users who are habituated to the existing experience show lower engagement with a changed feature initially, producing a deflated short-term effect that improves as users adapt. Both effects cause short-duration experiments to produce estimates that do not reflect the long-run causal effect of the change. The standard mitigation is to run experiments long enough to reach steady-state, and to plot the treatment effect over time — if the effect is growing or shrinking, the experiment has not yet stabilised.

Network interference (also called SUTVA violation — Stable Unit Treatment Value Assumption) occurs when a user’s outcome is affected by other users’ treatment assignments. This happens in social networks (treating one user changes their interactions with untreated friends), marketplace platforms (treating sellers affects buyer outcomes), and communication products. Standard A/B test analysis assumes independence between units; network interference violates this and biases treatment effect estimates in either direction. The solutions are cluster-based randomisation (randomising at the network cluster level rather than the user level), ego-network splitting, or using switchback designs for marketplace experiments. Our causal inference guide covers the SUTVA assumption and its violations in detail.

Multiple testing inflation is the systematic increase in false discovery rate that results from testing many hypotheses simultaneously — testing 20 metrics in a single experiment at α = 0.05 gives an expected one false positive even if no treatment effect exists. The Bonferroni correction (α/m for m tests) and the Benjamini-Hochberg procedure (controlling the false discovery rate rather than the family-wise error rate) are the standard corrections, covered alongside their power implications in our hypothesis testing guide.

Sequential Testing and the Peeking Problem

The most widespread statistical malpractice in industry A/B testing is “peeking”: checking the p-value of an ongoing experiment repeatedly and stopping when p < 0.05 is first observed, regardless of whether the pre-specified sample size has been reached. Peeking inflates the false positive rate dramatically: a nominally 5% test that is checked daily for 30 days has a true false positive rate of approximately 30%. Yet peeking is almost universal in organisations where business pressure to “know if it’s working” is high.

Sequential testing methods provide valid inference under continuous monitoring. Always-valid confidence sequences (Johari et al., 2022, used by Optimizely’s Stats Engine) construct confidence intervals that are valid at any stopping time, without requiring a pre-specified sample size. E-values (Grünwald et al.) are a fully sequential alternative to p-values with better composition properties. Group sequential designs (O’Brien-Fleming, Pocock boundaries) pre-specify a finite set of interim looks with adjusted significance thresholds that preserve the overall α. All three approaches allow valid early stopping for efficacy (stopping when a positive result is clear) or futility (stopping when the treatment effect is clearly zero or below the MDE). The connection to Bayesian hypothesis testing — where the posterior probability of H₁ is updated continuously as data arrives, with no p-value inflation — is worth noting: Bayesian A/B testing naturally supports sequential analysis.

Factorial Designs and Multi-Armed Bandits

When multiple product changes are being evaluated simultaneously, running separate A/B tests for each is inefficient and misses interactions between changes. Factorial designs assign subjects to all combinations of factor levels in a balanced design, enabling simultaneous estimation of main effects and interactions. A 2² design (two binary factors, four treatment cells: A₀B₀, A₀B₁, A₁B₀, A₁B₁) estimates both main effects and their interaction with a single experiment at the same total sample size as two separate A/B tests — but with the additional benefit of the interaction estimate. If the interaction is large (the effect of A depends on whether B is present), separate tests would give misleading results about each factor in isolation.

Multi-armed bandit (MAB) algorithms adaptively allocate traffic to arms (variants) based on observed performance, trading off exploration (learning about underperforming arms) and exploitation (directing traffic to better arms). The ε-greedy policy allocates fraction ε of traffic to random exploration and 1−ε to the current best arm. Thompson Sampling — drawing a sample from the posterior distribution of each arm’s reward rate and selecting the arm with the highest sample — is the Bayesian approach and typically outperforms ε-greedy. The reinforcement learning guide covers the multi-armed bandit problem as the simplest setting in RL, building up to contextual bandits (where the optimal arm depends on user features — essentially a personalisation policy) and full sequential decision problems. MABs are particularly valuable for short-lived campaigns (e-commerce promotions, news article selection) where the cost of exposing users to suboptimal variants during a fixed-horizon A/B test is high. The connection to recommendation systems is direct: the explore-exploit problem in recommenders is a contextual bandit problem.

The choice between A/B testing and MAB depends on the objective. A/B testing provides a clean causal estimate of the treatment effect and controls false positive rate precisely — it is the right choice when the goal is to make a confident, durable product decision. MAB minimises regret during the experiment — it is the right choice when the goal is to maximise short-term performance while learning. The model evaluation framework applies to experiment evaluation too: the metric you optimise during the bandit exploration (click-through rate, conversion rate) must be chosen carefully to avoid optimising a proxy that does not reflect the true business objective.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

The statistical mechanics of A/B testing — z-tests, t-tests, chi-squared tests, confidence intervals — are in our A/B Testing guide and Hypothesis Testing guide. Power analysis and sample size calculations for different test types are in our Statistics Fundamentals guide. Causal inference beyond randomised experiments — difference-in-differences, regression discontinuity, and instrumental variables — is in our Causal Inference guide. The multi-armed bandit and reinforcement learning connection is in our Reinforcement Learning guide. Bayesian A/B testing with Thompson Sampling is covered in our Bayesian Statistics guide.