A/B Testing and Statistical Significance – Complete Guide for Data Scientists

A/B testing — controlled randomised experiments to compare two or more variants — is the gold standard for making data-driven product and business decisions. Understanding how to design, execute, analyse, and draw correct conclusions from A/B tests is one of the most valued skills in applied data science. It requires combining statistical theory (hypothesis testing, power analysis, multiple comparisons) with practical engineering (random assignment, traffic splitting, metric selection) and business judgment. This guide covers the complete A/B testing workflow used at technology companies.

A/B testing is tested directly in data science case study interviews and connects to the statistical foundations in our Statistics Interview Q&A and Hypothesis Testing guide. The Bayesian alternative to frequentist A/B testing is covered in our Bayesian Statistics guide. The metrics design and data pipelines needed to run experiments at scale connect to our Data Engineering Interview Q&A and MLOps guide.

Experiment Design — The Most Important Phase

Most A/B test failures are design failures, not analysis failures. The design phase — choosing the right metric, calculating required sample size, defining success criteria before running — determines whether the test produces actionable conclusions or misleading results.

Step 1: Define the primary metric. One primary metric drives the go/no-go decision. Choosing multiple primary metrics guarantees false discoveries (multiple testing problem). The primary metric should be directly connected to the business goal (revenue, retention, conversion rate), sensitive enough to detect the effect size your change is expected to produce, and measurable within the experiment window.

Step 2: Calculate the required sample size before starting. Running an experiment until it reaches significance is p-hacking — it inflates the false positive rate to unacceptable levels. Calculate the required sample size upfront based on:

import numpy as np
from scipy import stats

def sample_size_two_proportions(p_control, mde_relative, alpha=0.05, power=0.80):
    p_treatment = p_control * (1 + mde_relative)
    p_pooled    = (p_control + p_treatment) / 2
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta  = stats.norm.ppf(power)
    numerator   = (z_alpha * np.sqrt(2 * p_pooled * (1 - p_pooled)) +
                   z_beta  * np.sqrt(p_control*(1-p_control) +
                                     p_treatment*(1-p_treatment))) ** 2
    denominator = (p_treatment - p_control) ** 2
    return int(np.ceil(numerator / denominator)), p_treatment

# Baseline 10% conversion, detect 5% relative lift (10% -> 10.5%)
n, p_treat = sample_size_two_proportions(p_control=0.10, mde_relative=0.05)
print('Required per group:', n, 'users')   # ~31,000 per group

# Sensitivity: how sample size scales with MDE
for mde in [0.02, 0.05, 0.10, 0.20]:
    n, _ = sample_size_two_proportions(0.10, mde)
    print('MDE', round(mde*100), '% lift -> N =', n, 'per group')

Variance Reduction with CUPED

CUPED (Controlled-experiment Using Pre-Experiment Data): A variance reduction technique that can halve required sample sizes by incorporating pre-experiment covariate data. Instead of analysing the raw metric Y, analyse the residual Y – theta*X where X is a pre-experiment version of the metric (e.g., revenue in the week before the experiment) and theta is the OLS coefficient. CUPED reduces variance without introducing bias because the pre-experiment covariate is uncorrelated with treatment assignment. Netflix, Airbnb, and most tech companies use CUPED as standard practice.

import pandas as pd
import numpy as np
from scipy import stats

def analyse_ab_test(df, metric_col, treatment_col, covariate_col=None):
    control   = df[df[treatment_col] == 0][metric_col]
    treatment = df[df[treatment_col] == 1][metric_col]

    if covariate_col:
        # CUPED: remove pre-experiment variance
        theta = df[metric_col].cov(df[covariate_col]) / df[covariate_col].var()
        df = df.copy()
        df['cuped'] = df[metric_col] - theta * df[covariate_col]
        control   = df[df[treatment_col] == 0]['cuped']
        treatment = df[df[treatment_col] == 1]['cuped']

    n_ctrl, n_treat   = len(control), len(treatment)
    mean_ctrl, mean_treat = control.mean(), treatment.mean()
    lift_abs = mean_treat - mean_ctrl
    lift_rel = lift_abs / mean_ctrl * 100
    t_stat, p_value = stats.ttest_ind(treatment, control, equal_var=False)
    se = np.sqrt(control.var()/n_ctrl + treatment.var()/n_treat)

    print('Control  mean:', round(mean_ctrl, 4), '  n:', n_ctrl)
    print('Treatment mean:', round(mean_treat, 4), '  n:', n_treat)
    print('Lift:', round(lift_abs, 4), '(' + str(round(lift_rel, 2)) + '%)')
    print('95% CI: [', round(lift_abs - 1.96*se, 4), ',', round(lift_abs + 1.96*se, 4), ']')
    print('p-value:', round(p_value, 4))
    print('Decision:', 'SHIP' if p_value < 0.05 and lift_rel > 0 else 'DO NOT SHIP')
    return p_value, lift_rel

Multiple Testing and the Five Most Common Mistakes

The multiple comparisons problem: If you test 20 independent hypotheses at alpha=0.05, you expect 1 false positive even if all null hypotheses are true. Corrections:

1. Peeking: Checking results and stopping as soon as p < 0.05 inflates false positive rates from 5% to 26%+ (if you peek 5 times). Fix: pre-register sample size; use sequential testing (mSPRT) for early stopping.

2. Novelty effect: Users engage more with anything new. A 20% lift in week 1 may settle to 2% by week 4. Fix: run long enough to capture the steady-state effect (at least 1–2 weeks).

3. Network effects: For social features, treating user A affects user B, violating SUTVA. Fix: cluster randomisation by social graph.

4. Sample Ratio Mismatch (SRM): If the control/treatment split differs from intended (50/50 becomes 48/52), randomisation is broken. Always run a chi-squared test on the sample ratio first. An SRM usually means a logging bug or survivorship bias.

5. Insufficient power: Under-powered tests miss real effects and inflate estimated effect sizes when they do reach significance (winner’s curse). Always calculate power upfront.

For Bayesian alternatives that avoid some of these frequentist pitfalls, our Bayesian Statistics guide covers Bayesian A/B testing with beta-binomial models. For the statistical foundations, our Hypothesis Testing guide and Statistics Interview Q&A cover Type I/II errors, power, and p-value interpretation. For case study interview frameworks, our Data Science Case Study guide walks through how to structure an experiment design answer end-to-end. The Model Evaluation guide covers evaluation methodology that complements experiment design.