Advanced A/B Testing – Statistical Significance, Power & Pitfalls

Basic A/B testing is straightforward — split users, measure the difference, check if it’s significant. But production A/B testing is full of subtleties that can lead you to wrong conclusions. This guide covers the advanced topics: statistical power, sample size planning, multiple testing corrections, novelty effects, and when to use Bayesian methods instead.

The Frequentist Framework – Quick Review

from scipy import stats
import numpy as np

# Control: 500 conversions out of 5000 (10%)
# Treatment: 575 conversions out of 5000 (11.5%)
control_conv   = 500; control_n   = 5000
treatment_conv = 575; treatment_n = 5000

stat, p_value = stats.proportions_ztest(
    [treatment_conv, control_conv],
    [treatment_n, control_n],
    alternative='larger'  # one-tailed: treatment > control
)

lift = (treatment_conv/treatment_n) / (control_conv/control_n) - 1
print(f"Lift: {lift:.1%}, p-value: {p_value:.4f}")
print("Significant" if p_value < 0.05 else "Not significant")

Statistical Power and Sample Size Planning

Most teams run A/B tests too short or without planning sample size. The result: tests are underpowered, meaning they frequently miss real effects (high false negative rate). Statistical power is the probability of detecting a true effect. 80% is the standard minimum; 90% is better for important decisions.

from statsmodels.stats.power import NormalIndPower

analysis = NormalIndPower()

# How many users per arm do we need?
n_per_arm = analysis.solve_power(
    effect_size=0.05,      # minimum detectable effect (MDE) as Cohen's h
    alpha=0.05,            # false positive rate
    power=0.80,            # desired power
    alternative='two-sided'
)
print(f"Required n per arm: {int(n_per_arm):,}")

# Or: what power do we have with our planned n?
power = analysis.solve_power(
    effect_size=0.05, alpha=0.05, nobs1=10000, alternative='two-sided')
print(f"Power with 10,000/arm: {power:.1%}")

Minimum Detectable Effect (MDE)

The MDE is the smallest effect you want to be able to detect reliably. Before running any test, decide: "If the treatment causes less than X% improvement, we wouldn't ship it anyway." Set your MDE to X%. A common mistake is setting MDE too small — if you want to detect a 0.1% conversion lift, you'll need millions of users. If a 2% lift is the business threshold, set MDE to 2% and your sample size will be much more manageable.

The Multiple Testing Problem

from statsmodels.stats.multitest import multipletests

# You ran 20 A/B tests simultaneously. 2 came back significant at p=0.05.
# But with 20 tests, you'd expect 1 false positive by chance!
p_values = [0.03, 0.15, 0.04, 0.67, 0.09, ...]  # p-values from 20 tests

# Bonferroni correction (conservative)
reject_bonferroni, p_corrected, _, _ = multipletests(p_values, alpha=0.05, method='bonferroni')

# Benjamini-Hochberg (less conservative, controls false discovery rate)
reject_bh, p_corrected_bh, _, _ = multipletests(p_values, alpha=0.05, method='fdr_bh')

print("Rejected after Bonferroni:", reject_bonferroni.sum())
print("Rejected after BH (FDR):", reject_bh.sum())

Novelty Effects and Long-Term Testing

A new feature often shows inflated performance in the first week as curious users engage with it. This novelty effect can disappear over time, making a short test misleadingly positive. Run tests for at least 2 full weeks (capturing weekly seasonality twice) and ideally 4 weeks. If your test is significant after week 1, don't stop — novelty effects are particularly common in UI changes.

Sequential Testing – Peeking Safely

The standard guidance is "don't peek at results before your predetermined sample size is reached." But teams need to stop harmful experiments early. Sequential testing provides a statistically valid way to check results continuously:

pip install sequential-testing

# Always Valid Inference (AVI) / e-values approach
# Or use Spotify's open-source sequential testing library

# Simple approach: Bonferroni-corrected alpha spending
n_looks = 5  # you'll check results 5 times
adjusted_alpha = 0.05 / n_looks  # 0.01 per look

# Only reject if p < 0.01 at any intermediate check
# (conservative but valid)

Bayesian A/B Testing

import numpy as np
from scipy import stats

# Bayesian: model conversions as Beta-distributed
# Prior: Beta(1,1) = uniform (no prior knowledge)
prior_alpha = 1; prior_beta = 1

# Update with observed data
control_alpha   = prior_alpha + control_conv
control_beta    = prior_beta  + (control_n - control_conv)
treatment_alpha = prior_alpha + treatment_conv
treatment_beta  = prior_beta  + (treatment_n - treatment_conv)

# Sample from posteriors
control_samples   = np.random.beta(control_alpha, control_beta,   100_000)
treatment_samples = np.random.beta(treatment_alpha, treatment_beta, 100_000)

prob_treatment_better = (treatment_samples > control_samples).mean()
expected_lift = (treatment_samples / control_samples - 1).mean()

print(f"P(treatment > control): {prob_treatment_better:.1%}")
print(f"Expected lift: {expected_lift:.2%}")

Bayesian A/B testing gives you a probability that treatment is better (not just a binary significant/not significant), allows early stopping with valid inference, and incorporates prior knowledge. The trade-off: results depend on the prior, which requires justification.

Common A/B Testing Mistakes

The most frequent errors data scientists make in A/B testing are stopping early when results look significant (without multiple testing correction), running tests without pre-specifying sample size (leading to underpowered tests), not accounting for SUTVA violations (when treatment and control users interact and influence each other), using the wrong statistical test for the metric type (t-test for binary conversion rates instead of z-test for proportions), and not checking for novelty effects by running tests long enough. Each of these can lead to shipping features that don't actually work or missing features that do.

Conclusion

Rigorous A/B testing is one of the highest-leverage skills for any data scientist working in a product context. The statistical machinery is not complicated once you understand power, MDE, and the multiple testing correction. The hardest part is the organisational discipline: committing to a sample size before starting, not peeking, and running tests long enough to observe true steady-state behaviour. Get this right and your A/B testing results will be trustworthy — which means the decisions your company makes based on them will be too.