Hypothesis Testing in Python: A Practical Guide (2026)
Hypothesis testing lets you make data-driven decisions with quantified uncertainty. Whether running A/B tests, comparing user groups, or validating model improvements, you need to know which test to use and how to interpret the results.
The Framework
State H0 (null — usually no effect). State H1 (alternative — what you want to show). Choose alpha (usually 0.05). Collect data. Compute test statistic. Calculate p-value. If p less than alpha, reject H0.
One-Sample t-Test
import numpy as np
from scipy import stats
sample = np.random.normal(loc=55, scale=12, size=40)
t_stat, p_value = stats.ttest_1samp(sample, popmean=50)
print(f't={t_stat:.4f} p={p_value:.4f}')
print('Reject H0' if p_value < 0.05 else 'Fail to reject H0')Independent Two-Sample t-Test (A/B Testing)
control = np.random.normal(loc=45, scale=10, size=100)
treatment = np.random.normal(loc=48, scale=10, size=100)
# Test equal variance first
levene_p = stats.levene(control, treatment).pvalue
t_stat, p_value = stats.ttest_ind(control, treatment, equal_var=(levene_p > 0.05))
# Effect size (Cohen's d)
pooled_std = np.std(np.concatenate([control, treatment]))
d = (treatment.mean() - control.mean()) / pooled_std
print(f't={t_stat:.4f} p={p_value:.4f} d={d:.4f}')Chi-Square Test of Independence
# Does button colour affect conversion?
observed = np.array([[200, 300], [250, 250], [180, 320]])
chi2, p, dof, expected = stats.chi2_contingency(observed)
print(f'chi2={chi2:.4f} dof={dof} p={p:.4f}')
if p < 0.05: print('Significant: button colour affects conversion')ANOVA + Tukey Post-Hoc
v1, v2 = np.random.normal(45,8,80), np.random.normal(50,8,80)
v3, v4 = np.random.normal(48,8,80), np.random.normal(52,8,80)
f_stat, p_value = stats.f_oneway(v1, v2, v3, v4)
print(f'F={f_stat:.4f} p={p_value:.4f}')
if p_value < 0.05:
from statsmodels.stats.multicomp import pairwise_tukeyhsd
all_data = np.concatenate([v1,v2,v3,v4])
labels = ['V1']*80+['V2']*80+['V3']*80+['V4']*80
print(pairwise_tukeyhsd(all_data, labels, alpha=0.05))Sample Size Calculation
from statsmodels.stats.power import TTestIndPower
baseline, mde = 0.10, 0.12
effect_size = (mde - baseline) / np.sqrt(baseline * (1 - baseline))
n = TTestIndPower().solve_power(effect_size=effect_size, power=0.80, alpha=0.05)
print(f'Required n per group: {int(np.ceil(n))}')Multiple Testing Correction
from statsmodels.stats.multitest import multipletests
raw_p = [0.01, 0.04, 0.12, 0.03, 0.28, 0.002, 0.19, 0.08, 0.35, 0.15]
reject, corrected, _, _ = multipletests(raw_p, alpha=0.05, method='fdr_bh')
for i, (r, c, rej) in enumerate(zip(raw_p, corrected, reject)):
print(f'Test {i+1}: raw={r:.3f} corrected={c:.3f} reject={rej}')Choosing the Right Test
Two groups, continuous, normal: t-test. Two groups, non-normal or ordinal: Mann-Whitney U. Three+ groups: ANOVA + Tukey. Two categorical variables: Chi-square. Same subjects, two timepoints: paired t-test. Correlation: Pearson (normal) or Spearman (non-normal).
Conclusion
Define hypotheses before looking at data. Choose the right test. Check assumptions. Compute and interpret effect size alongside p-value. P-values tell you significance; effect sizes tell you if it matters.
