Hypothesis Testing for Data Scientists – t-tests, Chi-squared, ANOVA and Non-Parametric Tests

Hypothesis testing is the formal statistical framework for making decisions from data — answering questions like “did this product change increase revenue?”, “are these two customer segments different?”, or “does this drug reduce recovery time?” Every A/B test, every experiment, every claim that a model has improved requires hypothesis testing to distinguish real effects from random noise. Despite being fundamental to data science, hypothesis tests are frequently misused: p-values misinterpreted, assumptions violated, and the wrong test applied. This guide covers the core tests every data scientist must know, when to use each, how to run them in Python, and the common mistakes to avoid.

Hypothesis testing underpins the A/B testing methodology in our A/B Testing guide and the statistical foundations in our Statistics Interview Q&A. The probability distributions that define test statistics are covered in our Probability Distributions guide. The Bayesian alternative to frequentist hypothesis testing is in our Bayesian Statistics guide. Model performance comparison using these tests is covered in our Model Evaluation guide.

The Framework — Null Hypothesis, p-values, and Error Types

Every hypothesis test follows the same structure. The null hypothesis H_0 is the default assumption of no effect or no difference — the claim we are trying to disprove. The alternative hypothesis H_1 is what we believe is true if H_0 is false. The test statistic summarises the data into a single number that measures how far the data is from what H_0 predicts. The p-value is the probability of observing a test statistic at least as extreme as the one computed, assuming H_0 is true. If p < alpha (the significance level, typically 0.05), we reject H_0.

What a p-value is NOT: A p-value is not the probability that H_0 is true, nor the probability that the result occurred by chance, nor the probability that H_1 is true. It is a conditional probability: P(data as extreme as observed | H_0 is true). A p-value of 0.03 means: if H_0 were true, we would see data this extreme only 3% of the time — not that there is a 97% chance the effect is real.

Type I error (false positive) rate is controlled by alpha — we set it before the test. Type II error (false negative) rate is controlled by statistical power — the probability of correctly detecting a real effect. Power depends on sample size, effect size, and alpha. A test with 80% power will miss 20% of real effects at the chosen effect size.

One-Sample and Two-Sample t-Tests

The t-test compares means when the data is approximately normally distributed (or n is large enough for CLT to apply). The test statistic follows a Student-t distribution with (n-1) or (n1+n2-2) degrees of freedom.

import numpy as np
from scipy import stats

# --- One-sample t-test: is the mean different from a hypothesised value? ---
# H0: mu = 100 (average order value is 100 INR)
orders = np.array([98, 115, 102, 87, 134, 109, 95, 121, 88, 110,
                   97, 118, 103, 92, 125, 107, 99, 116, 88, 112])
t_stat, p_value = stats.ttest_1samp(orders, popmean=100)
print('One-sample t-test:')
print('  t =', round(t_stat, 4), '  p =', round(p_value, 4))
print('  Mean:', round(orders.mean(), 2), '  95% CI:',
      stats.t.interval(0.95, df=len(orders)-1,
                       loc=orders.mean(), scale=stats.sem(orders)))

# --- Two-sample independent t-test: do two groups have different means? ---
# H0: mean(control) == mean(treatment)
control   = np.random.normal(loc=50.0, scale=12, size=200)
treatment = np.random.normal(loc=53.5, scale=11, size=185)

# Welch's t-test (equal_var=False) — does not assume equal variances
# Almost always prefer Welch's over Student's t-test
t_stat, p_value = stats.ttest_ind(treatment, control, equal_var=False)
effect_size = (treatment.mean() - control.mean()) / np.sqrt(
    (treatment.std()**2 + control.std()**2) / 2)  # Cohen's d

print('
Two-sample Welch t-test:')
print('  Control mean:', round(control.mean(), 3))
print('  Treatment mean:', round(treatment.mean(), 3))
print('  t =', round(t_stat, 4), '  p =', round(p_value, 4))
print('  Cohen d =', round(effect_size, 3),
      '(0.2=small, 0.5=medium, 0.8=large)')
print('  Decision:', 'Reject H0' if p_value < 0.05 else 'Fail to reject H0')

# --- Paired t-test: same subjects, two conditions ---
before = np.array([72, 85, 68, 91, 77, 83, 69, 88, 74, 80])
after  = np.array([68, 79, 65, 85, 71, 76, 62, 82, 70, 73])
t_stat, p_value = stats.ttest_rel(after, before)
print('
Paired t-test (before vs after intervention):')
print('  Mean reduction:', round((before - after).mean(), 2))
print('  p =', round(p_value, 4))

Chi-Squared Tests — Categorical Data

Chi-squared tests work with categorical data. The goodness-of-fit test asks: does this observed frequency distribution match a theoretical one? The test of independence asks: are two categorical variables associated?

from scipy.stats import chi2_contingency, chisquare

# --- Chi-squared test of independence ---
# H0: click-through rate is independent of button colour
# Observed: [clicked, not_clicked] for each colour
observed = np.array([
    [450, 1550],   # Red button:  450 clicked, 1550 didn't
    [520, 1480],   # Green button: 520 clicked, 1480 didn't
    [390, 1610],   # Blue button:  390 clicked, 1610 didn't
])

chi2, p_value, dof, expected = chi2_contingency(observed)
print('Chi-squared test of independence:')
print('  chi2 =', round(chi2, 4), '  dof =', dof, '  p =', round(p_value, 4))
print('  CTR by colour:', [round(r[0]/r.sum()*100, 1) for r in observed], '%')
if p_value < 0.05:
    print('  Button colour significantly affects CTR')

# Cramer's V — effect size for chi-squared (0=none, 1=perfect association)
n = observed.sum()
cramers_v = np.sqrt(chi2 / (n * (min(observed.shape) - 1)))
print('  Cramer V =', round(cramers_v, 3), '(effect size)')

# --- Goodness-of-fit test: does data follow a Poisson distribution? ---
counts     = np.array([41, 35, 17, 5, 2, 0])   # observed frequency of 0,1,2,3,4,5+ events
lam_hat    = sum(i * counts[i] for i in range(len(counts))) / counts.sum()
expected_p = stats.poisson.pmf(range(len(counts)), lam_hat)
expected_p[-1] = 1 - expected_p[:-1].sum()     # last bin: 5+
expected_f = expected_p * counts.sum()
chi2, p = chisquare(counts, expected_f)
print('
Poisson goodness-of-fit: p =', round(p, 4))

ANOVA and Non-Parametric Alternatives

One-way ANOVA tests whether the means of 3+ groups are equal. It partitions total variance into between-group variance and within-group variance — if between-group variance is much larger than within-group variance (F = MS_between / MS_within is large), at least one group mean is different. ANOVA assumes normality and equal variances (homoscedasticity); if assumptions are violated, use the Kruskal-Wallis test (the non-parametric equivalent).

# --- One-way ANOVA: do 3+ groups have different means? ---
group_a = np.random.normal(50, 10, 80)
group_b = np.random.normal(55, 11, 75)
group_c = np.random.normal(48, 9,  85)
group_d = np.random.normal(53, 10, 70)

f_stat, p_anova = stats.f_oneway(group_a, group_b, group_c, group_d)
print('One-way ANOVA: F =', round(f_stat, 4), '  p =', round(p_anova, 4))

if p_anova < 0.05:
    # Post-hoc: Tukey HSD — which specific pairs differ?
    from statsmodels.stats.multicomp import pairwise_tukeyhsd
    import pandas as pd
    all_data   = np.concatenate([group_a, group_b, group_c, group_d])
    all_labels = (['A']*80 + ['B']*75 + ['C']*85 + ['D']*70)
    tukey = pairwise_tukeyhsd(all_data, all_labels, alpha=0.05)
    print(tukey.summary())

# --- Non-parametric alternatives ---
# Mann-Whitney U: non-parametric two-sample test (compare medians)
u_stat, p_mw = stats.mannwhitneyu(treatment, control, alternative='two-sided')
print('
Mann-Whitney U: p =', round(p_mw, 4))

# Kruskal-Wallis: non-parametric one-way ANOVA
h_stat, p_kw = stats.kruskal(group_a, group_b, group_c, group_d)
print('Kruskal-Wallis: H =', round(h_stat, 4), '  p =', round(p_kw, 4))

# Shapiro-Wilk: test normality assumption (use before choosing t-test vs Mann-Whitney)
stat, p_norm = stats.shapiro(control[:50])   # Shapiro-Wilk best for n < 50
print('Shapiro-Wilk normality test: p =', round(p_norm, 4),
      '  Normal?' , 'Yes' if p_norm > 0.05 else 'No')

For the full statistical interview question coverage — including the precise definitions interviewers test — our Statistics Interview Q&A covers 40+ questions including p-value interpretation, confidence intervals, power calculations, and the central limit theorem. The A/B testing application of these tests — including CUPED variance reduction and multiple testing correction — is in our A/B Testing guide. The Bayesian alternative to all these frequentist tests is covered in our Bayesian Statistics guide. The probability distributions underlying these test statistics are covered in our Probability Distributions guide.