Cheat sheet

Statistics and A/B Testing

Hypothesis tests, confidence intervals, power, sample size, variance reduction, and the traps that break experiments.

	H₀ true	H₀ false
Reject H₀	Type I error (α)	Correct (power = 1 − β)
Keep H₀	Correct	Type II error (β)

Data	Test
Two means, large n	Welch's t-test (z-test at scale)
Two proportions	Two-proportion z-test or chi-square
Skewed or heavy-tailed	Mann–Whitney U or bootstrap
Paired measurements	Paired t-test
Many groups	ANOVA, then corrected pairwise tests
Ratio metrics	Delta method or bootstrap