Multiple Hypothesis Testing

One test at 5% fails 5% of the time. A thousand tests at 5% each fail about fifty times. Family-wise and FDR rules reset the bar for crowds.

$A thousand null tests yield about fifty false wins at single-test 0.05, cut by the flat Bonferroni bar at 0.00005 and the sloping BH line that caps the false share$

A thousand null tests yield about fifty false wins at single-test 0.05, cut by the flat Bonferroni bar at 0.00005 and the sloping BH line that caps the false share

Why Does This Exist?

Feature screens, prompt sweeps, and leaderboard climbs test hundreds of ideas and report the winner. At 5% each, 1,000 pure-noise ideas produce about 50 significant winners. Publishing the best without the denominator is selection bias with a p-value costume.

This page is the procedures companion to the multiple comparisons problem: that page proves why bulk testing breaks single-test rules, with the 64% false-positive rate at 20 tests. Here you learn which bar to use instead. Bonferroni controls the family-wise chance of any false win at $\alpha$ by testing each at $\alpha/m$ , where $m$ is the test count. Benjamini-Hochberg controls the fraction of false wins at $q$ by ranking p-values against $(k/m)q$ , where $k$ is the rank. It extends hypothesis testing to the bulk regime where question count dominates.

The one idea you need first

A p-value is uniform under the null: 5% of pure-noise tests fall below 0.05 by construction. Multiply by $m$ tests and the expected false count is $m \times 0.05$ ; with $m = 1000$ that is $50$ . The procedures below move the bar instead of moving the data.

Think of It Like This

Security lines for a stadium

One gate with a 5% false-alarm rate inconveniences a few fans. A thousand gates at the same rate stop hundreds of innocent fans while the announcer celebrates each catch.

Bonferroni searches every fan thoroughly: slow, almost no false stops, long lines. FDR accepts that 10% of stops are wrong but keeps the crowd moving, then re-checks whoever was stopped. Where it stops: stadium queues are independent lines, while tests share data and correlate, which bends both guarantees toward conservatism or optimism depending on the dependence.

How It Actually Works

Bonferroni for no false wins. Reject $p_i \leq \alpha/m$ for each of the $m$ tests, where $p_i$ is the $i$ -th p-value. With $m = 1000$ and $\alpha = 0.05$ , the per-test bar is $0.05/1000 = 0.00005$ , and the expected false count falls to $1000 \times 0.00005 = 0.05$ . Valid under any dependence, and severe: at large $m$ almost nothing clears it.

Holm for the same guarantee with more power. Sort the p-values ascending as $p_{(1)} \leq \cdots \leq p_{(m)}$ and compare the $k$ -th against $\alpha/(m - k + 1)$ , stopping at the first failure. It rejects everything Bonferroni rejects plus sometimes more, with the same family-wise guarantee and no extra assumptions. Prefer Holm to plain Bonferroni whenever software allows it.

Benjamini-Hochberg for a controlled false share. Sort ascending, find the largest rank $k$ with $p_{(k)} \leq (k/m)q$ , and reject the first $k$ . At $q = 0.1$ with 40 rejections, at most about 4 are false on average. The bar slopes upward with rank, so strong small p-values pull weaker ones through: a discovery engine rather than a firewall.

Storey for reclaiming power. Estimate the null proportion $\pi_0$ from the bulk of large p-values, then run BH at $q/\pi_0$ . When most tests are null, $\pi_0 \approx 1$ and nothing changes; when many are real signals, the bar loosens honestly.

Which procedure when

Use Bonferroni or Holm when one false win is costly: shipping a model, claiming a launch, publishing a confirmatory result. Use BH when you screen hundreds of candidates and validate the winners afterward: feature screens, prompt shortlists, segment hunts. Tighten $q$ when each rejection triggers spend, and confirm rejections on fresh data whenever the sweep size $m$ was not fixed in advance.

Worked example

Thousand pure-noise tests, $\alpha = 0.05$ : expect $1000 \times 0.05 = 50$ false wins, and $P(\text{at least one}) = 1 - 0.95^{1000} \approx 1 - 10^{-22} \approx 1$ . Certain embarrassment. Bonferroni at $0.00005$ cuts the expectation to $0.05$ .

BH ranking with sorted p-values $[0.001, 0.004, 0.03, 0.04]$ at $q = 0.1$ and $m = 100$ : the cutoffs $(k/m)q$ are $[0.001, 0.002, 0.003, 0.004]$ . Rank 1 passes because $0.001 \leq 0.001$ ; rank 2 fails because $0.004 \leq 0.002$ is false; ranks 3 and 4 fail likewise. The largest passing rank is $k = 1$ , so reject one. The $0.004$ runner-up fails $0.004 \leq 0.002$ : close is not enough.

Code

1import numpy as np2
3rng = np.random.default_rng(10)4pvals = rng.random(1000)  # all null, uniform5print(int((pvals < 0.05).sum()))  # -> ~506print(int((pvals < 0.05 / 1000).sum()))  # -> 0

Null uniforms trip the single-test bar about 50 times and the Bonferroni bar zero. The denominator decides the headline.

Watch Out For

Reporting the winner without the sweep size

Best of 500 prompts gains 3 points with p equal to 0.01 hides 499 losers. Unadjusted, that p-value is one draw of 500, and a 0.01 best is expected under pure noise. Leaderboards and peeking A/B tests run on this fuel.

Log and report $m$ : every prompt, seed, and checkpoint tested. Adjust the bar or hold out fresh data for the winner. No denominator, no claim.

Reading FDR as per-discovery certainty

FDR at $q = 0.1$ means about 10% of rejections are false on average, not that each rejection is 90% true. With 5 rejections, one false is normal, and which one is unknown.

Follow up rejections with validation. If each discovery triggers spend, tighten $q$ or confirm each one independently.

The Quick Version

Single-test 5% on 1,000 nulls yields about 50 false wins and near-certain embarrassment.
Bonferroni tests each at $\alpha/m$ ; at 1,000 tests the bar is 0.00005.
Holm steps the Bonferroni bar down rank by rank and is uniformly more powerful.
BH ranks p-values against $(k/m)q$ and caps the false share of wins.
The ranking example rejects one of four candidates at $q = 0.1$ ; near-misses still fail.
The inflation mechanism is the multiple comparisons problem; this page is its procedures.
Always report the sweep size $m$ ; unadjusted best-of-sweep p-values are fiction.
FDR controls the batch fraction, not each discovery's truth.