What should be done when dozens of variants or segments are tested at once?

On synthetic data — built so that the real winners are known — running 40 tests at the 5% level with no correction produces false victories almost for certain: 81% of series of 40 tests contain at least one false discovery, and on average 18% of a series' "victories" are false. The Bonferroni correction brings the risk of being wrong at least once down to 3.9%, but recovers only a third of the true effects. The Benjamini-Hochberg procedure controls something else: the average proportion of false discoveries among the discoveries. It holds that proportion at 4.0% — exactly the value theory predicts here, no lower — while recovering 52% of the true effects. The right choice depends on what a false victory costs relative to a true victory missed.

An e-commerce team tests 40 product-page variants at the same time, each against its current version. Each test receives 8,200 visitors on the variant and 8,200 on the current page — each of those two groups is called an "arm". The baseline conversion rate is 5%. The team wants to roll out "everything that wins". In our data, eight variants really do improve conversion by one point (from 5% to 6%); the other 32 have no effect. These data are synthetic: that is what makes it possible to know, test by test, who really won.

Among 40 null hypotheses "the variant does not change conversion", which ones should be rejected for the errors to stay under control? And which error should be controlled: the probability of at least one false discovery (FWER, family-wise error rate) or the expected proportion of false discoveries among the discoveries (FDR, false discovery rate)?

The 5% level guarantees that one test with no effect comes out "significant" in 5% of cases. With 32 tests with no effect, about 1.6 false victories per series are to be expected (32 × 0.05), and the probability of having at least one becomes very high: 80.6%, that is 1 − 0.95³². Each test taken on its own is correct; it is the family of tests that goes off the rails.

• Bonferroni demands of every test a level 40 times stricter (5% / 40). It bounds at 5% the probability that at least one false alarm appears in the series, at the price of many true effects missed. Storey and Tibshirani judge that control "much too conservative" in their own field, genomics, where thousands of hypotheses are tested and an appreciable share of them are expected to be true effects ^{[C01, C02, C12]}. Here it is our simulation that shows it, not the quotation.
• Holm applies the same idea step by step, from the smallest to the largest: the same guarantee, a few more true effects recovered.
• Benjamini-Hochberg accepts a few false discoveries, but in a bounded proportion. The FDR is the expected proportion of false discoveries among the rejected hypotheses ^{[C06, C07]}. It is a less strict guarantee, but a more powerful one ^{[C08, C09]}.

For each test: the sample size and the number of conversions of each arm, and the decision rule fixed before looking at the results (level α, level q). No personal data: totals per test are enough. The page's dataset holds 40 rows (test_id, sample sizes, conversions) and, because it is synthetic, a column giving the true effect.

• m = 40 tests, m₀ = 32 true null hypotheses (unknown in practice).
• α: the significance level chosen in advance, that is, the false-alarm risk accepted for one test (here 5%). q: the FDR level chosen in advance, that is, the proportion of false victories accepted among the variants retained (here 5% as well). k: the rank of a test once the p-values are sorted from smallest to largest.
• pᵢ: the p-value of test i, that is, the probability of observing a gap at least as large as the one measured if the variant changed nothing. It comes from a z test on two conversion rates, with a variance pooled across the two arms, in both directions: a rise as well as a fall.
• V: number of false discoveries; R: total number of discoveries. P(·) reads "probability that", E[·] "average value over a large number of series".
• FWER = P(V ≥ 1). FDR = E[V / max(R, 1)], the expected proportion of false discoveries.
• Bonferroni rejects i if pᵢ ≤ α / m.
• Holm sorts the p-values p₍₁₎ ≤ … ≤ p₍ₘ₎ and rejects as long as p₍ₖ₎ ≤ α / (m − k + 1).
• Benjamini-Hochberg looks for the largest k such that p₍ₖ₎ ≤ k · q / m and rejects the k smallest p-values ^[C10].

In plain terms: Bonferroni and Holm reason about the whole series — their level is strict enough that the probability of being wrong even once across the 40 tests stays below 5%. Benjamini-Hochberg reasons about the harvest: its level relaxes as small p-values pile up, because many small p-values signal true effects rather than noise.

1. Size fixed in advance: with 8,200 visitors per arm, a test has an 80% chance of detecting a true gain of 1 point on a 5% baseline, at the 5% level in both directions — this is what is called power. That size is fixed before launch, never after.
2. Generation of the conversions visitor by visitor with a declared congruential recursion (seed 20260916; x ← 1103515245 · x + 12345 mod 2³¹), identical in Python and in R.
3. A pooled-variance z test for each variant, then the four rules: no correction, Bonferroni, Holm, Benjamini-Hochberg (α = q = 0.05).
4. Exact values computed wherever the design allows it: probability of at least one false alarm, power per test, and the Benjamini-Hochberg FDR under independence. They are printed by the programs next to the simulated values.
5. Monte Carlo simulation: 10,000 series of 40 tests, statistics drawn directly from the normal law — the 32 variants with no effect centred on zero, the 8 true winners shifted by 2.81 standard deviations, the shift that a gain of 1 point brings with 8,200 visitors per arm — the standard deviation used is that of the difference under the alternative, not the pooled one the test puts in its denominator; the gap between the two stays below the simulation error. The simulation serves to check the exact values and to estimate what cannot be computed by hand, never as a source of truth.

Careful: the reference dataset is drawn visitor by visitor, whereas the 10,000 simulated series draw the test statistics directly. The simulation therefore measures the behaviour of the decision rules, not the quality of the z test's normal approximation.

Synthetic data, seed 20260916 — these are not market results.

One test followed end to end. The clearest test of the series is variant 5: 385 conversions out of 8,200 visitors for the current page (4.70%), 497 out of 8,200 for the variant (6.06%). Rate pooled across the two arms: (385 + 497) / 16,400 = 5.38%. The z statistic is (0.0606 − 0.0470) divided by the standard deviation of that difference, that is 3.88, which gives a two-sided p-value of 0.000106 — the smallest of the 40. Decision, rule by rule: with no correction, 0.000106 ≤ 0.05, retained; Bonferroni, 0.000106 ≤ 0.05/40 = 0.00125, retained; Holm, first rank, same level, retained; Benjamini-Hochberg, rank 1, level 1 × 0.05/40 = 0.00125, retained.

The test that separates the rules. The sixth smallest p-value is 0.006. It exceeds the Bonferroni level (0.00125) but stays under the Benjamini-Hochberg level at its rank (6 × 0.05/40 = 0.0075): it is the discovery that only the latter keeps.

Diagnostic of the series. Eight p-values fall below 0.05, where chance alone would produce 1.6: the pile-up near zero signals true effects, and that is what makes a correction necessary rather than optional.

Over 10,000 simulated series, compared with the exact values where they exist (simulation error in brackets):

How to read this: in the reference dataset, Benjamini-Hochberg makes one false discovery out of six. This is not an anomaly: the procedure guarantees an average proportion, not zero error in every series. And that average is not "under 5%" for comfort: under independence it is exactly m₀ q / m = 32 × 0.05 / 40 = 0.040. The simulated value, 0.039, is within one simulation error of the exact value: it is the simulation that is being checked, not the theory.

• Independent tests: in our simulation, each variant has its own control group. That is the condition under which the source establishes its FDR results ^[C11], and under which the exact value of 0.040 is guaranteed. Do your variants share the same control, or the same visitors? If so, the tests are dependent, and the three rules are not in the same boat: the Bonferroni and Holm bounds stay valid whatever the dependence, because they rest only on adding probabilities, whereas the Benjamini-Hochberg guarantee requires an extra condition on the dependence structure.
• Rule fixed before seeing the data: the level q and the list of tests are written down before launch. Did you do that? It is a discipline we impose on ourselves; Storey notes in passing that this obligation is constraining ^[C05], which is what motivated his q-values.
• Complete family: the 40 tests are all corrected together. Did you not remove tests after the fact? An assumption that cannot be tested afterwards: only pre-declaration guarantees it.
• A conservative assumption: the procedure starts from the premise that every variant could be without effect, which makes it stricter than necessary ^[C04]. Do you have an idea, based on your past tests, of the share of variants that really work? If that share is high, q-values recover power. Here, the exact FDR (0.040) is the share of variants with no effect (32/40) multiplied by q.

• The adjusted p-value (Bonferroni, Holm) is the p-value corrected for the number of tests: comparing it with 5% amounts to controlling the probability of making at least one false alarm in the whole family. The q-value answers another question: it gives the expected proportion of false discoveries if this test is declared a winner, together with all those whose p-value is smaller ^[C03]. Two different guarantees, not to be read the same way.
• The Monte Carlo simulation error is published for every figure (the column in brackets), and the exact values are given where they exist.
• Diagnostic to run on one's own data: the distribution of the p-values. A pile-up near 0 signals true effects; a flat distribution, mostly noise. On our reference dataset, 8 p-values fall below 0.05 where 1.6 were expected by chance.

In this precise scenario — 40 tests, 8 true winners, 80% power per test — the rules sort out as follows. With no correction, 80% of the true winners are recovered, but on average nearly one announced victory in five is false. Bonferroni brings the probability of being wrong at least once in the series down to 3.9%, and recovers only a third of the true winners. Holm offers exactly the same guarantee while recovering a few more (0.343 against 0.336): once the aim is to control the risk of being wrong at least once, there is no reason to prefer Bonferroni. Benjamini-Hochberg recovers half of the true winners while holding at 4.0% the average proportion of false victories among the variants retained. Change the number of tests or the share of true winners, and these values change: it is the trade-off that transposes, not the figures.

• Allowed: "With Benjamini-Hochberg at q = 5%, the proportion of false victories among the variants retained does not exceed 5% on average. On a given series it may be higher, or zero."
• Forbidden: "Each variant retained is a winner with 95% certainty." The guarantee bears on the set of discoveries, not on each one.
• Forbidden: transposing these percentages to another campaign without redoing the computation; they depend on the number of tests and on the share of true winners.
• Forbidden: presenting the figures on this page as market results. They come from synthetic data.

• If a false victory is costly (a heavy redesign, a contractual commitment): control the risk of being wrong at least once, with Holm's step-down procedure.
• If this is exploration and every discovery will be revalidated: control the proportion of false discoveries with Benjamini-Hochberg, then confirm the variants retained with a follow-up test.
• The procedure sorts the candidates; the decision to roll out stays human and must weigh the cost and the estimated effect, not only the "significant".

• Use it: several variants, segments, metrics or countries tested at the same time and interpreted together.
• Do not use it alone: a single pre-declared test, which calls for no correction; tests consulted along the way, before the planned end date, which call for a method designed for repeated looks; secondary metrics that are not used to decide.
• Alternatives: the q-values of Storey and Tibshirani, which estimate the share of variants with no effect and gain power ^{[C03, C04]}; hierarchical or pre-selected tests when the family is structured.

• Python 3.14.5, standard library only: python msc-p040-reference.py
• R 4.6.1, base only: Rscript msc-p040-reference.R
• The two programs print exactly the same lines. The sha256 digest of the output, after line-ending normalisation: f7fbde05e16c665db1da621cb9638d90c970a68897e805ba428117536607a9c5. Declared seed 20260916; congruential recursion 1103515245 · x + 12345 mod 2³¹. No third-party library: the software does not validate the method, it reproduces it — and the exact values printed next to the simulated ones make that checkable.

One table per test: p-value, p-value adjusted according to the pre-declared rule, decision, and the list of variants to be confirmed by a follow-up test.

• Storey, J. D. & Tibshirani, R. (2003). Statistical significance for genomewide studies. PNAS 100(16), 9440–9445. ↗ — full text verified ^{[C01–C05, C12]}.
• Storey, J. D. (2002). A direct approach to false discovery rates. JRSS B 64(3), 479–498. ↗ — full text verified ^[C06–C11].
• What these two sources do not cover: the Bonferroni, Holm and Benjamini-Hochberg procedures are implemented here from their definition, written out in full in the "The formal model" section and readable in the programs. Their original articles are not accessible in full text and therefore do not figure among our verified sources: this page rests on no source it has not read. The correctness of the results is checked another way, by the exact values the programs compute and print.
• Evidence level: demonstration on synthetic data, with exact reference values, reproducible in Python and in R. This is not a market result.