Can we stop an A/B test as soon as it becomes significant?

No, and the price of that habit is measurable. On synthetic data where no variant wins, a test read once, at its planned date, concludes wrongly in 4.875% of cases. The accepted risk was 5%, and it is held. The same test, read every day for four weeks and stopped on the first significant day, concludes wrongly in 26.870% of cases. More than one test in four names a winner that does not exist. Reading once a week does not save it: 12.020%. Waiting out the first week and then reading daily does not either: 18.300%.

There is an honest way to look every day. A so-called always-valid boundary stays legitimate whatever the number of readings, and it keeps its promise: 2.000% of false alarms over 84 daily readings, for 5% allowed. It costs time. To reach the power of the fixed-date test, that is, its ability to spot an improvement that really exists, it needs 52 days instead of 28, a factor of 1.86. Three other tunings, published below, go up to 2.39.

A team launches an A/B test on its product page. It receives 250 visitors a day, each assigned at random to one of the two versions with one chance in two, as a real A/B test does. The baseline conversion rate is 5%, and the planned duration four weeks, about 3,500 visitors per version. The dashboard refreshes every morning and shows a p-value: the probability of observing a gap at least as large as the one measured if the two versions were equivalent. The smaller it is, the more improbable the gap would be by chance alone. The temptation is permanent: if it is already significant after nine days, why wait for the planned date? The data on this page are synthetic: we know, by construction, which variants really win, which is what makes it possible to count the errors.

What is the probability that a decision procedure, stopping rule included, concludes there is a difference when no variant improves conversion? And conversely, what power and what delay does that procedure offer when the variant really does improve conversion?

• Estimand, that is, the quantity we are trying to estimate: the probability of concluding there is a difference, computed over the whole procedure, looks included, and not over an isolated test. That probability has a usual name, the size of the test: it is the false-alarm risk the procedure really delivers, as opposed to the 5% risk it displays.
• Population: two-version tests, with constant daily traffic and assignment drawn at random, whose metric is binary.
• Horizon: 28 planned days, monitoring pushed to 84 days to measure what the extra readings cost.

The p-value of a z test, and the 5% threshold it is compared with, assume a single reading, at a date fixed before launch. Read every day, the same statistic fluctuates: it crosses the threshold, falls back, crosses again. The rule "I stop at the first crossing" picks up those passing crossings. The authors of our main source note it in these terms: results are often monitored continuously with methods that assume a fixed sample ^[C02], and that monitoring inflates appreciably the risk of concluding wrongly, what statisticians call the type I error risk ^[C03]. This page measures by how much.

Three ways to decide, on exactly the same data:

1. Fixed date: the duration is chosen in advance, and we look only at the end. The announced risk is the real risk, but there is no stopping early.
2. First significant look: we read every day and stop as soon as the p-value falls below 5%. This is common practice, and this is the one that goes off the rails.
3. Always-valid boundary: we read every day, but compare the cumulative gap with a boundary that rises with the information accumulated, computed so that the probability of crossing it on any day stays below 5%. Such a boundary allows looking whenever one wants and stopping under any rule whatsoever, under the assumptions its authors state and that this page establishes by construction of its design ^[C04]. It comes from a time-uniform concentration framework, not from a fixed-sample inequality ^[C08].

The mathematical object behind the third rule is called a confidence sequence: a sequence of intervals valid uniformly over an unbounded horizon ^[C01]. That freedom has a price, which the authors state themselves: the intervals are wider than those resting on asymptotic approximations, that is, valid only when the sample is large ^[C05].

For each day and each version: the number of visitors and the number of conversions, cumulative. Nothing individual. And two things written down before launch: the planned duration and the decision rule, including how often it will be read. Without them, no guarantee can be computed after the fact.

This section addresses the practitioner. The decision maker can skip it and resume at "The complete worked example": the "In plain terms" sentence that ends this section is enough to read the tables.

• nᶜ, nᵛ: visitors received by the control and by the variant, random since the assignment is drawn at random. Cₜ, Vₜ: their cumulative conversions.
• α: the false-alarm risk accepted for the whole procedure, here 5%.
• Rules 1 and 2, z statistic with pooled variance: zₜ is the gap between the two observed rates, divided by √(p̄ₜ(1 − p̄ₜ)(1/nᶜ + 1/nᵛ)), where p̄ₜ is the rate of the two versions combined. We conclude there is a difference, in one direction or the other since the test is two-sided, if |zₜ| ≥ 1.96 (1.959964 in the programs). For rule 1 this reading takes place only at the planned date, day 28 in our design, and the tables replay the same rule for four different planned durations; for rule 2, on the first day that crosses.
• Rule 3, so-called normal mixture boundary: instead of a single threshold, it averages the thresholds of a family of possible gaps, which makes it valid whatever the real gap. It is read on the conversion clock and not on the visitor clock. We follow the raw gap Sₜ = Vₜ − Cₜ and the total number of conversions Wₜ = Cₜ + Vₜ, and conclude on the first day when

|Sₜ| ≥ √( (Wₜ + ρ) · log( (Wₜ + ρ) / (ρ α²) ) ).

Here log is the natural logarithm. The square on α, and not α alone, comes from the two-sided character of the test. This is the closed form of the two-sided normal mixture boundary of our source ^[C09], applied with an initial likelihood ratio equal to 1, the value the source itself fixes in all scalar cases ^[C14].

• Why this clock, and why the assignment must be drawn at random. Each visitor is assigned to a version by a draw with one chance in two. Under the no-effect hypothesis, the two versions convert at the same rate; a given visitor is therefore a "variant conversion" with probability p/2 and a "control conversion" with the same probability. Conditionally on a conversion, it belongs to the variant with probability exactly one half, independently of the others. The increment of Sₜ is therefore +1 or −1 on a coin flip. Such an increment is said to be sub-Gaussian with variance 1: its extreme deviations do not exceed those of a normal law with variance 1, which is the condition required by the proposition the boundary comes from: it asks that the cumulative gap, in both directions, be sub-Gaussian with the variance process being followed ^[C11]. It holds for any conversion rate, because the inequality cosh(λ) ≤ exp(λ²/2) holds for every real number λ, where λ is the tilting parameter used to compare those two laws.
• What would be false with a fixed size per version. If 125 visitors were imposed on each version, the number of variant conversions would follow, given the total, a hypergeometric law, that is, a draw without replacement: the increments would remain exchangeable among themselves, but would no longer be independent: every conversion attributed takes a place away from the other version, so knowing the first ones tells you about the next. Their conditional variance would fall below 1: on the cumulative gap the boundary actually reads, it would be about 0.95 throughout the horizon, instead of 1. The conclusion would stay conservative, since a draw without replacement is less dispersed, but the coin-flip argument would be false. This is why the assignment is drawn, and why the programs measure those two shares before computing anything.
• ρ fixes where the boundary is tightest, and that is not ρ itself: the source shows that this point sits at about 8.21 times ρ, and that it is this same ratio that gives the ρ minimising the threshold at a targeted date ^[C12]. The programs recover that ratio by a declared numerical search, instead of hard-coding it, and an invariant checks that it is indeed 8.212. The source signals that this tuning is decisive in practice ^[C10]: the page therefore publishes four tunings, including the one the source makes it possible to compute.

In plain terms: rule 2 compares the same gap with the same threshold every day, as if there were only one day. Rule 3 raises the threshold as we look, so that the set of looks, and not each one taken apart, keeps the 5% promise.

1. Invariants measured before any computation, with an error exit if one of them fails: the sub-Gaussianity inequality, over 2,000 values of λ, from 0.01 to 20; the share of visitors assigned to the variant, which must hold within half a point of 50%; the share of conversions falling in the variant, which must hold within one point of 50%; the ratio of the tightest point to the tuning, which must equal 8.212 as the source states; the closed form of the boundary, compared at five points with a different algebraic writing; the consumption of a single random draw per binomial law, checked by comparing the generator's state with one exact step of its recursion; the clock the boundary is read on, verified end to end on a short test whose verdicts are recomputed from the conversion total alone. Measured over 500,000 visitors: 50.090% assigned to the variant, and 50.091% of the 25,288 conversions falling in the variant.
2. Independent check of the boundary itself: 5,000 random walks of 2,000 steps, each step worth +1 or −1 on a coin flip. This is exactly the increment the boundary assumes, and the null hypothesis holds there by construction. The share of walks that cross the boundary must stay below 5%: it is 2.980%, and the program stops if it exceeds that threshold. This check bears on the boundary, not on the experimental design.
3. A second blocking check, on the design itself: 2,000 replications with no effect at all, read every day for 84 days, whose false-alarm rate must stay below 5%. It is 1.750%. And a measurement of the dangerous case: the same boundary applied to a sticky assignment, by blocks of ten visitors, gives 8.780% — beyond the allowed threshold, which shows that the guarantee depends on the design and not on the boundary alone. That measurement is not blocking, it is the warning.
4. Synthetic experiment generated by a pseudo-random number generator written out in the open (seed 20260916; x ← 1103515245 · x + 12345 mod 2³¹), identical in Python and in R.
5. Each day is drawn in three steps: the split of the 250 visitors between the two versions, then the conversions of each one. The three numbers are drawn exactly from the binomial law, by inversion: a number between 0 and 1 is drawn and the value matching it is read off the law, without any approximation.
6. Two scenarios: no winning variant (5% against 5%), and a real improvement of 1.5 point (5% against 6.5%).
7. The rules are applied to the same data. Six reading rhythms are evaluated: daily over 7, 14, 28 and 84 days, weekly over four weeks, and daily from day 7 onward only.
8. Rule 3 is evaluated for four tunings declared a priori: the one the source makes it possible to compute, and three calibrations at days 7, 28 and 56.
9. Exact size of the fixed-date test, computed and not simulated. For a given number of control conversions, the set of variant conversion counts that trigger rejection is the outside of a quadratic, a second-degree expression. The exact probability then follows from the cumulative distribution function of the binomial law, which gives the probability of obtaining at most a given number of conversions. It is read at equal sizes, which is what the design is in expectation. The simulation serves to check that computation, never as a source of truth.
10. 20,000 replications per scenario. The shares are printed as a count and as a rate to five decimals: a rate out of 20,000 is exact to five decimals, whereas at four decimals some values fall exactly halfway and Python and R round them in opposite directions.

Synthetic data, seed 20260916. These are not market results. The percentages are the exact ratio of the counts printed by the programs, given to three decimals, without rounding.

A test with no effect at all, declared conclusive. On the very first attempt, the program lands on a test where the two versions are identical. On day 9, the cumulative gap is −20 conversions and the statistic is −1.9907. The first-significant-look rule stops there and concludes there is a difference, to the variant's disadvantage, which would have been enough to reject it wrongly. Yet on day 28, at the planned date, the statistic is back to −0.6120 and the test does not conclude. The always-valid boundary, for its part, was not crossed over the 28 days simulated. Three rules, two verdicts: the daily reading concludes, the fixed date and the always-valid boundary do not. Only the first is wrong, and it is the only one whose displayed risk is worth nothing.

False alarms, when no variant wins (20,000 replications). In brackets, a simulation standard error, that is, a Monte Carlo standard deviation, and not a confidence interval: it measures the imprecision of the draw, not that of a real test.

Two more reasonable rhythms, over the same four weeks: a weekly reading, four looks, gives 12.020% (± 0.230) of false alarms; a daily reading from day 7, twenty-two looks, gives 18.300% (± 0.273). Spacing the looks out or waiting out the first week reduces the harm without removing it: the risk stays two to four times the one displayed. When the variant really wins, those two rhythms detect in 81.970% (± 0.272) and 85.500% (± 0.249) of cases, against 76.965% for the fixed date: the detection gained is paid for exactly by the false-alarm risk above, and therefore is not a comparison at equal risk.

Exact size of the fixed-date test, computed without simulation, at equal sizes between the two versions, which is what the design is in expectation: 4.9718% at 7 days, 5.0075% at 14 days, 4.9795% at 28 days, 4.9973% at 84 days. The test therefore holds its 5% risk to within three hundredths of a point. The gap between that exact value and the simulated column is 0.7 standard error at day 28, which is ordinary draw noise.

Detections, when the variant really wins by 1.5 point:

Beware of the middle column. The daily reading detects more often, but at the price of a false-alarm risk five and a half times higher: 26.870% against 4.875%. Two procedures are compared on their power only if they hold the same risk. Here, only the fixed date and the always-valid boundary hold it.

The tuning of the boundary, published because it is decisive. Same boundary, same data, four tunings declared in advance. The first is the one the source makes it possible to compute: since the tightest point sits at 8.21 times ρ, targeting the planned date imposes ρ = 350 ÷ 8.21 ≈ 42.6 conversions, and not the 350 expected at that date. The other three follow the opposite instinct, the one that takes for ρ the conversions expected at a date. Since the boundary is in reality tightest 8.21 times further on, that choice tunes it for a date well after the one it is believed to target: day 57 for the tuning called "day 7", day 230 for the day 28 one. The table shows what that instinct costs.

Those equalising days are to be taken to within a day: the day before, the power of the boundary is still 76.185%, 76.475%, 76.630% and 75.855%, for a target of 76.965% itself known to within 0.298 point.

When the variant really wins, the reference boundary concludes on day 31 at the median, over the 95.255% of cases where it concludes within 84 days. The daily reading stops on day 8 at the median, over the 87.090% of cases where it concludes within 28 days. The two medians therefore bear on different windows and do not compare term by term. And that same haste, applied to the no-effect scenario, is what produces 26.870% of false alarms there.

• Assignment drawn at random, with one chance in two, visitor by visitor. This is the load-bearing assumption: it is what makes every conversion fall on a coin flip and makes the boundary legitimate. Does your tool really split at random, and does it do so visitor by visitor rather than by fixed quotas? Legitimacy comes from the construction of the design, not from the measurement: the two shares the programs check, with a tolerance of half a point over 500,000 visitors, detect only a gross splitting error. They cannot see a dependence between visitors.
• Assignment independent from one visitor to the next, and not sticky. This is the most fragile assumption in practice, and the most dangerous: an assignment by cookie or by session keeps the same version for all of a person's visits. The shares stay at one half on average, the programs' checks go green, and the guarantee falls. We measured it, with blocks of ten visitors sharing one version: 8.780% of false alarms for 5% promised, nearly double. Does your tool assign every visit at random, or does it stick a version to each visitor?
• A single possible conversion per visitor. Do your visitors come back several times, or share a device?
• Stable traffic and baseline rate: our synthetic experiment has no seasonality, no peak day, no drift of the base rate. Does your weekend traffic look like your Tuesday traffic? A drift common to both versions does not bias the comparison; a drift specific to one version does.
• A single metric, a single test. How many variants and how many metrics does your dashboard display side by side? As soon as there are several, read together, the problem changes in nature and calls for a multiple-testing correction.
• Stopping rule declared before launch. An assumption that cannot be checked after the fact: only a written pre-declaration guarantees it.
• Finite monitoring horizon. The 2.000% is a rate over 84 readings. The guarantee invoked holds over an unbounded horizon, but the measured figure grows with the number of readings and must not be read as the level of the boundary.
• Data drawn from the very model the test assumes: independent conversions, constant rates, balanced assignment. An assumption true by construction here, and therefore favourable to the three rules. On real data, departures from that model add to the figures on this page, they do not subtract from them.

• The Monte Carlo simulation error is published next to every rate in the three result tables; the threshold column, for its part, is deterministic and has none. Over 20,000 replications it is about 0.15 point for a rate of 5% and 0.31 point for a rate of 27%. The page's main contrast, 4.875% against 26.870%, is 22 points: it is out of all proportion to those standard errors.
• The computed exact size, for its part, depends on no draw. It is what the page relies on to state that the fixed-date test holds its risk, and not the simulated figure, whose gap to 5% would not have been conclusive.
• The always-valid boundary is conservative: it guarantees 5% and delivers 2.000% with the reference tuning, less still with the late tunings. That is the price of a guarantee that assumes nothing about the way of looking. And it is the price of this boundary: the same source offers another one, tighter when the increments are bounded, as they are here ^[C13]. We did not implement it, so the factor 1.86 is the cost of the normal mixture at this tuning, not the cost of always-validity in general.
• Its price is measured, and is not deduced from the threshold. The boundary's threshold is 3.1140 in z units, against 1.959964 for the fixed-date test, that is 1.589 times more; the cost in duration, for its part, is 1.86. The two do not convert into one another, because the boundary's threshold falls as information accumulates and because all the intermediate readings count. The authors do bound the price in width at less than a doubling ^[C06], but for another of their boundaries than the one used here, and our latest tuning reaches 2.293 times the fixed threshold in z units: we therefore claim no conformity to that statement, only the measurements above.
• What this simulation cannot settle: a single traffic level (250 visitors a day), a single baseline rate (5%), a single real improvement (1.5 point). With less traffic or a lower rate, the passing crossings are more frequent still; nothing here says by how much.
• Diagnostic to run on one's own tests: count the readings, and check that the split between versions really is drawn at random. If nobody knows how many times the dashboard was consulted before the decision, the displayed risk is meaningless.

The 5% threshold is not a property of the p-value: it is a property of the procedure that uses it. On our synthetic experiment, reading every day and stopping at the first crossing turns a 5% risk into a risk of 26.870% over four weeks, and of 35.385% over twelve. Reading once a week brings it back to 12.020%, which is still more than double the displayed risk. On another traffic level those values will change; the mechanism will not. The inflation does not come from a flaw in the test but from the stopping rule, which selects the favourable days.

The remedy exists and works, but it is not free. An always-valid boundary allows daily reading and holds its guarantee, at the price of a duration lengthened by a factor of 1.86 with the tuning the source makes it possible to compute, and up to 2.39 with a badly chosen tuning. Another boundary from the same source, tighter for bounded increments, would do better; this page does not quantify it ^[C13]. Between the two, the most economical solution remains the oldest: fix the duration in advance and decide only at the end.

• Allowed: "With a duration fixed in advance and a single reading, our risk of concluding wrongly is 5%; the exact size of the test, computed at equal sizes, is 4.9795% at the planned date."
• Allowed: "If we want to read the test every day while keeping that risk, we need an always-valid boundary, and we must plan, on this experimental design, a duration 1.86 times longer with the tuning computed according to the source, for the same power."
• Forbidden: "The test is significant today, therefore the variant wins." Without a declared stopping rule, that sentence has no risk attached to it.
• Forbidden: transposing the 26.870% to another test, nor the duration factor. Both depend on the number of readings, on daily traffic, on the real effect and on the tuning of the boundary. It is the mechanism that transposes, not the values.
• Forbidden: applying this boundary to a test whose assignment is sticky, by cookie or by session. Measured here: 8.780% of false alarms for 5% promised. That is the dangerous direction, and none of the programs' checks detects it.
• Forbidden: reading the two shares measured by the programs as proof that the boundary is legitimate. They detect only a gross splitting error; legitimacy comes from the construction of the design.
• Forbidden: presenting these figures as market results. The data are synthetic, and drawn from the very model the test assumes.

Write down, before launch: the duration, the decision metric and the reading frequency. Then choose with full knowledge. For a test whose stakes justify a quick decision, plan an always-valid boundary, tune it by computation rather than by eye, and budget the duration accordingly. For a routine test, close the dashboard until the planned date. And if someone has already looked ten times, say so: the decision stays human, but it must be taken with the real risk in view, not with the one the tool displays.

• Use it: as soon as a test can be consulted before its end date, which is the case for almost every online test, including when the intermediate look is a single one — a case the mechanism covers but that this page does not quantify, the most frugal rhythm it measures counting four looks.
• Do not use it alone: with several variants or several metrics read together, where a multiple-testing correction is needed as well; with very irregular traffic, a split fixed instead of drawn at random, or a base rate that drifts, where the comparison itself must be reconsidered.
• Alternatives, both outside the scope of our sources and therefore stated without an evidence tag:

• group sequential designs reserve a share of the risk for each intermediate analysis planned in the protocol. They suit the case where the number of readings is known in advance, where the boundary on this page assumes nothing.
• Bayesian approaches formulate their guarantees differently. They do not compare term by term with an error rate.

• Python 3.14.5, standard library only: msc-p048-reference.py, run with python msc-p048-reference.py
• R 4.6.1, base only: msc-p048-reference.R, run with Rscript msc-p048-reference.R
• The two programs print exactly the same lines. The sha256 digest of the output, after line-ending normalisation, is recorded in manifest.json. Declared seed 20260916, 20,000 replications per scenario.
• Four precautions were necessary for the two languages to print the same thing, and they are in the code. The first term of the binomial law is built by repeated multiplications rather than by a power, whose last bit differs between Python and R. The generator's multiplier is split into two halves in the R version, which has no integers wide enough to hold the exact product. The rates are printed to five decimals, where a rate out of 20,000 is exact. And the computation of the per-version sizes is parenthesised, because R's integer division binds more tightly than multiplication.
• Reproducing the day 9 example requires respecting the order in which the generator is consumed: invariants measured, search for the example, no-effect scenario, then improvement scenario. The check of the boundary by random walks consumes a separate stream, reset to the seed.

For a given test: the duration and the decision rule written down before launch, the number of readings actually made, the verdict according to the declared rule, and, if the test was read several times, the real risk of the procedure followed rather than that of the fixed-date test.

• Howard, S. R., Ramdas, A., McAuliffe, J. & Sekhon, J. (2018). Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics 49(2), 1055–1080 (2021). ↗ — full text verified on the open-access arXiv version, arXiv:1810.08240v6 ^{[C01, C02, C03, C04, C05, C06, C09, C10, C11, C12, C13, C14]}.
• Howard, S. R., Ramdas, A., McAuliffe, J. & Sekhon, J. (2018). Exponential line-crossing inequalities, the arXiv document title verified here; published under the title Time-uniform Chernoff bounds via nonnegative supermartingales, Probability Surveys 17, 257–317 (2020). ↗ — full text verified on the open-access arXiv version, arXiv:1808.03204v6 ^{[C07, C08]}.
• The boundary and its attribution: the form used here is the two-sided normal mixture boundary of the first source. Its authors trace the mixture method to a lineage going back to Ville and to Wald, then to Darling and Robbins. We therefore cite it to the source we have read, and not to a text we do not hold. The choice of the initial likelihood ratio is our reading, and not a verified excerpt.
• Evidence level: demonstration on synthetic data, two scenarios, six reading rhythms, four tunings, 20,000 replications. The exact size of the test is computed alongside the simulated values; the boundary is checked separately on random walks where the null hypothesis holds by construction; the assumption that makes it legitimate is established by construction of the design, and two aggregate shares are checked before any computation to detect a gross splitting error. The dangerous case, sticky assignment, is measured apart. Reproducible in Python and in R. This is not a market result.