Research & Evidence

Search for a method

Search titles, questions, territories and MSC identifiers.

    ← All methods
    METHOD DOSSIERMSC-P-011Experimentation and causalityVerified scientific dossier

    How do you design an A/B test that actually estimates an effect?

    Before the test, specify population, randomization unit, variant, primary metric, MDE, duration, exclusions and intention-to-treat analysis.

    Scientific editorial team: Marketing Science Center

    Direct answer

    Estimate an ITT with known precision for the eligible population.

    Before the test, specify population, randomization unit, variant, primary metric, MDE, duration, exclusions and intention-to-treat analysis.

    Kohavi et al., 2009

    01

    Direct answer

    A credible A/B test is sized before exposure using one primary metric, a minimum detectable effect, alpha, power, randomization unit and attrition. For 8.0% versus 8.7%, MSC-P-011 requires 24,516 units per arm, or 31,326 after 10% attrition and a 1.15 design effect.

    02

    Scientific question

    How many units per arm must be randomized to detect a prespecified absolute difference in a binary outcome at chosen error rates?

    03

    Population, unit and horizon

    The unit is the eligible randomization unit analysed in its assigned arm over a fixed window. User, household, store and geography are not interchangeable.

    04

    Estimand and metric

    The primary estimand is the absolute ITT difference p₁−p₀ for a binary conversion. The 0.7-point MDE, or 8.75% relative, is a decision choice, not a predicted effect.

    05

    Required data

    Declare eligible population, credible baseline rate, MDE, primary metric and window, alpha, power, allocation ratio, attrition, clustering, contamination and traffic constraints.

    06

    Assumptions

    Assume two independent arms, 1:1 allocation, binary outcome, two-sided test, adequate normal approximation, ITT analysis and no unit reuse. Clustering and repeated measures need adapted calculations.

    07

    Planning formula

    n≈[z₁₋α/2√(2p̄(1−p̄))+z₁₋β√(p₀(1−p₀)+p₁(1−p₁))]²/(p₁−p₀)², where p̄=(p₀+p₁)/2.

    08

    Reproducible calculation

    With p₀=0.08, p₁=0.087, α=0.05 and power 0.80: z=1.959964 and 0.841621. The script calculates 24,515.438 then rounds upward.

    09

    Planning results

    Raw size: 24,516 units per arm. After a 1.15 design effect and 10% attrition: 31,326 per arm, 62,652 total.

    MSC-P-011 · two-arm binary design
    BaselineTargetRaw / armAdjusted / armTotal
    8.0%8.7%24,51631,32662,652

    10

    Planning uncertainty

    Sample size is sensitive to baseline rate and MDE. Recalculate over plausible ranges, round upward and treat traffic and attrition uncertainty as feasibility risk.

    11

    Pre-launch protocol

    Freeze population, unit, variants, primary metric, window, MDE, size, stopping rule, exclusions, ITT analysis, missing-data handling and sample-ratio checks.

    12

    Diagnostics

    After launch check assignment integrity, sample-ratio mismatch, contamination, differential attrition, outcome availability, exposure and change log.

    13

    Common errors

    Common errors are choosing MDE to obtain a small sample, peeking, multiplying metrics, randomizing users but analysing sessions, stopping at p<0.05 or excluding after assignment.

    14

    Interpretation

    Power of 80% means an 80% probability of rejecting H₀ under the exact planning effect and assumptions, not an 80% chance the variant is better.

    15

    Supported decision

    Decide whether the test is feasible, which MDE is decision-useful and how much traffic and time to reserve before launch.

    16

    Unsupported decision

    Calculated size proves neither absence of bias, metric relevance nor a real effect. It cannot justify causal conclusions when randomization or ITT is compromised.

    17

    Implementation

    The CC0 CSV documents the design; MIT Python uses the standard normal distribution, rejects invalid parameters, rounds upward and reports raw and adjusted size.

    18

    Expected deliverable

    Deliver question, population, unit, variants, metric and window, absolute and relative MDE, alpha, power, formula, adjustments, traffic, duration, diagnostics, analysis and decision rule.

    19

    Scientific sources

    Cohen supports power reasoning; CONSORT requires reporting sample-size determination, outcomes, randomization, numbers analysed and precision. Neither source predicts the synthetic 8.7% rate.

    1. Cohen (1992) ↗Full text verified
    2. Moher et al. (2010) ↗Full text verified

    Dataset · Tool

    Tool · MSC-T01A/B sample size→

    Method connections

    Parent territoryMeasurement and causality: how can a marketing effect be established?

    Read next

    MSC-P-012How many observations does an experiment need?→MSC-P-013How do you measure campaign incrementality with a control group?→MSC-P-017How do you detect selection, contamination and attrition in an experiment?→