Research & Evidence

Search for a method

Search titles, questions, territories and MSC identifiers.

    ← All methods
    METHOD DOSSIERMSC-P-012Experimentation and causalityVerified scientific dossier

    How many observations does an experiment need?

    Sample size depends on baseline rate or variance, MDE, alpha, power, allocation, attrition and clustering. It is calculated before the test and rounded up.

    Scientific editorial team: Marketing Science Center

    Direct answer

    Size a test for a predefined minimum effect.

    Sample size depends on baseline rate or variance, MDE, alpha, power, allocation, attrition and clustering. It is calculated before the test and rounded up.

    Cohen, 1992CONSORT 2010 Explanation and Elaboration

    01

    Operational summary

    Scientific question and scope

    How many observations does an experiment need?

    Supported

    Size a test for a predefined minimum effect.

    Forbidden

    Use observed post-hoc power as evidence.

    02

    Three reading levels

    1. 01

      Decision-maker — Connect the result to a declared decision, useful threshold and error cost.

    2. 02

      Practitioner — Fix population, unit, horizon, available variables and analysis rule before calculation.

    3. 03

      Analyst — Reproduce the calculation, quantify uncertainty and document diagnostics, failures and sensitivities.

    03

    Concrete marketing situation

    Sample size depends on baseline rate or variance, MDE, alpha, power, allocation, attrition and clustering. It is calculated before the test and rounded up.

    04

    Scientific question and scope

    How many observations does an experiment need?

    n per arm for a binary outcome, MDE, α and 1−β; equal allocation, normal approximation

    Required data

    randomization unit per arm

    Population, unit of analysis, origin date and horizon must be declared in the deliverable. Without them, the estimand silently changes.

    05

    Why a simple analysis can fail

    • Use observed post-hoc power as evidence.
    • Is the primary binary outcome and its fixed observation window declared?
    • Are p0 and p1 defensible planning values rather than post hoc observed rates?
    • Does the assignment structure require attrition, unequal allocation or cluster design adjustments?

    06

    Method intuition

    The method does not automatically turn an association into evidence. It links a declared question to an estimand, specification, compatible data and a bounded interpretation rule.

    n per arm for a binary outcome, MDE, α and 1−β; equal allocation, normal approximation

    07

    Required data

    Scientific symbol dictionary

    p0
    Baseline event probability in the comparator arm. Unit: probability · Type: number in (0, 1) · Role: input
    p1
    Target event probability representing the minimum effect the design is powered to detect. Unit: probability · Type: number in (0, 1) · Role: input
    p_bar
    Planning midpoint (p0 + p1) / 2 for the equal-allocation normal approximation. Unit: probability · Type: number · Role: derived
    alpha
    Prespecified two-sided type-I error probability. Unit: probability · Type: number in (0, 1) · Role: input
    power
    Prespecified probability of rejecting the null at the target difference under the planning model. Unit: probability · Type: number in (0, 1) · Role: input
    z_alpha
    Standard-normal quantile 1 − alpha/2, computed from alpha. Unit: dimensionless · Type: positive number · Role: derived
    z_power
    Standard-normal quantile at the declared power, computed from power. Unit: dimensionless · Type: number · Role: derived
    n_raw
    Unrounded normal-approximation sample size per arm before design adjustments. Unit: randomization unit per arm · Type: positive number · Role: derived
    n_planned
    Operational sample size per arm after upward rounding and any separately declared attrition or clustering adjustment. Unit: randomization unit per arm · Type: positive integer · Role: output

    Exact sealed engine inputs

    baseline_rate
    Value: 0.10 · Unit: probability · Type: number · Data status: synthetic
    target_rate
    Value: 0.12 · Unit: probability · Type: number · Data status: synthetic
    alpha_two_sided
    Value: 0.05 · Unit: probability · Type: number · Data status: parameter
    power
    Value: 0.80 · Unit: probability · Type: number · Data status: parameter
    attrition_rate
    Value: 0 · Unit: probability · Type: number · Data status: parameter
    design_effect
    Value: 1 · Unit: dimensionless · Type: number · Data status: parameter

    08

    Formal model

    Formal model

    p_bar = (baseline_rate + target_rate)/2; z_alpha = Φ⁻¹(1−alpha_two_sided/2); z_power = Φ⁻¹(power); n_raw = [z_alpha×sqrt(2p_bar(1−p_bar)) + z_power×sqrt(baseline_rate(1−baseline_rate)+target_rate(1−target_rate))]^2/(target_rate−baseline_rate)^2; n_planned = ceil(n_raw×design_effect/(1−attrition_rate))

    Evidence claims: MSC-P012-C01 · MSC-P012-C02 · MSC-P012-C03 · MSC-P012-C04 · MSC-P012-C05

    Scientific question and scope

    n per arm for a binary outcome, MDE, α and 1−β; equal allocation, normal approximation

    09

    Declared calculation

    1. 01

      Validate binary-outcome rates, two-sided alpha, power, equal allocation and a nonzero target difference.

    2. 02

      Compute both normal quantiles from alpha and power, then evaluate the two-proportion planning formula.

    3. 03

      Round upward and report the approximation scope; apply attrition or design effects only when explicitly supplied.

    10

    Numerical example or application case

    Worked example — synthetic data or declared parameters; no real observations

    Sealed inputs

    • baseline_rate=0.10 [probability]
    • target_rate=0.12 [probability]
    • alpha_two_sided=0.05 [probability]
    • power=0.80 [probability]
    • attrition_rate=0 [probability]
    • design_effect=1 [dimensionless]

    Reproducible results

    • computed_z_alpha=1.959964
    • computed_z_power=0.841621
    • raw_n_per_arm=3840.847482
    • planned_n_per_arm=3841

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    11

    Validity assumptions

    • Is the primary binary outcome and its fixed observation window declared?
    • Are p0 and p1 defensible planning values rather than post hoc observed rates?
    • Does the assignment structure require attrition, unequal allocation or cluster design adjustments?

    12

    Diagnostics and uncertainty

    Diagnostics and uncertainty

    Is the primary binary outcome and its fixed observation window declared? · Are p0 and p1 defensible planning values rather than post hoc observed rates? · Does the assignment structure require attrition, unequal allocation or cluster design adjustments?

    Bounded diagnostic

    Normal approximation with equal allocation; quantiles are computed from alpha and power, then the result is rounded upward.

    Explicit uncertainty contract · design_sensitivity

    Method: Deterministic reference-engine calculation on the sealed input slice.

    Target: Sensitivity of required sample size to prespecified alpha, power, attrition and design effect.

    Engine evidence: metric.raw_n_per_arm= · metric.planned_n_per_arm= · metric.z_alpha_two_sided= · metric.z_power=

    Interpretation: The result is a deterministic planning quantity conditional on all declared design inputs, not a confidence interval for the future effect.

    Stop when

    Use observed post-hoc power as evidence.

    13

    Result interpretation

    • Size a test for a predefined minimum effect.
    • The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.

    14

    Supported and forbidden conclusions

    Supported

    Size a test for a predefined minimum effect.

    Forbidden

    Use observed post-hoc power as evidence.

    15

    Possible marketing decision

    1. 01

      Size a test for a predefined minimum effect.

    2. 02

      The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.

    16

    When to use — when to stop

    Use when

    Size a test for a predefined minimum effect.

    randomization unit per arm

    Stop when

    Use observed post-hoc power as evidence.

    Methodological alternatives

    • Use an exact or simulation-based binary design when the normal approximation is weak.
    • Use a cluster-randomized calculation with intracluster correlation when assignment is clustered.
    • Use sequential or group-sequential design only with a prespecified monitoring rule.

    17

    Reproducibility contract

    Python and R are the executable references. SPSS and SAS remain secondary syntaxes until checked on the same data, specification and diagnostics.

    The engine selects only this page’s explicit slice and branch. Its hash, outputs, and diagnostics remain sealed in the atomic dossier; no generic fallback branch is allowed.

    Slice fingerprint: e2ce0f592a6aa84d9930f38499580ab88972e6eba77945d2b617d51b9f8a8bc1

    Available asset · MSC-T01

    A/B sample size

    Formal model

    p_bar = (baseline_rate + target_rate)/2; z_alpha = Φ⁻¹(1−alpha_two_sided/2); z_power = Φ⁻¹(power); n_raw = [z_alpha×sqrt(2p_bar(1−p_bar)) + z_power×sqrt(baseline_rate(1−baseline_rate)+target_rate(1−target_rate))]^2/(target_rate−baseline_rate)^2; n_planned = ceil(n_raw×design_effect/(1−attrition_rate))

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    18

    Expected final deliverable

    1. 01

      How many observations does an experiment need? — randomization unit per arm

    2. 02

      p0 · p1 · p_bar · alpha · power · z_alpha · z_power · n_raw · n_planned

    3. 03

      p_bar = (baseline_rate + target_rate)/2; z_alpha = Φ⁻¹(1−alpha_two_sided/2); z_power = Φ⁻¹(power); n_raw = [z_alpha×sqrt(2p_bar(1−p_bar)) + z_power×sqrt(baseline_rate(1−baseline_rate)+target_rate(1−target_rate))]^2/(target_rate−baseline_rate)^2; n_planned = ceil(n_raw×design_effect/(1−attrition_rate))

    4. 04

      Is the primary binary outcome and its fixed observation window declared? · Are p0 and p1 defensible planning values rather than post hoc observed rates? · Does the assignment structure require attrition, unequal allocation or cluster design adjustments?

    5. 05

      Size a test for a predefined minimum effect. / Use observed post-hoc power as evidence.

    19

    Scientific sources and evidence status

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    1. MSC-P012-C01 — Power planning covers a difference between two independent proportions.
    2. MSC-P012-C02 — Planning jointly depends on effect size, significance criterion and power.
    3. MSC-P012-C03 — Sample-size determination must name the primary outcome and all quantities used.
    4. MSC-P012-C04 — Attrition and non-compliance adjustments must be reported.
    5. MSC-P012-C05 — Effect sizes support prospective study planning.
    1. Cohen, 1992
    2. CONSORT 2010 Explanation and Elaboration
    3. Lakens, 2013

    Dataset · Tool

    Tool · MSC-T01A/B sample size→

    Method connections

    Parent territoryStatistical decision methods: choose, quantify, validate

    Read next

    MSC-P-011How do you design an A/B test that actually estimates an effect?→MSC-P-013How do you measure campaign incrementality with a control group?→MSC-P-003How should uncertainty in a marketing result be expressed?→