How many observations does an experiment need?
Sample size depends on baseline rate or variance, MDE, alpha, power, allocation, attrition and clustering. It is calculated before the test and rounded up.
Scientific editorial team : Marketing Science Center
Direct answer
Size a test for a predefined minimum effect.
Sample size depends on baseline rate or variance, MDE, alpha, power, allocation, attrition and clustering. It is calculated before the test and rounded up.
01
Operational summary
Scientific question and scope
How many observations does an experiment need?
Supported
Size a test for a predefined minimum effect.
Forbidden
Use observed post-hoc power as evidence.
02
Three reading levels
- 01
Decision-maker — Connect the result to a declared decision, useful threshold and error cost.
- 02
Practitioner — Fix population, unit, horizon, available variables and analysis rule before calculation.
- 03
Analyst — Reproduce the calculation, quantify uncertainty and document diagnostics, failures and sensitivities.
03
Concrete marketing situation
Sample size depends on baseline rate or variance, MDE, alpha, power, allocation, attrition and clustering. It is calculated before the test and rounded up.
04
Scientific question and scope
How many observations does an experiment need?
n per arm for a binary outcome, MDE, α and 1−β; equal allocation, normal approximation
Required data
randomization unit per arm
Population, unit of analysis, origin date and horizon must be declared in the deliverable. Without them, the estimand silently changes.
05
Why a simple analysis can fail
- Use observed post-hoc power as evidence.
- Two-sided binary-outcome approximation
- Equal allocation
- Round upward and adjust for attrition and clustering
06
Method intuition
The method does not automatically turn an association into evidence. It links a declared question to an estimand, specification, compatible data and a bounded interpretation rule.
n per arm for a binary outcome, MDE, α and 1−β; equal allocation, normal approximation
07
Required data
Scientific symbol dictionary
p0- Baseline event probability in the comparator arm. Unit:
probability· Type:number in (0, 1)· Role:input p1- Target event probability representing the minimum effect the design is powered to detect. Unit:
probability· Type:number in (0, 1)· Role:input p_bar- Planning midpoint (p0 + p1) / 2 for the equal-allocation normal approximation. Unit:
probability· Type:number· Role:derived alpha- Prespecified two-sided type-I error probability. Unit:
probability· Type:number in (0, 1)· Role:input power- Prespecified probability of rejecting the null at the target difference under the planning model. Unit:
probability· Type:number in (0, 1)· Role:input z_alpha- Standard-normal quantile 1 − alpha/2, computed from alpha. Unit:
dimensionless· Type:positive number· Role:derived z_power- Standard-normal quantile at the declared power, computed from power. Unit:
dimensionless· Type:number· Role:derived n_raw- Unrounded normal-approximation sample size per arm before design adjustments. Unit:
randomization unit per arm· Type:positive number· Role:derived n_planned- Operational sample size per arm after upward rounding and any separately declared attrition or clustering adjustment. Unit:
randomization unit per arm· Type:positive integer· Role:output
Exact sealed engine inputs
baseline_rate- Value:
0.10· Unit:probability· Type:number· Data status:synthetic target_rate- Value:
0.12· Unit:probability· Type:number· Data status:synthetic alpha_two_sided- Value:
0.05· Unit:probability· Type:number· Data status:parameter power- Value:
0.80· Unit:probability· Type:number· Data status:parameter attrition_rate- Value:
0· Unit:probability· Type:number· Data status:parameter design_effect- Value:
1· Unit:dimensionless· Type:number· Data status:parameter
08
Formal model
Formal model
p_bar = (baseline_rate + target_rate)/2; z_alpha = Φ⁻¹(1−alpha_two_sided/2); z_power = Φ⁻¹(power); n_raw = [z_alpha×sqrt(2p_bar(1−p_bar)) + z_power×sqrt(baseline_rate(1−baseline_rate)+target_rate(1−target_rate))]^2/(target_rate−baseline_rate)^2; n_planned = ceil(n_raw×design_effect/(1−attrition_rate))Evidence claims: MSC-P012-C01 · MSC-P012-C02 · MSC-P012-C03 · MSC-P012-C04 · MSC-P012-C05
Scientific question and scope
n per arm for a binary outcome, MDE, α and 1−β; equal allocation, normal approximation
09
Declared calculation
- 01
Validate binary-outcome rates, two-sided alpha, power, equal allocation and a nonzero target difference.
- 02
Compute both normal quantiles from alpha and power, then evaluate the two-proportion planning formula.
- 03
Round upward and report the approximation scope; apply attrition or design effects only when explicitly supplied.
10
Numerical example or application case
Worked example — synthetic data or declared parameters; no real observations
Sealed inputs
baseline_rate=0.10 [probability]target_rate=0.12 [probability]alpha_two_sided=0.05 [probability]power=0.80 [probability]attrition_rate=0 [probability]design_effect=1 [dimensionless]
Reproducible results
computed_z_alpha=1.959964computed_z_power=0.841621raw_n_per_arm=3840.847482planned_n_per_arm=3841
Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.
11
Validity assumptions
- Is the primary binary outcome and its fixed observation window declared?
- Are p0 and p1 defensible planning values rather than post hoc observed rates?
- Does the assignment structure require attrition, unequal allocation or cluster design adjustments?
12
Diagnostics and uncertainty
Diagnostics and uncertainty
Two-sided binary-outcome approximation · Equal allocation · Round upward and adjust for attrition and clustering
Bounded diagnostic
Normal approximation with equal allocation; quantiles are computed from alpha and power, then the result is rounded upward.
Explicit uncertainty contract · design_sensitivity
Method : Deterministic reference-engine calculation on the sealed input slice.
Target : Sensitivity of required sample size to prespecified alpha, power, attrition and design effect.
Engine evidence : metric.raw_n_per_arm= · metric.planned_n_per_arm= · metric.z_alpha_two_sided= · metric.z_power=
Interpretation : The result is a deterministic planning quantity conditional on all declared design inputs, not a confidence interval for the future effect.
Stop when
Use observed post-hoc power as evidence.
13
Result interpretation
- Size a test for a predefined minimum effect.
- The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.
14
Supported and forbidden conclusions
Supported
Size a test for a predefined minimum effect.
Forbidden
Use observed post-hoc power as evidence.
15
Possible marketing decision
- 01
Size a test for a predefined minimum effect.
- 02
The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.
16
When to use — when to stop
Use when
Size a test for a predefined minimum effect.
randomization unit per arm
Stop when
Use observed post-hoc power as evidence.
Methodological alternatives
- Use an exact or simulation-based binary design when the normal approximation is weak.
- Use a cluster-randomized calculation with intracluster correlation when assignment is clustered.
- Use sequential or group-sequential design only with a prespecified monitoring rule.
17
Reproducibility contract
Python and R are the executable references. SPSS and SAS remain secondary syntaxes until checked on the same data, specification and diagnostics.
The engine selects only this page’s explicit slice and branch. Its hash, outputs, and diagnostics remain sealed in the atomic dossier; no generic fallback branch is allowed.
Slice fingerprint: e2ce0f592a6aa84d9930f38499580ab88972e6eba77945d2b617d51b9f8a8bc1
Available asset · MSC-T01
A/B sample size
CSV · CC0
msc-validation-inputs-v1.csv ↓Python · MIT
msc-validation-reference-v1.py ↓R · MIT
msc-validation-reference-v1.R ↓SPSS / SAS · MIT · inspection only
SPSS ↓SAS ↓Formal model
p_bar = (baseline_rate + target_rate)/2; z_alpha = Φ⁻¹(1−alpha_two_sided/2); z_power = Φ⁻¹(power); n_raw = [z_alpha×sqrt(2p_bar(1−p_bar)) + z_power×sqrt(baseline_rate(1−baseline_rate)+target_rate(1−target_rate))]^2/(target_rate−baseline_rate)^2; n_planned = ceil(n_raw×design_effect/(1−attrition_rate))Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.
18
Expected final deliverable
- 01
How many observations does an experiment need? —
randomization unit per arm - 02
p0 · p1 · p_bar · alpha · power · z_alpha · z_power · n_raw · n_planned
- 03
p_bar = (baseline_rate + target_rate)/2; z_alpha = Φ⁻¹(1−alpha_two_sided/2); z_power = Φ⁻¹(power); n_raw = [z_alpha×sqrt(2p_bar(1−p_bar)) + z_power×sqrt(baseline_rate(1−baseline_rate)+target_rate(1−target_rate))]^2/(target_rate−baseline_rate)^2; n_planned = ceil(n_raw×design_effect/(1−attrition_rate)) - 04
Two-sided binary-outcome approximation · Equal allocation · Round upward and adjust for attrition and clustering
- 05
Size a test for a predefined minimum effect. / Use observed post-hoc power as evidence.
19
Scientific sources and evidence status
Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.
MSC-P012-C01— Power planning covers a difference between two independent proportions.MSC-P012-C02— Planning jointly depends on effect size, significance criterion and power.MSC-P012-C03— Sample-size determination must name the primary outcome and all quantities used.MSC-P012-C04— Attrition and non-compliance adjustments must be reported.MSC-P012-C05— Effect sizes support prospective study planning.
Dataset · Tool
Method connections