How do you design an A/B test that actually estimates an effect?
Before the test, specify population, randomization unit, variant, primary metric, MDE, duration, exclusions and intention-to-treat analysis.
Direct answer
Estimate an ITT with known precision for the eligible population.
Before the test, specify population, randomization unit, variant, primary metric, MDE, duration, exclusions and intention-to-treat analysis.
ITT = mean(Y assigned B) − mean(Y assigned A)ESTIMAND
Estimation profile
- Unit of analysis
- Assigned unit × group × periodControlled scientific terminology
randomization unit - Exact estimand
ITT = E[Y|Z=1] − E[Y|Z=0]- Model or deliverable
ITT = mean(Y assigned B) − mean(Y assigned A)
01—08
Verifiable analysis framework
- 01Question
How do you design an A/B test that actually estimates an effect?
- 02Estimand
ITT = E[Y|Z=1] − E[Y|Z=0] · randomization unit
- 03Data
unit_id · assignment · outcome · pre_period · exposure
- 04Model or deliverable
ITT = mean(Y assigned B) − mean(Y assigned A)
- 05Declared calculation
ITT = mean(Y assigned B) − mean(Y assigned A) → ITT = E[Y|Z=1] − E[Y|Z=0]
- 06Uncertainty checks
Sample ratio mismatch · Pre-period balance · Missing outcomes
- 07Method-specific validation
Check data, estimate stability and interpretation limits. Sample ratio mismatch · Pre-period balance · Missing outcomes
- 08Limitations
Change the metric or duration after seeing the result.
Required variables
unit_idassignmentoutcomepre_periodexposure
Checks · Controlled scientific terminology
- Sample ratio mismatch
- Pre-period balance
- Missing outcomes
Estimate an ITT with known precision for the eligible population.
Change the metric or duration after seeing the result.
Scientific sources
1 source
Dataset · Tool
Method connections