When should you run a marketing experiment?
An experiment is preferred when the question concerns the effect of an action and assignment can be controlled without disproportionate risk. Otherwise, an observational design must state its assumptions.
Scientific editorial team: Marketing Science Center
Direct answer
Choose a design proportionate to the decision and risk.
An experiment is preferred when the question concerns the effect of an action and assignment can be controlled without disproportionate risk. Otherwise, an observational design must state its assumptions.
Jackson et al., 2022Hernán & Robins, Causal Inference: What If
01
Operational summary
Scientific question and scope
When should you run a marketing experiment?
Supported
Choose a design proportionate to the decision and risk.
Forbidden
Demand strong causal claims when no credible variation exists.
02
Three reading levels
- 01
Decision-maker — Connect the result to a declared decision, useful threshold and error cost.
- 02
Practitioner — Fix population, unit, horizon, available variables and analysis rule before calculation.
- 03
Analyst — Reproduce the calculation, quantify uncertainty and document diagnostics, failures and sensitivities.
03
Concrete marketing situation
An experiment is preferred when the question concerns the effect of an action and assignment can be controlled without disproportionate risk. Otherwise, an observational design must state its assumptions.
04
Scientific question and scope
When should you run a marketing experiment?
Local governance screen for whether a causal experiment is worth detailed design; not EVSI/EVPI
Required data
eligible unit × assignment × fixed value horizon
Population, unit of analysis, origin date and horizon must be declared in the deliverable. Without them, the estimand silently changes.
05
Why a simple analysis can fail
- Demand strong causal claims when no credible variation exists.
- Are all monetary quantities evaluated over the same horizon and currency basis?
- Was the valuation scenario fixed before seeing an experiment result?
- Are risk probabilities and consequences explicitly documented rather than embedded in an unexplained score?
06
Method intuition
The method does not automatically turn an association into evidence. It links a declared question to an estimand, specification, compatible data and a bounded interpretation rule.
Local governance screen for whether a causal experiment is worth detailed design; not EVSI/EVPI
07
Required data
Scientific symbol dictionary
local_decision_value- Locally scored monetary value of resolving the declared decision within one prespecified valuation scenario and horizon; not EVSI or EVPI. Unit:
euro over declared horizon· Type:number· Role:input experiment_cost- Total incremental monetary cost of designing, operating and analyzing the experiment on the same horizon. Unit:
euro· Type:nonnegative number· Role:input expected_operational_harm- Expected monetary harm under a declared probability model for operational and ethical risks. Unit:
euro· Type:nonnegative number· Role:input local_screen- Local governance balance used to decide whether detailed experimental design is worth pursuing; not an effect estimate or formal value of information. Unit:
euro· Type:number· Role:output
Exact sealed engine inputs
local_decision_value_eur- Value:
25000· Unit:EUR· Type:number· Data status:synthetic experiment_cost_eur- Value:
12000· Unit:EUR· Type:number· Data status:synthetic operational_risk_eur- Value:
5000· Unit:EUR· Type:number· Data status:synthetic anticipated_gain_pp- Value:
2.0· Unit:percentage_point· Type:number· Data status:synthetic minimum_useful_gain_pp- Value:
1.5· Unit:percentage_point· Type:number· Data status:parameter action_reversible- Value:
1· Unit:binary· Type:integer· Data status:synthetic assignment_feasible- Value:
1· Unit:binary· Type:integer· Data status:synthetic value_horizon_days- Value:
30· Unit:day· Type:integer· Data status:parameter valuation_scenario_prespecified- Value:
1· Unit:binary· Type:integer· Data status:parameter risk_probability_model_declared- Value:
1· Unit:binary· Type:integer· Data status:parameter
08
Formal model
Formal model
local_net_screen_eur = local_decision_value_eur − experiment_cost_eur − operational_risk_eur; passes_local_screen = 1 only if local_net_screen_eur > 0, anticipated_gain_pp >= minimum_useful_gain_pp, action_reversible = 1, assignment_feasible = 1, valuation_scenario_prespecified = 1, risk_probability_model_declared = 1, and value_horizon_days > 0Evidence claims: MSC-P010-C01 · MSC-P010-C02 · MSC-P010-C03 · MSC-P010-C04 · MSC-P010-C05 · MSC-P010-C06 · MSC-P010-C07 · MSC-P010-C08 · MSC-P010-C09
Scientific question and scope
Local governance screen for whether a causal experiment is worth detailed design; not EVSI/EVPI
09
Declared calculation
- 01
Align decision value, cost and harm to one currency, valuation scenario and fixed horizon; require both scenario and risk-model declarations.
- 02
Subtract experiment cost and expected operational harm from the locally scored decision value, then compare anticipated gain with the minimum useful gain.
- 03
Label the output as a non-universal governance screen and prohibit VOI or causal-effect language.
10
Numerical example or application case
Application example — synthetic data or declared parameters; no real observations
Sealed inputs
local_decision_value_eur=25000 [EUR]experiment_cost_eur=12000 [EUR]operational_risk_eur=5000 [EUR]anticipated_gain_pp=2.0 [percentage_point]minimum_useful_gain_pp=1.5 [percentage_point]action_reversible=1 [binary]assignment_feasible=1 [binary]value_horizon_days=30 [day]valuation_scenario_prespecified=1 [binary]risk_probability_model_declared=1 [binary]
Reproducible results
local_net_screen=€8,000passes_local_screen=true
Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.
11
Validity assumptions
- Are all monetary quantities evaluated over the same horizon and currency basis?
- Was the valuation scenario fixed before seeing an experiment result?
- Are risk probabilities and consequences explicitly documented rather than embedded in an unexplained score?
12
Diagnostics and uncertainty
Diagnostics and uncertainty
Are all monetary quantities evaluated over the same horizon and currency basis? · Was the valuation scenario fixed before seeing an experiment result? · Are risk probabilities and consequences explicitly documented rather than embedded in an unexplained score?
Bounded diagnostic
Local, non-universal governance screen only—not value of information and not an effect estimate.
Explicit uncertainty contract · not_estimable_from_sealed_inputs
Method: Explicit non-estimability assessment against the sealed input schema.
Target: Uncertainty in the local value, cost, risk and gain inputs.
Engine evidence: diagnostic=local_nonuniversal_governance_screen_not_voi_or_effect_estimate
Interpretation: The sealed point inputs contain no probability distributions, so this governance screen cannot quantify EVSI, EVPI or decision uncertainty.
Stop when
Demand strong causal claims when no credible variation exists.
13
Result interpretation
- Choose a design proportionate to the decision and risk.
- The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.
14
Supported and forbidden conclusions
Supported
Choose a design proportionate to the decision and risk.
Forbidden
Demand strong causal claims when no credible variation exists.
15
Possible marketing decision
- 01
Choose a design proportionate to the decision and risk.
- 02
The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.
16
When to use — when to stop
Use when
Choose a design proportionate to the decision and risk.
eligible unit × assignment × fixed value horizon
Stop when
Demand strong causal claims when no credible variation exists.
Methodological alternatives
- Use EVPI or EVSI under a fully specified probabilistic decision model.
- Use a qualitative feasibility and ethics gate when monetary valuation would be misleading.
- Run a small pilot only after defining stopping rules and the causal estimand.
17
Reproducibility contract
Python and R are the executable references. SPSS and SAS remain secondary syntaxes until checked on the same data, specification and diagnostics.
The engine selects only this page’s explicit slice and branch. Its hash, outputs, and diagnostics remain sealed in the atomic dossier; no generic fallback branch is allowed.
Slice fingerprint: 9f3be1ae9f83565bb0f6f7351672c8284fe0eaf705c212e574f0545f899f4c14
CSV · CC0
msc-validation-inputs-v1.csv ↓Python · MIT
msc-validation-reference-v1.py ↓R · MIT
msc-validation-reference-v1.R ↓SPSS / SAS · MIT · inspection only
SPSS ↓SAS ↓Formal model
local_net_screen_eur = local_decision_value_eur − experiment_cost_eur − operational_risk_eur; passes_local_screen = 1 only if local_net_screen_eur > 0, anticipated_gain_pp >= minimum_useful_gain_pp, action_reversible = 1, assignment_feasible = 1, valuation_scenario_prespecified = 1, risk_probability_model_declared = 1, and value_horizon_days > 0Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.
18
Expected final deliverable
- 01
When should you run a marketing experiment? —
eligible unit × assignment × fixed value horizon - 02
local_decision_value · experiment_cost · expected_operational_harm · local_screen
- 03
local_net_screen_eur = local_decision_value_eur − experiment_cost_eur − operational_risk_eur; passes_local_screen = 1 only if local_net_screen_eur > 0, anticipated_gain_pp >= minimum_useful_gain_pp, action_reversible = 1, assignment_feasible = 1, valuation_scenario_prespecified = 1, risk_probability_model_declared = 1, and value_horizon_days > 0 - 04
Are all monetary quantities evaluated over the same horizon and currency basis? · Was the valuation scenario fixed before seeing an experiment result? · Are risk probabilities and consequences explicitly documented rather than embedded in an unexplained score?
- 05
Choose a design proportionate to the decision and risk. / Demand strong causal claims when no credible variation exists.
19
Scientific sources and evidence status
Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.
MSC-P010-C01— The causal target contrasts mean potential outcomes under two defined interventions.MSC-P010-C02— Consistency links the observed outcome to the potential outcome under the assigned policy.MSC-P010-C03— The causal estimand assumes one customer's assignment does not alter another customer's outcome.MSC-P010-C04— Fitting a regression does not itself establish a causal design.MSC-P010-C05— Advance specification covers both the research question and the analysis plan before outcomes are observed.MSC-P010-C06— Value-of-information methods support research prioritization and study design.MSC-P010-C07— Formal VoI requires probabilistic parameter uncertainty.MSC-P010-C08— Formal VoI requires decision-specific losses or benefits.MSC-P010-C09— ENBS is based on EVSI, not an arbitrary local score.
Method connections