Research & Evidence

Search for a method

Search titles, questions, territories and MSC identifiers.

    ← All methods
    METHOD DOSSIERMSC-P-010Experimentation and causalityVerified scientific dossier

    When should you run a marketing experiment?

    An experiment is preferred when the question concerns the effect of an action and assignment can be controlled without disproportionate risk. Otherwise, an observational design must state its assumptions.

    Scientific editorial team: Marketing Science Center

    Direct answer

    Choose a design proportionate to the decision and risk.

    An experiment is preferred when the question concerns the effect of an action and assignment can be controlled without disproportionate risk. Otherwise, an observational design must state its assumptions.

    Jackson et al., 2022Hernán & Robins, Causal Inference: What If

    01

    Operational summary

    Scientific question and scope

    When should you run a marketing experiment?

    Supported

    Choose a design proportionate to the decision and risk.

    Forbidden

    Demand strong causal claims when no credible variation exists.

    02

    Three reading levels

    1. 01

      Decision-maker — Connect the result to a declared decision, useful threshold and error cost.

    2. 02

      Practitioner — Fix population, unit, horizon, available variables and analysis rule before calculation.

    3. 03

      Analyst — Reproduce the calculation, quantify uncertainty and document diagnostics, failures and sensitivities.

    03

    Concrete marketing situation

    An experiment is preferred when the question concerns the effect of an action and assignment can be controlled without disproportionate risk. Otherwise, an observational design must state its assumptions.

    04

    Scientific question and scope

    When should you run a marketing experiment?

    Local governance screen for whether a causal experiment is worth detailed design; not EVSI/EVPI

    Required data

    eligible unit × assignment × fixed value horizon

    Population, unit of analysis, origin date and horizon must be declared in the deliverable. Without them, the estimand silently changes.

    05

    Why a simple analysis can fail

    • Demand strong causal claims when no credible variation exists.
    • Are all monetary quantities evaluated over the same horizon and currency basis?
    • Was the valuation scenario fixed before seeing an experiment result?
    • Are risk probabilities and consequences explicitly documented rather than embedded in an unexplained score?

    06

    Method intuition

    The method does not automatically turn an association into evidence. It links a declared question to an estimand, specification, compatible data and a bounded interpretation rule.

    Local governance screen for whether a causal experiment is worth detailed design; not EVSI/EVPI

    07

    Required data

    Scientific symbol dictionary

    local_decision_value
    Locally scored monetary value of resolving the declared decision within one prespecified valuation scenario and horizon; not EVSI or EVPI. Unit: euro over declared horizon · Type: number · Role: input
    experiment_cost
    Total incremental monetary cost of designing, operating and analyzing the experiment on the same horizon. Unit: euro · Type: nonnegative number · Role: input
    expected_operational_harm
    Expected monetary harm under a declared probability model for operational and ethical risks. Unit: euro · Type: nonnegative number · Role: input
    local_screen
    Local governance balance used to decide whether detailed experimental design is worth pursuing; not an effect estimate or formal value of information. Unit: euro · Type: number · Role: output

    Exact sealed engine inputs

    local_decision_value_eur
    Value: 25000 · Unit: EUR · Type: number · Data status: synthetic
    experiment_cost_eur
    Value: 12000 · Unit: EUR · Type: number · Data status: synthetic
    operational_risk_eur
    Value: 5000 · Unit: EUR · Type: number · Data status: synthetic
    anticipated_gain_pp
    Value: 2.0 · Unit: percentage_point · Type: number · Data status: synthetic
    minimum_useful_gain_pp
    Value: 1.5 · Unit: percentage_point · Type: number · Data status: parameter
    action_reversible
    Value: 1 · Unit: binary · Type: integer · Data status: synthetic
    assignment_feasible
    Value: 1 · Unit: binary · Type: integer · Data status: synthetic
    value_horizon_days
    Value: 30 · Unit: day · Type: integer · Data status: parameter
    valuation_scenario_prespecified
    Value: 1 · Unit: binary · Type: integer · Data status: parameter
    risk_probability_model_declared
    Value: 1 · Unit: binary · Type: integer · Data status: parameter

    08

    Formal model

    Formal model

    local_net_screen_eur = local_decision_value_eur − experiment_cost_eur − operational_risk_eur; passes_local_screen = 1 only if local_net_screen_eur > 0, anticipated_gain_pp >= minimum_useful_gain_pp, action_reversible = 1, assignment_feasible = 1, valuation_scenario_prespecified = 1, risk_probability_model_declared = 1, and value_horizon_days > 0

    Evidence claims: MSC-P010-C01 · MSC-P010-C02 · MSC-P010-C03 · MSC-P010-C04 · MSC-P010-C05 · MSC-P010-C06 · MSC-P010-C07 · MSC-P010-C08 · MSC-P010-C09

    Scientific question and scope

    Local governance screen for whether a causal experiment is worth detailed design; not EVSI/EVPI

    09

    Declared calculation

    1. 01

      Align decision value, cost and harm to one currency, valuation scenario and fixed horizon; require both scenario and risk-model declarations.

    2. 02

      Subtract experiment cost and expected operational harm from the locally scored decision value, then compare anticipated gain with the minimum useful gain.

    3. 03

      Label the output as a non-universal governance screen and prohibit VOI or causal-effect language.

    10

    Numerical example or application case

    Application example — synthetic data or declared parameters; no real observations

    Sealed inputs

    • local_decision_value_eur=25000 [EUR]
    • experiment_cost_eur=12000 [EUR]
    • operational_risk_eur=5000 [EUR]
    • anticipated_gain_pp=2.0 [percentage_point]
    • minimum_useful_gain_pp=1.5 [percentage_point]
    • action_reversible=1 [binary]
    • assignment_feasible=1 [binary]
    • value_horizon_days=30 [day]
    • valuation_scenario_prespecified=1 [binary]
    • risk_probability_model_declared=1 [binary]

    Reproducible results

    • local_net_screen=€8,000
    • passes_local_screen=true

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    11

    Validity assumptions

    • Are all monetary quantities evaluated over the same horizon and currency basis?
    • Was the valuation scenario fixed before seeing an experiment result?
    • Are risk probabilities and consequences explicitly documented rather than embedded in an unexplained score?

    12

    Diagnostics and uncertainty

    Diagnostics and uncertainty

    Are all monetary quantities evaluated over the same horizon and currency basis? · Was the valuation scenario fixed before seeing an experiment result? · Are risk probabilities and consequences explicitly documented rather than embedded in an unexplained score?

    Bounded diagnostic

    Local, non-universal governance screen only—not value of information and not an effect estimate.

    Explicit uncertainty contract · not_estimable_from_sealed_inputs

    Method: Explicit non-estimability assessment against the sealed input schema.

    Target: Uncertainty in the local value, cost, risk and gain inputs.

    Engine evidence: diagnostic=local_nonuniversal_governance_screen_not_voi_or_effect_estimate

    Interpretation: The sealed point inputs contain no probability distributions, so this governance screen cannot quantify EVSI, EVPI or decision uncertainty.

    Stop when

    Demand strong causal claims when no credible variation exists.

    13

    Result interpretation

    • Choose a design proportionate to the decision and risk.
    • The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.

    14

    Supported and forbidden conclusions

    Supported

    Choose a design proportionate to the decision and risk.

    Forbidden

    Demand strong causal claims when no credible variation exists.

    15

    Possible marketing decision

    1. 01

      Choose a design proportionate to the decision and risk.

    2. 02

      The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.

    16

    When to use — when to stop

    Use when

    Choose a design proportionate to the decision and risk.

    eligible unit × assignment × fixed value horizon

    Stop when

    Demand strong causal claims when no credible variation exists.

    Methodological alternatives

    • Use EVPI or EVSI under a fully specified probabilistic decision model.
    • Use a qualitative feasibility and ethics gate when monetary valuation would be misleading.
    • Run a small pilot only after defining stopping rules and the causal estimand.

    17

    Reproducibility contract

    Python and R are the executable references. SPSS and SAS remain secondary syntaxes until checked on the same data, specification and diagnostics.

    The engine selects only this page’s explicit slice and branch. Its hash, outputs, and diagnostics remain sealed in the atomic dossier; no generic fallback branch is allowed.

    Slice fingerprint: 9f3be1ae9f83565bb0f6f7351672c8284fe0eaf705c212e574f0545f899f4c14

    Formal model

    local_net_screen_eur = local_decision_value_eur − experiment_cost_eur − operational_risk_eur; passes_local_screen = 1 only if local_net_screen_eur > 0, anticipated_gain_pp >= minimum_useful_gain_pp, action_reversible = 1, assignment_feasible = 1, valuation_scenario_prespecified = 1, risk_probability_model_declared = 1, and value_horizon_days > 0

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    18

    Expected final deliverable

    1. 01

      When should you run a marketing experiment? — eligible unit × assignment × fixed value horizon

    2. 02

      local_decision_value · experiment_cost · expected_operational_harm · local_screen

    3. 03

      local_net_screen_eur = local_decision_value_eur − experiment_cost_eur − operational_risk_eur; passes_local_screen = 1 only if local_net_screen_eur > 0, anticipated_gain_pp >= minimum_useful_gain_pp, action_reversible = 1, assignment_feasible = 1, valuation_scenario_prespecified = 1, risk_probability_model_declared = 1, and value_horizon_days > 0

    4. 04

      Are all monetary quantities evaluated over the same horizon and currency basis? · Was the valuation scenario fixed before seeing an experiment result? · Are risk probabilities and consequences explicitly documented rather than embedded in an unexplained score?

    5. 05

      Choose a design proportionate to the decision and risk. / Demand strong causal claims when no credible variation exists.

    19

    Scientific sources and evidence status

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    1. MSC-P010-C01 — The causal target contrasts mean potential outcomes under two defined interventions.
    2. MSC-P010-C02 — Consistency links the observed outcome to the potential outcome under the assigned policy.
    3. MSC-P010-C03 — The causal estimand assumes one customer's assignment does not alter another customer's outcome.
    4. MSC-P010-C04 — Fitting a regression does not itself establish a causal design.
    5. MSC-P010-C05 — Advance specification covers both the research question and the analysis plan before outcomes are observed.
    6. MSC-P010-C06 — Value-of-information methods support research prioritization and study design.
    7. MSC-P010-C07 — Formal VoI requires probabilistic parameter uncertainty.
    8. MSC-P010-C08 — Formal VoI requires decision-specific losses or benefits.
    9. MSC-P010-C09 — ENBS is based on EVSI, not an arbitrary local score.
    1. Jackson et al., 2022
    2. Hernán & Robins, Causal Inference: What If
    3. Arnold et al., 2020
    4. Nosek et al., 2018

    Method connections

    Parent territoryMeasurement and causality: how can a marketing effect be established?

    Read next

    MSC-P-011How do you design an A/B test that actually estimates an effect?→MSC-P-012How many observations does an experiment need?→MSC-P-017How do you detect selection, contamination and attrition in an experiment?→