Research & Evidence

Search for a method

Search titles, questions, territories and MSC identifiers.

    ← All methods
    METHOD DOSSIERMSC-P-001Evidence foundationsVerified scientific dossier

    How do you turn a marketing claim into a testable question?

    A question becomes testable when it fixes population, unit, intervention, comparator, outcome, horizon, minimum useful effect, and a result capable of contradicting the hypothesis.

    Scientific editorial team: Marketing Science Center

    Direct answer

    Write a measurable protocol before choosing a method.

    A question becomes testable when it fixes population, unit, intervention, comparator, outcome, horizon, minimum useful effect, and a result capable of contradicting the hypothesis.

    Nosek et al. (2018), The preregistration revolutionICH E9(R1), Estimands and sensitivity analysis

    01

    Problem to solve

    A claim such as “email increases purchases” specifies no population, comparison, horizon, or outcome that could contradict it. It therefore cannot guide a verifiable protocol or decision.

    02

    Operational summary

    This dossier turns the claim into a contract containing a population, unit, exposure, comparator, outcome, horizon, minimum useful effect, analysis plan, and falsifier. Its output validates contract completeness, not a marketing effect.

    1. Decision-maker: verify the decision and useful threshold.
    2. Practitioner: fix population, unit, outcome, and horizon.
    3. Analyst: seal estimand, analysis, uncertainty, and falsifier.

    03

    Concrete marketing situation

    A team wants to decide whether an email campaign deserves a test. The population is limited to eligible subscribers, the unit is the subscriber, the comparator is no email, and the outcome is a purchase within seven days.

    04

    Scientific question and scope

    Among eligible subscribers, does assignment to email rather than no email increase purchase within seven days by at least 1.0 percentage point? The planned estimand is an intention-to-treat risk difference.

    05

    Why a simple formulation fails

    Without a unit, dependence among observations is unknown. Without a comparator, the effect is undefined. Without a horizon, the outcome can move after observation. Without a threshold and falsifier, almost any result can be presented as favorable.

    06

    Method intuition

    A testable question fixes in advance what will be observed, for whom, against what, when, and under which rule the result may support or contradict the hypothesis. It thereby separates idea generation from confirmatory testing.

    07

    Required data

    The register requires population and eligibility, assignment and analysis units, intervention, comparator, outcome definition and window, estimand, intercurrent-event strategy, missing-outcome rule, estimator, interval method, alpha, multiplicity, threshold, direction, three decision outcomes, protocol deviations, and data status.

    08

    Formal model

    Q=(P,U,X,C,Y,H,Δ*,D,A,F), where P is population, U unit, X exposure, C comparator, Y outcome, H horizon, Δ* minimum useful effect, D direction, A analysis plan, and F falsifier.

    09

    Declared calculation

    Python and R read the supplied path, require one row, validate fields, H>0, Δ*>0, 95% ↔ alpha 0.05 consistency, the direction-bound pair, ITT denominator, intercurrent events, fail-closed missing-outcome rule, multiplicity, and deviations, then compute the same canonical SHA-256.

    10

    End-to-end numerical example

    The protocol fixes H=7 days, Δ*=1.0 point, and one contrast. The Miettinen-Nurminen CI inverts the constrained binomial score, applies N/(N-1), tolerance 1e-8, 100 iterations, 95% level, and two-sided alpha 0.05. Lower bound ≥ +1.0: threshold supported; upper bound < +1.0: falsified; otherwise inconclusive. protocol_complete=true; SHA-256=86a315abd306359f92ec5439f361511d80a7516af3516525c8bd1272014caa74; no effect estimated.

    Synthetic protocol example—no observations and no estimated effect.

    MSC-P-001-EXAMPLE
    WindowThresholdEstimandOutput
    71.0 ppplanned ITT risk differenceprotocol_complete=true

    11

    Validity assumptions

    Fields must denote observable, nonredundant objects. The unit must match the assignment mechanism. The outcome must be measurable for all groups over the same horizon. The useful threshold must come from the decision, not future results.

    12

    Diagnostics and uncertainty

    Check population, unique assignments, complete transaction-log coverage, delivery, opening, crossover, outside exposure, timing, exclusions, multiplicity, and deviations. Anticipated precision can be planned from sample size and assumed rates; the empirical interval can only be calculated after observation.

    13

    Result interpretation

    protocol_complete=true means only that the register contains the required fields and passes deterministic rules. It proves neither design feasibility, valid assignment, nor that email will produce the minimum useful effect.

    14

    Supported and forbidden conclusions

    Supported: the question, planned estimand, threshold, and falsifier are explicit. Forbidden: email increases purchases, the protocol guarantees an outcome, or merely stating a threshold makes the future test causal.

    15

    Possible marketing decision

    The team may accept, revise, or reject the protocol before collection, estimate test cost, verify variable availability, and decide whether the 1.0-point threshold justifies the experiment. It cannot yet act on a nonexistent observed effect.

    16

    When to use and when to stop

    Use before collection or before outcomes are opened. Stop if population, comparator, outcome, or horizon cannot be fixed, if the threshold has no decision rationale, or if the plan is rewritten after results are seen without an exploratory label.

    17

    Reproducible implementations

    The CC0 CSV is the sole register. Python 3.13 and R 4.5 are the reference validators: same supplied path, rules, question, and SHA-256. SPSS and SAS are explicitly limited to import and inspection; they do not validate protocol_complete. No seed is needed because nothing is simulated.

    18

    Expected final deliverable

    Deliver the canonical question, versioned and hashed register, rationale for Δ*, analysis plan, missing-data rule, falsifier, protocol deviations, and confirmatory or exploratory label for every analysis.

    19

    References and evidence level

    Nosek et al. support prespecification and falsifiability. ICH E9(R1) supports estimands, intercurrent events, and replication. SAS 9.4 documentation seals the corrected MN algorithm, numerical stopping rule, and distinction from Mee. All three full texts are verified. None validates the email example or synthetic threshold.

    1. Nosek et al. (2018) ↗Full text verified · author-deposited manuscript
    2. ICH E9(R1) (2020) ↗Full text verified · official Step 5 guideline
    3. SAS 9.4 · PROC FREQ ↗Full text verified · official algorithm documentation

    Dataset · Tool

    Dataset · MSC-P-001-QUESTIONSynthetic testable-question register→

    Method connections

    Parent territoryStatistical decision methods: choose, quantify, validate

    Read next

    MSC-P-002Correlation or causality: what can an analysis actually support?→MSC-P-010When should you run a marketing experiment?→MSC-P-018Predictive or causal regression: what are you trying to estimate?→