How do you turn a marketing claim into a testable question?
A question becomes testable when it fixes population, unit, intervention, comparator, outcome, horizon, minimum useful effect, and a result capable of contradicting the hypothesis.
Scientific editorial team : Marketing Science Center
Direct answer
Write a measurable protocol before choosing a method.
A question becomes testable when it fixes population, unit, intervention, comparator, outcome, horizon, minimum useful effect, and a result capable of contradicting the hypothesis.
Nosek et al. (2018), The preregistration revolutionICH E9(R1), Estimands and sensitivity analysis
01
Problem to solve
A claim such as “email increases purchases” specifies no population, comparison, horizon, or outcome that could contradict it. It therefore cannot guide a verifiable protocol or decision.
02
Operational summary
This dossier turns the claim into a contract containing a population, unit, exposure, comparator, outcome, horizon, minimum useful effect, analysis plan, and falsifier. Its output validates contract completeness, not a marketing effect.
- Decision-maker: verify the decision and useful threshold.
- Practitioner: fix population, unit, outcome, and horizon.
- Analyst: seal estimand, analysis, uncertainty, and falsifier.
03
Concrete marketing situation
A team wants to decide whether an email campaign deserves a test. The population is limited to eligible subscribers, the unit is the subscriber, the comparator is no email, and the outcome is a purchase within seven days.
04
Scientific question and scope
Among eligible subscribers, does assignment to email rather than no email increase purchase within seven days by at least 1.0 percentage point? The planned estimand is an intention-to-treat risk difference.
05
Why a simple formulation fails
Without a unit, dependence among observations is unknown. Without a comparator, the effect is undefined. Without a horizon, the outcome can move after observation. Without a threshold and falsifier, almost any result can be presented as favorable.
06
Method intuition
A testable question fixes in advance what will be observed, for whom, against what, when, and under which rule the result may support or contradict the hypothesis. It thereby separates idea generation from confirmatory testing.
07
Required data
The register requires population and eligibility, assignment and analysis units, intervention, comparator, outcome definition and window, estimand, intercurrent-event strategy, missing-outcome rule, estimator, interval method, alpha, multiplicity, threshold, direction, three decision outcomes, protocol deviations, and data status.
08
Formal model
Q=(P,U,X,C,Y,H,Δ*,D,A,F), where P is population, U unit, X exposure, C comparator, Y outcome, H horizon, Δ* minimum useful effect, D direction, A analysis plan, and F falsifier.
09
Declared calculation
Python and R read the supplied path, require one row, validate fields, H>0, Δ*>0, 95% ↔ alpha 0.05 consistency, the direction-bound pair, ITT denominator, intercurrent events, fail-closed missing-outcome rule, multiplicity, and deviations, then compute the same canonical SHA-256.
10
End-to-end numerical example
The protocol fixes H=7 days, Δ*=1.0 point, and one contrast. The Miettinen-Nurminen CI inverts the constrained binomial score, applies N/(N-1), tolerance 1e-8, 100 iterations, 95% level, and two-sided alpha 0.05. Lower bound ≥ +1.0: threshold supported; upper bound < +1.0: falsified; otherwise inconclusive. protocol_complete=true; SHA-256=86a315abd306359f92ec5439f361511d80a7516af3516525c8bd1272014caa74; no effect estimated.
Synthetic protocol example—no observations and no estimated effect.
| Window | Threshold | Estimand | Output |
|---|---|---|---|
| 7 | 1.0 pp | planned ITT risk difference | protocol_complete=true |
11
Validity assumptions
Fields must denote observable, nonredundant objects. The unit must match the assignment mechanism. The outcome must be measurable for all groups over the same horizon. The useful threshold must come from the decision, not future results.
12
Diagnostics and uncertainty
Check population, unique assignments, complete transaction-log coverage, delivery, opening, crossover, outside exposure, timing, exclusions, multiplicity, and deviations. Anticipated precision can be planned from sample size and assumed rates; the empirical interval can only be calculated after observation.
13
Result interpretation
protocol_complete=true means only that the register contains the required fields and passes deterministic rules. It proves neither design feasibility, valid assignment, nor that email will produce the minimum useful effect.
14
Supported and forbidden conclusions
Supported: the question, planned estimand, threshold, and falsifier are explicit. Forbidden: email increases purchases, the protocol guarantees an outcome, or merely stating a threshold makes the future test causal.
15
Possible marketing decision
The team may accept, revise, or reject the protocol before collection, estimate test cost, verify variable availability, and decide whether the 1.0-point threshold justifies the experiment. It cannot yet act on a nonexistent observed effect.
16
When to use and when to stop
Use before collection or before outcomes are opened. Stop if population, comparator, outcome, or horizon cannot be fixed, if the threshold has no decision rationale, or if the plan is rewritten after results are seen without an exploratory label.
17
Reproducible implementations
The CC0 CSV is the sole register. Python 3.13 and R 4.5 are the reference validators: same supplied path, rules, question, and SHA-256. SPSS and SAS are explicitly limited to import and inspection; they do not validate protocol_complete. No seed is needed because nothing is simulated.
CSV · CC0
msc-p001-testable-question.csv ↓Python · MIT
msc-p001-reference.py ↓R · MIT
msc-p001-reference.R ↓SPSS / SAS · MIT · inspection only
SPSS ↓SAS ↓18
Expected final deliverable
Deliver the canonical question, versioned and hashed register, rationale for Δ*, analysis plan, missing-data rule, falsifier, protocol deviations, and confirmatory or exploratory label for every analysis.
19
References and evidence level
Nosek et al. support prespecification and falsifiability. ICH E9(R1) supports estimands, intercurrent events, and replication. SAS 9.4 documentation seals the corrected MN algorithm, numerical stopping rule, and distinction from Mee. All three full texts are verified. None validates the email example or synthetic threshold.
- Nosek et al. (2018) ↗Full text verified · author-deposited manuscript
- ICH E9(R1) (2020) ↗Full text verified · official Step 5 guideline
- SAS 9.4 · PROC FREQ ↗Full text verified · official algorithm documentation
Dataset · Tool
Method connections

