Research & Evidence

Search for a method

Search titles, questions, territories and MSC identifiers.

    ← All methods
    METHOD DOSSIERMSC-P-004Evidence foundationsVerified scientific dossier

    Statistical significance or effect size: which result should be interpreted?

    A p-value measures compatibility with a null model. Effect size describes observed magnitude and should be paired with an interval and a practical relevance threshold.

    Scientific editorial team: Marketing Science Center

    Direct answer

    Assess detection, magnitude and precision together.

    A p-value measures compatibility with a null model. Effect size describes observed magnitude and should be paired with an interval and a practical relevance threshold.

    American Statistical Association, 2016Lakens, 2013

    01

    The decision rule

    Always interpret four objects together: the effect estimate, its interval, the p-value from the declared test, and a practical-importance threshold fixed before seeing the result. A small p-value measures neither magnitude nor usefulness; a large p-value does not demonstrate absence.

    American Statistical Association (2016)

    02

    Three reading levels

    1. 1

      Decision-maker: ask whether the interval clears the business threshold, not only whether p is below 0.05.

    2. 2

      Practitioner: report absolute difference, relative lift, interval, p-value, sample sizes and multiplicity rule.

    3. 3

      Analyst: define the estimand, sampling process, δ, interval level and sensitivity analysis.

    03

    Concrete marketing situation

    A team compares two conversion rates and must decide whether the difference justifies rollout. It has a p-value, but change costs require at least +1 conversion point. The real question is: which magnitudes remain compatible with the data, and do they clear that threshold?

    04

    Exact scientific question

    For the declared populations, units and windows, what is the descriptive difference Δ = πB − πA, with what precision, and is its interval compatible with zero, a useful effect δ, or an equivalence zone [−δ, +δ]?

    05

    Why p < 0.05 is insufficient

    • With a very large n, a tiny difference can yield a small p-value without economic value.
    • A nonsignificant result may simply be too imprecise to distinguish absence, harm or useful benefit.
    • Choosing δ after seeing the data turns the business rule into an after-the-fact justification.

    American Statistical Association (2016) Lakens, Scheel & Isager (2018)

    06

    Intuition before equations

    • The p-value answers a compatibility question under a declared null model; it does not give the probability that H0 is true.
    • Effect size gives magnitude on a chosen scale; the interval shows values still compatible with the data and model.
    • Threshold δ translates cost, margin or risk into a minimum useful difference; it is not a universal constant.

    Lakens (2013)

    07

    Required data

    • Two independent binomial groups, sample sizes nA/nB and conversions yA/yB; no causal assignment is claimed.
    • Same conversion definition, same observation window and explicitly comparable populations.
    • Practical threshold δ = 1 point fixed before analysis; alpha = 5% and test family declared.

    08

    Formal model and symbols

    Effect

    πA,πB; Δ=πB−πAp̂A=yA/nA; p̂B=yB/nB; Δ̂=p̂B−p̂A; lift=Δ̂/p̂A

    nA,nB: units; yA,yB: conversions; π: population parameter; p̂: observed rate; Δ: estimand; Δ̂: estimator; δ: minimum useful difference fixed before analysis.

    Uncertainty

    H0:Δ=0; p̂pool=(yA+yB)/(nA+nB); SE0=√[p̂pool(1−p̂pool)(1/nA+1/nB)]; z=Δ̂/SE0; p=2Φ(−|z|)SE=√[p̂A(1−p̂A)/nA+p̂B(1−p̂B)/nB]; CI95=Δ̂±z0.975SE; CI90=Δ̂±z0.95SETOST: H01:Δ≤−δ; H02:Δ≥+δ; max(p1,p2)<α

    α is the declared type-I error rate; Φ is the standard-normal CDF; zq is its q quantile. The Wald CI is used here only for large counts; prefer a score method when the approximation is fragile.

    Fagerland, Lydersen & Laake (2015) Lakens, Scheel & Isager (2018)

    09

    Declared calculation, step by step

    1. 01

      Fix population, window, metric, alpha, multiplicity and δ before results.

    2. 02

      Compute rates, Δ in points and relative lift without changing units mid-interpretation.

    3. 03

      Compute p under H0:Δ=0 and the 95% CI for Δ; for equivalence, also test [−δ,+δ] with two one-sided tests.

    4. 04

      Classify the result against zero and δ, then document the decision and what it does not prove.

    10

    Four results that do not mean the same thing

    Synthetic illustration — teaching values, not observed

    Four results that do not mean the same thing
    ScenarioCounts A → BRates A → BΔ̂Relative liftp95% CI90% CITOST pDecision
    detectable_negligible500000/50000 → 500000/5100010.000% → 10.200%0.200 pp2.000%0.0009046[0.0819; 0.3181][0.1009; 0.2991]1.62e-40detectable-but-negligible
    inconclusive1000/100 → 1000/12010.000% → 12.000%2.000 pp20.000%0.152918[-0.7411; 4.7411][-0.3004; 4.3004]0.762700inconclusive
    practically_convincing10000/1000 → 10000/125010.000% → 12.500%2.500 pp25.000%2.21e-8[1.6248; 3.3752][1.7655; 3.2345]0.999609practically-convincing-benefit
    practically_equivalent50000/5000 → 50000/502510.000% → 10.050%0.050 pp0.500%0.792373[-0.3223; 0.4223][-0.2624; 0.3624]2.85e-7practically-equivalent

    The first scenario rejects both Δ=0 and effects of at least 1 pp: it is statistically detectable and contained in the practical-negligibility zone. This joint result never follows from the point estimate alone.

    CSV Python · MIT R · MIT SPSS · MIT SAS · MIT

    11

    Validity assumptions

    • Are units independent within and between groups? If not, does the variance account for the actual dependence?
    • Does each unit have a binary outcome, a stable probability within its group, and one observation in the window?
    • Are counts large enough for the normal approximation? If not, is an appropriate binomial or score method used?
    • Does δ come from a documented value function fixed before analysis rather than from the observed result?

    12

    Diagnostics and uncertainty

    Check all four binomial counts, denominators and absence of duplicate units.

    Report the 95% CI for Δ and 90% CI for TOST; excessive width means insufficient information.

    Repeat the reading for defensible δ values and the preregistered multiplicity correction.

    13

    Interpreting the result

    • Detectable but negligible: zero is excluded and the 90% CI lies entirely within [−δ,+δ]; the point estimate alone is insufficient.
    • Inconclusive: the interval contains both zero and useful effects; collect more data or decide under explicit risk.
    • Practically convincing: the 95% CI clears δ in the useful direction under the declared assumptions.
    • Practically equivalent: the 90% CI lies within [−δ,+δ] and both one-sided tests pass.

    14

    Supported and forbidden conclusions

    Supported

    • Describe magnitude and precision for observed populations.
    • State which useful or negligible values remain compatible with the data.

    Forbidden

    • Turn significance into importance, probability of H0 or causality.
    • Interpret p ≥ 0.05 as evidence of absence without a justified equivalence test.

    15

    Possible marketing decision

    1. 01

      Roll out only if the plausible range clears the value threshold and operational risks are acceptable.

    2. 02

      Stop for futility only when practical equivalence is supported around a defensible δ.

    3. 03

      Otherwise treat the result as inconclusive and choose among more data, a bounded pilot or a risk-based decision.

    16

    When to use, and when not to use

    Use

    • Comparisons of two independent proportions already justified by the analysis plan.
    • Decision reading after checking design, data and multiplicity.

    Do not use

    • To choose the test: see MSC-P-039.
    • For power and n planning: see MSC-P-012; for causality: see MSC-P-013.

    17

    Reproducible implementations

    Python 3.13 · reference

    python msc-p004-reference.py --csv msc-p004-significance-effect-size.csv

    R 4.5 · reference

    Rscript msc-p004-reference.R msc-p004-significance-effect-size.csv

    SPSS 31 · secondary

    INSERT FILE='msc-p004-reference.sps'.

    SAS 9.4 · secondary

    %include 'msc-p004-reference.sas';

    Python and R are the verified executable references. The secondary SPSS and SAS syntaxes apply the same counts, formulas and decision rules; run them in their native environments before operational use.

    18

    Expected final deliverable

    1. 01

      Question, population, unit, window and design.

    2. 02

      Absolute Δ, relative lift, p-value, 95% CI and preregistered δ.

    3. 03

      Decision class, assumptions, multiplicity, limits and impossible decision.

    19

    Scientific sources and evidence level

    1. The p-value measures neither size nor importance and should not decide alone.

    2. Lakens (2013)Methodological article, CC BY

      Effect sizes communicate magnitude and support comparison, synthesis and planning.

    3. Lakens, Scheel & Isager (2018)Methodological tutorial, open access

      Equivalence requires bounds based on the smallest effect of interest and two one-sided tests.

    4. The Wald CI for two proportions requires large counts and remains less robust than score intervals.

    Method connections

    Parent territoryStatistical decision methods: choose, quantify, validate

    Read next

    MSC-P-003How should uncertainty in a marketing result be expressed?→MSC-P-012How many observations does an experiment need?→MSC-P-013How do you measure campaign incrementality with a control group?→MSC-P-039Which statistical test should you choose?→