Research & Evidence

Search for a method

Search titles, questions, territories and MSC identifiers.

    ← All methods
    METHOD DOSSIERMSC-P-017Experimentation and causalityVerified scientific dossier

    How do you detect selection, contamination and attrition in an experiment?

    The diagnostic follows units from eligibility to outcome: assignment, exposure, contamination, missingness and exclusions. ITT remains based on assignment.

    Scientific editorial team: Marketing Science Center

    Direct answer

    Characterize threats and run sensitivity analyses.

    The diagnostic follows units from eligibility to outcome: assignment, exposure, contamination, missingness and exclusions. ITT remains based on assignment.

    CONSORT 2010 Explanation and ElaborationHernán & Robins, Causal Inference: What If

    01

    Operational summary

    Scientific question and scope

    How do you detect selection, contamination and attrition in an experiment?

    Supported

    Characterize threats and run sensitivity analyses.

    Forbidden

    Automatically repair severe differential attrition.

    02

    Three reading levels

    1. 01

      Decision-maker — Connect the result to a declared decision, useful threshold and error cost.

    2. 02

      Practitioner — Fix population, unit, horizon, available variables and analysis rule before calculation.

    3. 03

      Analyst — Reproduce the calculation, quantify uncertainty and document diagnostics, failures and sensitivities.

    03

    Concrete marketing situation

    The diagnostic follows units from eligibility to outcome: assignment, exposure, contamination, missingness and exclusions. ITT remains based on assignment.

    04

    Scientific question and scope

    How do you detect selection, contamination and attrition in an experiment?

    Diagnostic evidence, not a new causal effect

    Required data

    eligible unit through analysis flow

    Population, unit of analysis, origin date and horizon must be declared in the deliverable. Without them, the estimand silently changes.

    05

    Why a simple analysis can fail

    • Automatically repair severe differential attrition.
    • Do assigned counts reconcile with the randomization log and prespecified allocation?
    • Is outcome missingness compared by assignment arm under a declared rule?
    • Are exposure and contamination defined without conditioning the primary ITT analysis on post-assignment behavior?

    06

    Method intuition

    The method does not automatically turn an association into evidence. It links a declared question to an estimand, specification, compatible data and a bounded interpretation rule.

    Diagnostic evidence, not a new causal effect

    07

    Required data

    Scientific symbol dictionary

    assigned_arm
    Number of eligible units randomized to the declared arm. Unit: unit · Type: nonnegative integer · Role: input
    observed_arm
    Number of assigned units with the primary outcome observed under the prespecified missing-outcome rule. Unit: unit · Type: nonnegative integer · Role: input
    exposed_arm
    Number of assigned units that received the measured exposure under the prespecified exposure definition. Unit: unit · Type: nonnegative integer · Role: input
    attrition_arm
    Share of assigned units without an observed primary outcome in that arm. Unit: proportion · Type: number in [0, 1] · Role: output
    exposure_T
    Exposure share among treatment-assigned units. Unit: proportion · Type: number in [0, 1] · Role: diagnostic
    contamination_C
    Exposure share among control-assigned units. Unit: proportion · Type: number in [0, 1] · Role: diagnostic

    Exact sealed engine inputs

    assigned_treatment
    Value: 5000 · Unit: unit · Type: integer · Data status: synthetic
    assigned_control
    Value: 5000 · Unit: unit · Type: integer · Data status: synthetic
    observed_treatment
    Value: 4800 · Unit: unit · Type: integer · Data status: synthetic
    observed_control
    Value: 4900 · Unit: unit · Type: integer · Data status: synthetic
    exposed_treatment
    Value: 4500 · Unit: unit · Type: integer · Data status: synthetic
    exposed_control
    Value: 200 · Unit: unit · Type: integer · Data status: synthetic

    08

    Formal model

    Formal model

    attrition_treatment = 1 − observed_treatment/assigned_treatment; attrition_control = 1 − observed_control/assigned_control; differential_attrition = attrition_treatment − attrition_control; treatment_exposure = exposed_treatment/assigned_treatment; control_contamination = exposed_control/assigned_control

    Evidence claims: MSC-P017-C01 · MSC-P017-C02 · MSC-P017-C03 · MSC-P017-C04 · MSC-P017-C05

    Scientific question and scope

    Diagnostic evidence, not a new causal effect

    09

    Declared calculation

    1. 01

      Reconcile eligible, assigned, observed and exposed counts by randomized arm and reject impossible count relationships.

    2. 02

      Compute arm-specific attrition, differential attrition, treatment exposure and control contamination rates.

    3. 03

      Report implementation diagnostics only; do not claim that these rates repair bias or constitute an SRM test.

    10

    Numerical example or application case

    Application example — synthetic data or declared parameters; no real observations

    Sealed inputs

    • assigned_treatment=5000 [unit]
    • assigned_control=5000 [unit]
    • observed_treatment=4800 [unit]
    • observed_control=4900 [unit]
    • exposed_treatment=4500 [unit]
    • exposed_control=200 [unit]

    Reproducible results

    • attrition_treatment/control=0.04/0.02
    • differential_attrition=0.02
    • exposure/contamination=0.90/0.04

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    11

    Validity assumptions

    • Do assigned counts reconcile with the randomization log and prespecified allocation?
    • Is outcome missingness compared by assignment arm under a declared rule?
    • Are exposure and contamination defined without conditioning the primary ITT analysis on post-assignment behavior?

    12

    Diagnostics and uncertainty

    Diagnostics and uncertainty

    Do assigned counts reconcile with the randomization log and prespecified allocation? · Is outcome missingness compared by assignment arm under a declared rule? · Are exposure and contamination defined without conditioning the primary ITT analysis on post-assignment behavior?

    Bounded diagnostic

    These diagnostics reveal implementation risks; they do not repair bias.

    Explicit uncertainty contract · not_estimable_from_sealed_inputs

    Method: Explicit non-estimability assessment against the sealed input schema.

    Target: Sampling uncertainty of attrition, exposure and contamination rates.

    Engine evidence: diagnostic=diagnostics_do_not_repair_bias

    Interpretation: Aggregate flow counts alone do not identify the dependence or missingness process needed for valid uncertainty intervals.

    Stop when

    Automatically repair severe differential attrition.

    13

    Result interpretation

    • Characterize threats and run sensitivity analyses.
    • The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.

    14

    Supported and forbidden conclusions

    Supported

    Characterize threats and run sensitivity analyses.

    Forbidden

    Automatically repair severe differential attrition.

    15

    Possible marketing decision

    1. 01

      Characterize threats and run sensitivity analyses.

    2. 02

      The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.

    16

    When to use — when to stop

    Use when

    Characterize threats and run sensitivity analyses.

    eligible unit through analysis flow

    Stop when

    Automatically repair severe differential attrition.

    Methodological alternatives

    • Run an exact sample-ratio-mismatch test when allocation probabilities and assignment logs are available.
    • Use sensitivity bounds for missing outcomes or noncompliance.
    • Report both ITT and prespecified treatment-received analyses without replacing the ITT estimand.

    17

    Reproducibility contract

    Python and R are the executable references. SPSS and SAS remain secondary syntaxes until checked on the same data, specification and diagnostics.

    The engine selects only this page’s explicit slice and branch. Its hash, outputs, and diagnostics remain sealed in the atomic dossier; no generic fallback branch is allowed.

    Slice fingerprint: 86981a5527088e6e95da14a6c0091530f1f492c73e0b0dec47bdbd5cbda693e0

    Formal model

    attrition_treatment = 1 − observed_treatment/assigned_treatment; attrition_control = 1 − observed_control/assigned_control; differential_attrition = attrition_treatment − attrition_control; treatment_exposure = exposed_treatment/assigned_treatment; control_contamination = exposed_control/assigned_control

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    18

    Expected final deliverable

    1. 01

      How do you detect selection, contamination and attrition in an experiment? — eligible unit through analysis flow

    2. 02

      assigned_arm · observed_arm · exposed_arm · attrition_arm · exposure_T · contamination_C

    3. 03

      attrition_treatment = 1 − observed_treatment/assigned_treatment; attrition_control = 1 − observed_control/assigned_control; differential_attrition = attrition_treatment − attrition_control; treatment_exposure = exposed_treatment/assigned_treatment; control_contamination = exposed_control/assigned_control

    4. 04

      Do assigned counts reconcile with the randomization log and prespecified allocation? · Is outcome missingness compared by assignment arm under a declared rule? · Are exposure and contamination defined without conditioning the primary ITT analysis on post-assignment behavior?

    5. 05

      Characterize threats and run sensitivity analyses. / Automatically repair severe differential attrition.

    19

    Scientific sources and evidence status

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    1. MSC-P017-C01 — Sample-size determination must name the primary outcome and all quantities used.
    2. MSC-P017-C02 — Attrition and non-compliance adjustments must be reported.
    3. MSC-P017-C03 — Analysis by randomized assignment preserves the intended comparison.
    4. MSC-P017-C04 — Consistency links the observed outcome to the potential outcome under the assigned policy.
    5. MSC-P017-C05 — The causal estimand assumes one customer's assignment does not alter another customer's outcome.
    1. CONSORT 2010 Explanation and Elaboration
    2. Hernán & Robins, Causal Inference: What If

    Method connections

    Parent territoryMeasurement and causality: how can a marketing effect be established?

    Read next

    MSC-P-010When should you run a marketing experiment?→MSC-P-011How do you design an A/B test that actually estimates an effect?→MSC-P-013How do you measure campaign incrementality with a control group?→