Research & Evidence

Search for a method

Search titles, questions, territories and MSC identifiers.

    ← All methods
    METHOD DOSSIERMSC-P-029Customer scienceVerified scientific dossier

    Which customers have the highest probability of churn?

    Churn is predicted for a fixed decision date and horizon. Temporal separation, leakage prevention and calibration matter as much as discrimination.

    Scientific editorial team: Marketing Science Center

    Direct answer

    Prioritize human review using calibrated probabilistic risk.

    Churn is predicted for a fixed decision date and horizon. Temporal separation, leakage prevention and calibration matter as much as discrimination.

    scikit-learn average_precision_scoreDimitriadis et al., 2023

    01

    Operational summary

    Scientific question and scope

    Which customers have the highest probability of churn?

    Supported

    Prioritize human review using calibrated probabilistic risk.

    Forbidden

    Claim which action will reduce churn from the predictive model alone.

    02

    Three reading levels

    1. 01

      Decision-maker — Connect the result to a declared decision, useful threshold and error cost.

    2. 02

      Practitioner — Fix population, unit, horizon, available variables and analysis rule before calculation.

    3. 03

      Analyst — Reproduce the calculation, quantify uncertainty and document diagnostics, failures and sensitivities.

    03

    Concrete marketing situation

    Churn is predicted for a fixed decision date and horizon. Temporal separation, leakage prevention and calibration matter as much as discrimination.

    04

    Scientific question and scope

    Which customers have the highest probability of churn?

    P(churn within H | information available at t)

    Required data

    customer at decision date

    Population, unit of analysis, origin date and horizon must be declared in the deliverable. Without them, the estimand silently changes.

    05

    Why a simple analysis can fail

    • Claim which action will reduce churn from the predictive model alone.
    • Was every feature available at the declared prediction time and was the holdout later in time?
    • Are tied prediction scores grouped consistently when average precision is computed?
    • Are calibration, discrimination and decision utility assessed separately on a sufficiently large deployment-representative sample?

    06

    Method intuition

    The method does not automatically turn an association into evidence. It links a declared question to an estimand, specification, compatible data and a bounded interpretation rule.

    P(churn within H | information available at t)

    07

    Required data

    Scientific symbol dictionary

    p_i
    Predicted churn probability for holdout customer i using only information available at the decision date. Unit: probability · Type: number strictly between 0 and 1 · Role: input
    y_i
    Observed binary churn outcome by the fixed horizon for holdout customer i. Unit: binary · Type: 0 or 1 · Role: input
    Brier
    Mean squared probability error on the temporal holdout. Unit: squared probability · Type: nonnegative number · Role: output
    log_loss
    Mean negative Bernoulli log likelihood on the temporal holdout. Unit: nat per observation · Type: nonnegative number · Role: output
    calibration_gap
    Mean predicted probability minus observed event rate; negative means average underprediction under this sign convention. Unit: probability · Type: number in [-1, 1] · Role: output
    AP_threshold
    Non-interpolated average precision computed over unique prediction thresholds with tied scores grouped; not generic PR-AUC. Unit: proportion · Type: number in [0, 1] · Role: output

    Exact sealed engine inputs

    predicted_risk_1
    Value: 0.05 · Unit: probability · Type: number · Data status: synthetic
    predicted_risk_2
    Value: 0.10 · Unit: probability · Type: number · Data status: synthetic
    predicted_risk_3
    Value: 0.20 · Unit: probability · Type: number · Data status: synthetic
    predicted_risk_4
    Value: 0.30 · Unit: probability · Type: number · Data status: synthetic
    predicted_risk_5
    Value: 0.45 · Unit: probability · Type: number · Data status: synthetic
    predicted_risk_6
    Value: 0.60 · Unit: probability · Type: number · Data status: synthetic
    predicted_risk_7
    Value: 0.75 · Unit: probability · Type: number · Data status: synthetic
    predicted_risk_8
    Value: 0.90 · Unit: probability · Type: number · Data status: synthetic
    outcome_1
    Value: 0 · Unit: binary · Type: integer · Data status: synthetic
    outcome_2
    Value: 0 · Unit: binary · Type: integer · Data status: synthetic
    outcome_3
    Value: 0 · Unit: binary · Type: integer · Data status: synthetic
    outcome_4
    Value: 1 · Unit: binary · Type: integer · Data status: synthetic
    outcome_5
    Value: 0 · Unit: binary · Type: integer · Data status: synthetic
    outcome_6
    Value: 1 · Unit: binary · Type: integer · Data status: synthetic
    outcome_7
    Value: 1 · Unit: binary · Type: integer · Data status: synthetic
    outcome_8
    Value: 1 · Unit: binary · Type: integer · Data status: synthetic
    temporal_holdout_declared
    Value: 1 · Unit: binary · Type: integer · Data status: parameter

    08

    Formal model

    Formal model

    temporal_holdout_declared = 1; Brier = mean((p_i−y_i)^2); log_loss = −mean(y_i×log(p_i)+(1−y_i)×log(1−p_i)); calibration_gap = mean(p_i)−mean(y_i); AP_threshold = grouped_average_precision(p_i, y_i)

    Evidence claims: MSC-P029-C01 · MSC-P029-C02 · MSC-P029-C03 · MSC-P029-C04 · MSC-P029-C05 · MSC-P029-C06 · MSC-P029-C07 · MSC-P029-C08 · MSC-P029-C09 · MSC-P029-C10

    Scientific question and scope

    P(churn within H | information available at t)

    09

    Declared calculation

    1. 01

      Freeze the temporal holdout and validate paired probabilities/outcomes with both outcome classes represented.

    2. 02

      Compute Brier score, log loss, threshold-grouped average precision and the predicted-minus-observed calibration gap.

    3. 03

      Report discrimination and calibration separately and label the tiny synthetic holdout as insufficient for deployment or causal validation.

    10

    Numerical example or application case

    Synthetic illustration — teaching values, not observed — synthetic data or declared parameters; no real observations

    Sealed inputs

    • predicted_risk_1=0.05 [probability]
    • predicted_risk_2=0.10 [probability]
    • predicted_risk_3=0.20 [probability]
    • predicted_risk_4=0.30 [probability]
    • predicted_risk_5=0.45 [probability]
    • predicted_risk_6=0.60 [probability]
    • predicted_risk_7=0.75 [probability]
    • predicted_risk_8=0.90 [probability]
    • outcome_1=0 [binary]
    • outcome_2=0 [binary]
    • outcome_3=0 [binary]
    • outcome_4=1 [binary]
    • outcome_5=0 [binary]
    • outcome_6=1 [binary]
    • outcome_7=1 [binary]
    • outcome_8=1 [binary]
    • temporal_holdout_declared=1 [binary]

    Reproducible results

    • Brier=0.122188
    • log_loss=0.385684
    • threshold_grouped_average_precision=0.95
    • mean_predicted_minus_observed=-0.08125

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    11

    Validity assumptions

    • Was every feature available at the declared prediction time and was the holdout later in time?
    • Are tied prediction scores grouped consistently when average precision is computed?
    • Are calibration, discrimination and decision utility assessed separately on a sufficiently large deployment-representative sample?

    12

    Diagnostics and uncertainty

    Diagnostics and uncertainty

    Was every feature available at the declared prediction time and was the holdout later in time? · Are tied prediction scores grouped consistently when average precision is computed? · Are calibration, discrimination and decision utility assessed separately on a sufficiently large deployment-representative sample?

    Bounded diagnostic

    The sign convention is predicted minus observed. Average precision is threshold-grouped and is not generic PR-AUC. Tiny synthetic illustration only.

    Explicit uncertainty contract · not_estimable_from_sealed_inputs

    Method: Explicit non-estimability assessment against the sealed input schema.

    Target: Sampling uncertainty of holdout calibration and discrimination metrics.

    Engine evidence: diagnostic=average_precision_is_threshold_grouped_not_generic_pr_auc_tiny_synthetic_holdout_not_deployment_validation_or_causal_effect

    Interpretation: Eight synthetic holdout observations are insufficient for defensible confidence intervals or subgroup uncertainty.

    Stop when

    Claim which action will reduce churn from the predictive model alone.

    13

    Result interpretation

    • Prioritize human review using calibrated probabilistic risk.
    • The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.

    14

    Supported and forbidden conclusions

    Supported

    Prioritize human review using calibrated probabilistic risk.

    Forbidden

    Claim which action will reduce churn from the predictive model alone.

    15

    Possible marketing decision

    1. 01

      Prioritize human review using calibrated probabilistic risk.

    2. 02

      The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.

    16

    When to use — when to stop

    Use when

    Prioritize human review using calibrated probabilistic risk.

    customer at decision date

    Stop when

    Claim which action will reduce churn from the predictive model alone.

    Methodological alternatives

    • Use calibration curves with uncertainty by risk range and subgroup.
    • Use ROC analysis only for discrimination questions, not calibration.
    • Use decision-curve or cost-sensitive threshold analysis when action utility is the target.

    17

    Reproducibility contract

    Python and R are the executable references. SPSS and SAS remain secondary syntaxes until checked on the same data, specification and diagnostics.

    The engine selects only this page’s explicit slice and branch. Its hash, outputs, and diagnostics remain sealed in the atomic dossier; no generic fallback branch is allowed.

    Slice fingerprint: 05348452a7d6e9c0921fab5459274f7d09e9ba76b92659dd05112935133f85f7

    Formal model

    temporal_holdout_declared = 1; Brier = mean((p_i−y_i)^2); log_loss = −mean(y_i×log(p_i)+(1−y_i)×log(1−p_i)); calibration_gap = mean(p_i)−mean(y_i); AP_threshold = grouped_average_precision(p_i, y_i)

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    18

    Expected final deliverable

    1. 01

      Which customers have the highest probability of churn? — customer at decision date

    2. 02

      p_i · y_i · Brier · log_loss · calibration_gap · AP_threshold

    3. 03

      temporal_holdout_declared = 1; Brier = mean((p_i−y_i)^2); log_loss = −mean(y_i×log(p_i)+(1−y_i)×log(1−p_i)); calibration_gap = mean(p_i)−mean(y_i); AP_threshold = grouped_average_precision(p_i, y_i)

    4. 04

      Was every feature available at the declared prediction time and was the holdout later in time? · Are tied prediction scores grouped consistently when average precision is computed? · Are calibration, discrimination and decision utility assessed separately on a sufficiently large deployment-representative sample?

    5. 05

      Prioritize human review using calibrated probabilistic risk. / Claim which action will reduce churn from the predictive model alone.

    19

    Scientific sources and evidence status

    Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.

    1. MSC-P029-C01 — Probabilistic scores should be evaluated on a declared test set.
    2. MSC-P029-C02 — Calibration concerns observed event frequency conditional on forecast probability.
    3. MSC-P029-C03 — ROC-based discrimination does not determine probability calibration.
    4. MSC-P029-C04 — Calibration and discrimination require distinct diagnostics.
    5. MSC-P029-C05 — Brier and logarithmic scores are proper probability-scoring rules.
    6. MSC-P029-C06 — Predictive and causal objectives alter the full modelling process, not just the final metric.
    7. MSC-P029-C07 — Predictive accuracy should be evaluated on observations excluded from model fitting.
    8. MSC-P029-C08 — The reported AP uses threshold-specific precision weighted by recall increments.
    9. MSC-P029-C09 — The executable convention is non-interpolated.
    10. MSC-P029-C10 — Average precision is not labeled as generic trapezoidal PR-AUC.
    1. scikit-learn average_precision_score
    2. Dimitriadis et al., 2023
    3. Shmueli, 2010

    Method connections

    Parent territoryCustomer and choice science: behavior, value and heterogeneity

    Read next

    MSC-P-026Logit vs Probit: how do you choose for purchase probability?→MSC-P-018Predictive or causal regression: what are you trying to estimate?→MSC-P-019How do you diagnose a marketing regression before interpreting it?→