Which customers have the highest probability of churn?
Churn is predicted for a fixed decision date and horizon. Temporal separation, leakage prevention and calibration matter as much as discrimination.
Scientific editorial team : Marketing Science Center
Direct answer
Prioritize human review using calibrated probabilistic risk.
Churn is predicted for a fixed decision date and horizon. Temporal separation, leakage prevention and calibration matter as much as discrimination.
scikit-learn average_precision_scoreDimitriadis et al., 2023
01
Operational summary
Scientific question and scope
Which customers have the highest probability of churn?
Supported
Prioritize human review using calibrated probabilistic risk.
Forbidden
Claim which action will reduce churn from the predictive model alone.
02
Three reading levels
- 01
Decision-maker — Connect the result to a declared decision, useful threshold and error cost.
- 02
Practitioner — Fix population, unit, horizon, available variables and analysis rule before calculation.
- 03
Analyst — Reproduce the calculation, quantify uncertainty and document diagnostics, failures and sensitivities.
03
Concrete marketing situation
Churn is predicted for a fixed decision date and horizon. Temporal separation, leakage prevention and calibration matter as much as discrimination.
04
Scientific question and scope
Which customers have the highest probability of churn?
P(churn within H | information available at t)
Required data
customer at decision date
Population, unit of analysis, origin date and horizon must be declared in the deliverable. Without them, the estimand silently changes.
05
Why a simple analysis can fail
- Claim which action will reduce churn from the predictive model alone.
- Temporal holdout
- Brier and log-loss
- Threshold-grouped non-interpolated average precision
06
Method intuition
The method does not automatically turn an association into evidence. It links a declared question to an estimand, specification, compatible data and a bounded interpretation rule.
P(churn within H | information available at t)
07
Required data
Scientific symbol dictionary
p_i- Predicted churn probability for holdout customer i using only information available at the decision date. Unit:
probability· Type:number strictly between 0 and 1· Role:input y_i- Observed binary churn outcome by the fixed horizon for holdout customer i. Unit:
binary· Type:0 or 1· Role:input Brier- Mean squared probability error on the temporal holdout. Unit:
squared probability· Type:nonnegative number· Role:output log_loss- Mean negative Bernoulli log likelihood on the temporal holdout. Unit:
nat per observation· Type:nonnegative number· Role:output calibration_gap- Mean predicted probability minus observed event rate; negative means average underprediction under this sign convention. Unit:
probability· Type:number in [-1, 1]· Role:output AP_threshold- Non-interpolated average precision computed over unique prediction thresholds with tied scores grouped; not generic PR-AUC. Unit:
proportion· Type:number in [0, 1]· Role:output
Exact sealed engine inputs
predicted_risk_1- Value:
0.05· Unit:probability· Type:number· Data status:synthetic predicted_risk_2- Value:
0.10· Unit:probability· Type:number· Data status:synthetic predicted_risk_3- Value:
0.20· Unit:probability· Type:number· Data status:synthetic predicted_risk_4- Value:
0.30· Unit:probability· Type:number· Data status:synthetic predicted_risk_5- Value:
0.45· Unit:probability· Type:number· Data status:synthetic predicted_risk_6- Value:
0.60· Unit:probability· Type:number· Data status:synthetic predicted_risk_7- Value:
0.75· Unit:probability· Type:number· Data status:synthetic predicted_risk_8- Value:
0.90· Unit:probability· Type:number· Data status:synthetic outcome_1- Value:
0· Unit:binary· Type:integer· Data status:synthetic outcome_2- Value:
0· Unit:binary· Type:integer· Data status:synthetic outcome_3- Value:
0· Unit:binary· Type:integer· Data status:synthetic outcome_4- Value:
1· Unit:binary· Type:integer· Data status:synthetic outcome_5- Value:
0· Unit:binary· Type:integer· Data status:synthetic outcome_6- Value:
1· Unit:binary· Type:integer· Data status:synthetic outcome_7- Value:
1· Unit:binary· Type:integer· Data status:synthetic outcome_8- Value:
1· Unit:binary· Type:integer· Data status:synthetic temporal_holdout_declared- Value:
1· Unit:binary· Type:integer· Data status:parameter
08
Formal model
Formal model
temporal_holdout_declared = 1; Brier = mean((p_i−y_i)^2); log_loss = −mean(y_i×log(p_i)+(1−y_i)×log(1−p_i)); calibration_gap = mean(p_i)−mean(y_i); AP_threshold = grouped_average_precision(p_i, y_i)Evidence claims: MSC-P029-C01 · MSC-P029-C02 · MSC-P029-C03 · MSC-P029-C04 · MSC-P029-C05 · MSC-P029-C06 · MSC-P029-C07 · MSC-P029-C08 · MSC-P029-C09 · MSC-P029-C10
Scientific question and scope
P(churn within H | information available at t)
09
Declared calculation
- 01
Freeze the temporal holdout and validate paired probabilities/outcomes with both outcome classes represented.
- 02
Compute Brier score, log loss, threshold-grouped average precision and the predicted-minus-observed calibration gap.
- 03
Report discrimination and calibration separately and label the tiny synthetic holdout as insufficient for deployment or causal validation.
10
Numerical example or application case
Synthetic illustration — teaching values, not observed — synthetic data or declared parameters; no real observations
Sealed inputs
predicted_risk_1=0.05 [probability]predicted_risk_2=0.10 [probability]predicted_risk_3=0.20 [probability]predicted_risk_4=0.30 [probability]predicted_risk_5=0.45 [probability]predicted_risk_6=0.60 [probability]predicted_risk_7=0.75 [probability]predicted_risk_8=0.90 [probability]outcome_1=0 [binary]outcome_2=0 [binary]outcome_3=0 [binary]outcome_4=1 [binary]outcome_5=0 [binary]outcome_6=1 [binary]outcome_7=1 [binary]outcome_8=1 [binary]temporal_holdout_declared=1 [binary]
Reproducible results
Brier=0.122188log_loss=0.385684threshold_grouped_average_precision=0.95mean_predicted_minus_observed=-0.08125
Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.
11
Validity assumptions
- Was every feature available at the declared prediction time and was the holdout later in time?
- Are tied prediction scores grouped consistently when average precision is computed?
- Are calibration, discrimination and decision utility assessed separately on a sufficiently large deployment-representative sample?
12
Diagnostics and uncertainty
Diagnostics and uncertainty
Temporal holdout · Brier and log-loss · Threshold-grouped non-interpolated average precision · Calibration sign: mean predicted risk minus observed event rate
Bounded diagnostic
The sign convention is predicted minus observed. Average precision is threshold-grouped and is not generic PR-AUC. Tiny synthetic illustration only.
Explicit uncertainty contract · not_estimable_from_sealed_inputs
Method : Explicit non-estimability assessment against the sealed input schema.
Target : Sampling uncertainty of holdout calibration and discrimination metrics.
Engine evidence : diagnostic=average_precision_is_threshold_grouped_not_generic_pr_auc_tiny_synthetic_holdout_not_deployment_validation_or_causal_effect
Interpretation : Eight synthetic holdout observations are insufficient for defensible confidence intervals or subgroup uncertainty.
Stop when
Claim which action will reduce churn from the predictive model alone.
13
Result interpretation
- Prioritize human review using calibrated probabilistic risk.
- The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.
14
Supported and forbidden conclusions
Supported
Prioritize human review using calibrated probabilistic risk.
Forbidden
Claim which action will reduce churn from the predictive model alone.
15
Possible marketing decision
- 01
Prioritize human review using calibrated probabilistic risk.
- 02
The result is conditional on the declared population, horizon, specification and diagnostics. It must not be extended to another decision without new justification.
16
When to use — when to stop
Use when
Prioritize human review using calibrated probabilistic risk.
customer at decision date
Stop when
Claim which action will reduce churn from the predictive model alone.
Methodological alternatives
- Use calibration curves with uncertainty by risk range and subgroup.
- Use ROC analysis only for discrimination questions, not calibration.
- Use decision-curve or cost-sensitive threshold analysis when action utility is the target.
17
Reproducibility contract
Python and R are the executable references. SPSS and SAS remain secondary syntaxes until checked on the same data, specification and diagnostics.
The engine selects only this page’s explicit slice and branch. Its hash, outputs, and diagnostics remain sealed in the atomic dossier; no generic fallback branch is allowed.
Slice fingerprint: 05348452a7d6e9c0921fab5459274f7d09e9ba76b92659dd05112935133f85f7
CSV · CC0
msc-validation-inputs-v1.csv ↓Python · MIT
msc-validation-reference-v1.py ↓R · MIT
msc-validation-reference-v1.R ↓SPSS / SAS · MIT · inspection only
SPSS ↓SAS ↓Formal model
temporal_holdout_declared = 1; Brier = mean((p_i−y_i)^2); log_loss = −mean(y_i×log(p_i)+(1−y_i)×log(1−p_i)); calibration_gap = mean(p_i)−mean(y_i); AP_threshold = grouped_average_precision(p_i, y_i)Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.
18
Expected final deliverable
- 01
Which customers have the highest probability of churn? —
customer at decision date - 02
p_i · y_i · Brier · log_loss · calibration_gap · AP_threshold
- 03
temporal_holdout_declared = 1; Brier = mean((p_i−y_i)^2); log_loss = −mean(y_i×log(p_i)+(1−y_i)×log(1−p_i)); calibration_gap = mean(p_i)−mean(y_i); AP_threshold = grouped_average_precision(p_i, y_i) - 04
Temporal holdout · Brier and log-loss · Threshold-grouped non-interpolated average precision · Calibration sign: mean predicted risk minus observed event rate
- 05
Prioritize human review using calibrated probabilistic risk. / Claim which action will reduce churn from the predictive model alone.
19
Scientific sources and evidence status
Verified dossier: claims are linked to passage-level sources, the Python/R calculation is reproduced, limits are explicit, and independent scientific review is complete.
MSC-P029-C01— Probabilistic scores should be evaluated on a declared test set.MSC-P029-C02— Calibration concerns observed event frequency conditional on forecast probability.MSC-P029-C03— ROC-based discrimination does not determine probability calibration.MSC-P029-C04— Calibration and discrimination require distinct diagnostics.MSC-P029-C05— Brier and logarithmic scores are proper probability-scoring rules.MSC-P029-C06— Predictive and causal objectives alter the full modelling process, not just the final metric.MSC-P029-C07— Predictive accuracy should be evaluated on observations excluded from model fitting.MSC-P029-C08— The reported AP uses threshold-specific precision weighted by recall increments.MSC-P029-C09— The executable convention is non-interpolated.MSC-P029-C10— Average precision is not labeled as generic trapezoidal PR-AUC.
Method connections