project-012 · Classical ML · telecom
Churn risk is not persuadability: targeting a retention budget and designing the test that measures it
A telecom's retention team can offer a discount to 10% of its customers this month. Whom should it target, and how will it know whether the offers worked? A churn model ranks risk, but the value of an offer depends on how much it changes behaviour, which the historical data cannot show.
At a glance
Data
churn (OpenML data id 40701) (public) · 5,000 rows x 21 columns (20 features + class)
Details on dataBaseline
Business rule: flag customers with four or more service calls or an international plan, ranked by service calls. Second baseline: regularized logistic regression.
Details on baselineValidation
Stratified 80/20 split; repeated stratified 5-fold CV (5 repeats) on the 80% for selection; test used once. No temporal validation is possible because no dates exist;
Details on validationMetric
PR-AUC, lift and recall at the top decile, Brier score, reliability by decile, all with intervals.
Details on metric
Success criterion
- Held-out lift in the top decile and PR-AUC for the selected model, with bootstrap intervals, compared with the business rule (four or more customer-service calls, or international plan).
- A targeting table showing expected retained margin under three stated, hypothetical uplift assumptions, and the break-even uplift for the offer.
- A randomized holdout design within the targeted deciles, with a power calculation for detecting the break-even uplift.
Context
The OpenML churn dataset has 5,000 telephony customers with plan flags, usage minutes, calls and charges by time of day, customer-service calls and a churn label (707 churners). OpenML lists the author as unknown, with copies on PMLB and BigML; the observation window and churn definition are undocumented. Charges are likely computed from minutes, and phone_number is an identifier. This is a deliberately thin-provenance dataset: the modelling is easy, the judgement about what it supports is the work.
Data
- Dataset
- churn (OpenML data id 40701) (public)
- Licence
- public (as stated by OpenML)
- Size
- 5,000 rows x 21 columns (20 features + class)
- Verification
- identity verified
Access
sklearn.datasets.fetch_openml(data_id=40701, as_frame=True) or openml.org download; no account. Cache with the
OpenML version and checksum.
Grain
One row per customer account at an undocumented snapshot; label = churned (window undocumented).
| Column | Meaning |
|---|---|
| state / area_code | State (stored as numeric codes on OpenML; treat as categorical) and area code (3 values) |
| account_length | Account age (unit undocumented) |
| phone_number | Identifier; never a feature |
| international_plan / voice_mail_plan | Plan flags (0/1) |
| number_vmail_messages | Voice-mail messages |
| total_day/eve/night/intl _minutes | Usage and charges by time band |
| number_customer_service_calls | Customer-service calls (stored as nominal 0-9) |
| class | 1 = churned |
EDA and data-quality rules
phone_number uniqueness; drop it from features.
Check whether each charge column is an exact linear function of its minutes column; if so keep one of each pair.
Compare area_code with state: three area codes against many states suggests synthetic or recoded geography; record the finding and avoid geographic claims.
Convert number_customer_service_calls to numeric and inspect the churn rate by count.
Class balance (707 of 5,000) and duplicates on all non-id columns.
Baseline
Business rule: flag customers with four or more service calls or an international plan, ranked by service calls. Second baseline: regularized logistic regression. Random targeting anchors the lift at 1.
Features
Minutes by band (charges removed if redundant), total minutes, international plan x international minutes, service calls (numeric and capped), voice-mail flag, account length; state target-encoded inside folds or dropped in a robustness run.
Candidate methods
L2 logistic regression with a few interactions
Transparent, quick, and a fair test of whether trees add anything.
Gradient-boosted trees (LightGBM or CatBoost with native categorical state)
Captures threshold effects such as the jump at several service calls.
Validation design
Stratified 80/20 split; repeated stratified 5-fold CV (5 repeats) on the 80% for selection; test used once. No temporal validation is possible because no dates exist; the report states this as a deployment blocker.
Leakage checks
Undocumented windows: if usage is summed over a period that overlaps the churn date, churners show lower usage mechanically. phone_number and any other identifiers are excluded; target encoding happens inside folds.
Bias and representativeness
Geography may proxy for demographics; treated customers in future data will have outcomes changed by the offer, which biases retraining unless a holdout is kept.
Robustness checks
With and without state; seed variation; capped versus raw service calls; calibration across deciles.
Technical metrics
PR-AUC, lift and recall at the top decile, Brier score, reliability by decile, all with intervals.
Business interpretation
Expected retained margin = sum over targeted customers of churn probability x uplift x margin - offer cost, under hypothetical uplift, margin and cost; break-even uplift; sample size for the holdout test.
Error slices
International-plan holders.
Customers with four or more service calls (does the model add anything beyond the rule?).
Short account length.
Voice-mail plan holders versus not.
Implementation steps
Fetch, cache, checksum; run EDA rules; decide on redundant charges and identifiers.
Build baselines and candidates; select with repeated CV.
Evaluate once on the test split; compute decile lift and calibration.
Build the targeting table under three uplift assumptions and the break-even uplift.
Design the randomized holdout (allocation, duration, power) and the analysis plan.
Reproduction
Learner-built. One command; cached OpenML file with version; seeds in config; pinned environment.
Limitations
Unknown author, window and churn definition; results cannot be generalised to any real operator.
No dates: no out-of-time validation, no tenure-aware survival framing.
Uplift is assumed, not estimated.
Production and monitoring
Monthly scoring, offer logging, a permanent random holdout to measure incremental retention, and retraining that accounts for treated customers. Once experiment data exist, move from risk ranking to uplift modelling.
Interview questions
Why might targeting the highest churn-risk customers waste the retention budget?
How would you measure whether the retention offers work?
OpenML says the author is unknown. How does that change what you would claim in an interview portfolio?
Postmortem and alternatives
The right next model is an uplift model (for example a two-model approach) trained on randomized offer data; a survival model on tenure would answer when, not only whether, customers leave.
Rubric3 criteria
| Criterion | Meets | Exceeds |
|---|---|---|
| Data skepticism | Identifies identifiers, redundant charges and undocumented windows. | Shows the area_code/state finding and adjusts claims accordingly. |
| Model evaluation | Decile lift and PR-AUC with intervals against the business rule. | Shows where the model adds value beyond the rule by slice. |
| Decision and experiment design | Targeting table with uplift assumptions and break-even. | Complete holdout design with power and a retraining plan that handles treated customers. |
References5
- churn (OpenML data id 40701)
Dataset identity and its weak provenance.
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Designing the randomized holdout and its analysis.
- scikit-learn User Guide: Target Encoder (Preprocessing data, 8.3.4.2)
Fold-safe target encoding of state.
- CatBoost: unbiased boosting with categorical features
Ordered target statistics for categorical features.
- The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets
PR-based evaluation for the minority class.