project-014 · Classical ML · insurance
Caravan-policy propensity for a direct-mail campaign: many weak signals, one held-out shot
An insurer will mail a caravan (mobile home) insurance offer to 800 of 4,000 prospects (20%). Which 800 should receive it, and how many current or likely policy holders will the mailing reach compared with random selection?
At a glance
Data
Insurance Company Benchmark (COIL 2000) (UCI 125) (public) · 5,822 training records x 86 attributes; 4,000 evaluation records with targets in a separate file; under 2 MB
Details on dataBaseline
Random selection (expected hits = 800 x base rate). One-variable rule: rank by the single attribute with the best training-fold lift, chosen inside CV.
Details on baselineValidation
Repeated stratified 10-fold CV (5 repeats) on the training file to estimate hits in the top 20% and their spread; all selection inside folds; model frozen;
Details on validationMetric
Hits in top 20% (primary), lift, PR-AUC, CV spread; evaluation-set hits with bootstrap interval.
Details on metric
Success criterion
- Model frozen using only the 5,822 training records; then scored once on the 4,000 evaluation records and compared with the official evaluation targets.
- Number of policy holders in the selected 800, with a bootstrap interval, versus the random expectation (800 x training base rate) and a one-variable rule chosen on training folds.
- Repeated-CV estimate of hits in the top 20% on training data, with its spread, reported before scoring the evaluation file, and compared with the evaluation result afterwards.
- Sensitive zip-code variables (religion, ethnic-composition subtypes) audited and excluded from targeting or justified in writing.
Context
The CoIL Challenge 2000 data come from a Dutch data-mining company and a real insurance problem. Each customer has 86 attributes: 43 sociodemographic attributes derived from the zip code (identical for everyone in the same zip) and 42 product-ownership attributes, plus the target CARAVAN (owns a mobile-home policy). Most attributes are ordinal bins (for example L3 percentage bands, L4 contribution bands), not raw values. The target is policy ownership, not response to a mailing, so the model finds look-alikes of current holders. The dataset's bias-variance paper analyses why challenge entries differed so much.
Data
- Dataset
- Insurance Company Benchmark (COIL 2000) (UCI 125) (public)
- Licence
- CC BY 4.0 (as stated on the UCI page)
- Size
- 5,822 training records x 86 attributes; 4,000 evaluation records with targets in a separate file; under 2 MB
- Verification
- identity verified
Access
Direct download from the UCI page (ticdata2000.txt, ticeval2000.txt, tictgts2000.txt, TicDataDescr.txt); no account. The data dictionary is also linked from the page. Fallback: none needed.
Grain
One row per customer; attributes 1-43 are zip-code aggregates shared by customers in the same zip.
| Column | Meaning |
|---|---|
| MOSTYPE / MOSHOOFD | Customer subtype (L0, 41 levels) and main type (L2, 10 levels) |
| MGEMLEEF | Average age band of the zip area (L1) |
| MGODRK ... MKOOPKLA (attributes 6-43) | Zip-area shares (religion, relations, education, occupation, social class, housing, cars, insurance, income bands; L3 percentage bins) and purchasing power class |
| PWAPART ... PBYSTAND (44-64) | Contribution bands for each insurance product (L4) |
| AWAPART ... ABYSTAND (65-85) | Number of policies per product |
| CARAVAN | Number of mobile-home policies (0-1); the target |
EDA and data-quality rules
Treat binned attributes as ordinal codes with documented band meanings; never as raw percentages or amounts.
Base rate of CARAVAN in training; expected random hits in 800 computed from it.
Identify customers with identical zip-level attributes (proxy for same zip) and check whether CV folds put near-identical rows on both sides.
Check redundancy between contribution (P*) and count (A*) pairs for the same product.
Flag sensitive zip-level attributes (religion shares, ethnic-composition subtypes) for the audit.
Baseline
Random selection (expected hits = 800 x base rate). One-variable rule: rank by the single attribute with the best training-fold lift, chosen inside CV.
Features
Ordinal codes as numeric for linear models and as-is for trees; product-ownership indicators; a few domain aggregates (number of products held, car-policy contribution band); sociodemographic block tested as a group. Sensitive attributes excluded in the primary model.
Candidate methods
L2 logistic regression on a compact feature set selected inside CV
Low variance, which matters when the signal is weak and positives are few.
Gradient boosting with shallow trees, strong regularization and early stopping
Checks for interactions between ownership and area profile; its variance across CV repeats is part of the evaluation.
Validation design
Repeated stratified 10-fold CV (5 repeats) on the training file to estimate hits in the top 20% and their spread; all selection inside folds; model frozen; evaluation file scored once. Report the CV estimate before opening the targets file.
Leakage checks
Reusing the evaluation targets during development turns them into a validation set; open them once. Zip-level duplicates across folds can inflate CV.
Bias and representativeness
Targeting on zip-level religion or ethnicity proxies raises discrimination concerns; ecological attributes describe areas, not individuals. Ownership-based labels reproduce past sales patterns.
Robustness checks
Spread across CV repeats; with and without the sociodemographic block; budgets of 10%, 20%, 30%.
Technical metrics
Hits in top 20% (primary), lift, PR-AUC, CV spread; evaluation-set hits with bootstrap interval.
Business interpretation
Expected policies per mailing and cost per acquisition under hypothetical mailing cost and policy margin; value of the model versus random at the same budget.
Error slices
Customer main type (MOSHOOFD).
Car-policy holders versus not.
Purchasing power class.
Fire-policy holders.
Implementation steps
Download, checksum, parse with the dictionary; run the EDA rules.
Compute baselines; build candidates; run repeated CV and record the estimate and spread.
Run the sensitive-attribute audit and decide exclusions.
Freeze the model; score the evaluation file; open the targets once; compare with the CV estimate.
Write the campaign memo: selection, expected hits, uncertainty, profile of selected prospects, label caveat.
Reproduction
Learner-built. One command; raw files with checksums; seeds in config; the evaluation-target read is a separate, logged step so reviewers can see it happened once.
Limitations
The target is ownership, not response; the campaign effect is unknown.
Zip-level attributes are ecological and coarse.
Year-2000 Dutch market; transfer to current markets is untested.
Production and monitoring
For a real campaign, run a randomized mailing (model-selected versus random) to measure response and uplift, then retrain on response labels. Log selections and outcomes; review sensitive-attribute use with compliance.
Interview questions
Your CV estimate of hits varies a lot across repeats. Which number do you report and why?
Should zip-level religion shares be used for targeting even if they improve lift?
Why might a model that finds look-alikes of current holders be the wrong tool for a mailing?
Postmortem and alternatives
A response or uplift model trained on campaign outcomes would answer the actual question. The challenge itself is a reminder that leaderboard differences on weak-signal data are often variance, which the bias-variance paper analyses.
Rubric3 criteria
| Criterion | Meets | Exceeds |
|---|---|---|
| Evaluation discipline | Model frozen before the single evaluation-target read; CV estimate recorded first. | Explains the gap between CV and evaluation hits using variance arguments. |
| Weak-signal modelling | Low-variance candidate and regularized boosting compared on hits in top 20% with spread. | Shows how much of the lift comes from product ownership versus zip profile. |
| Responsible targeting | Audits sensitive zip attributes and documents exclusions. | Quantifies the lift cost of exclusions and recommends a policy. |
References4
- Insurance Company Benchmark (COIL 2000)
Dataset identity, files and dictionary.
- A Bias-Variance Analysis of a Real World Learning Problem: The CoIL Challenge 2000
Bias-variance analysis of challenge entries.
- Cross-validation: evaluating estimator performance (scikit-learn User Guide 3.1)
Repeated stratified CV.
- Greedy function approximation: A gradient boosting machine
Gradient boosting and shrinkage.