project-011 · Classical ML · finance

Probability of default for a card portfolio: calibrated, monotone, explainable and audited

A card issuer wants next-month probability of default (PD) for existing accounts to decide which accounts go on a credit-line watchlist. Risk management requires calibrated PDs, per-account reasons for watchlisting, behaviour that matches credit logic (more delinquency never lowers risk), and a disparity check across sex and age groups.

Dataset: Default of Credit Card Clients (UCI 350)
Licence: CC BY 4.0 (as stated on the UCI page)
Reference run: none
Lessons: day 11 day 12 day 13 day 16 day 17

Data
Default of Credit Card Clients (UCI 350) (public) · 30,000 rows x 23 features + ID + target
Details on data
Baseline
Base-rate PD for calibration comparisons, and a logistic scorecard on the most recent repayment status (PAY0) alone, which is the classic strong single signal in…
Details on baseline
Validation
Stratified 80/20 split with a fixed seed; 5-fold stratified CV on the 80% for tuning and binning; calibration fitted on out-of-fold predictions;
Details on validation
Metric
ROC-AUC/Gini, KS, Brier, log loss, calibration slope and intercept, reliability curve; group-wise FPR/FNR and watchlist rates with intervals.
Details on metric

Success criterion

On a stratified 20% held-out test set:

Calibration slope between 0.9 and 1.1 and expected calibration error at most 0.02; Brier skill over the base rate reported.
Discrimination (ROC-AUC, KS) of the selected model at least matches the PAY_0-only scorecard baseline; any loss versus an unconstrained model is quantified.
Monotone constraints on delinquency features verified by partial dependence.
Watchlist rates and false-positive / false-negative rates by sex and age band reported with intervals, with any gap beyond a pre-stated tolerance escalated.

Context

The UCI data describe 30,000 credit card clients in Taiwan with April-September 2005 repayment status, bill and payment amounts, and whether they defaulted the following month. The dataset paper argues that the estimated probability of default is more valuable for risk management than a binary label. All clients are existing cardholders, so the data say nothing about applicants who were declined.

Data

Dataset: Default of Credit Card Clients (UCI 350) (public)
Licence: CC BY 4.0 (as stated on the UCI page)
Size: 30,000 rows x 23 features + ID + target
Verification: identity verified

Access

Direct download from the UCI page (an .xls file, 5.3 MB; reading it with pandas needs the xlrd engine, which is not in the academy's pinned dependencies) or ucimlrepo.fetch_ucirepo(id=350) as offered on the page; no account. Fallback: the ucimlrepo route avoids the .xls reader.

Grain

One row per client; features summarise the six months before the target month.

Column	Meaning
LIMIT_BAL (X1)	Amount of given credit in NT dollars, including family credit
SEX / EDUCATION / MARRIAGE / AGE (X2-X5)	Demographics; codes as documented on the UCI page
PAY_0	Repayment status Sep back to Apr 2005; the page names X6 PAY_0 and X7 PAY_2 (there is no PAY_1)
BILL_AMT1-6 (X12-X17)	Bill statement amounts, Sep back to Apr 2005
PAY_AMT1-6 (X18-X23)	Amounts paid, Sep back to Apr 2005
Y	Default payment next month (1 = yes)

EDA and data-quality rules

Check every categorical and repayment-status column for codes outside the documented ranges and write an explicit mapping decision for each.
Reconcile the naming gap (PAY_0 then PAY_2) and the month order of BILL_AMT and PAY_AMT before building ratios.
Negative bill amounts (credit balances) are counted and handled explicitly in utilisation features.
Confirm the header rows of the .xls (or use ucimlrepo) and that ID is unique.
Default rate by sex, age band and education, for the later disparity audit.

Baseline

Base-rate PD for calibration comparisons, and a logistic scorecard on the most recent repayment status (PAY_0) alone, which is the classic strong single signal in behavioural scoring.

Features

Utilisation (bill / limit), payment ratio (payment / previous bill, with the month alignment documented), maximum and count of delinquent months, trend in bills, limit bands. SEX, MARRIAGE and AGE are excluded from model inputs but kept for the audit; EDUCATION is reviewed as a potential proxy. Binning or weight-of-evidence transforms are fitted inside folds.

Candidate methods

Logistic regression scorecard on binned features
Transparent points-based reasons, naturally calibrated, the format risk teams already validate.
Gradient boosting with monotone constraints on delinquency and utilisation features, then calibration
Captures interactions while keeping credit logic; calibration is checked and fixed on a held-out fold.

Validation design

Stratified 80/20 split with a fixed seed; 5-fold stratified CV on the 80% for tuning and binning; calibration fitted on out-of-fold predictions; the 20% test set used once. The data are one cohort, so no out-of-time test is possible; the report says a real deployment needs one.

Leakage checks

All features predate the target month, but binning, imputation and calibrators fitted on the full data leak; fit them in folds. Do not create features from the target month.

Bias and representativeness

Sex, age and marital status are sensitive attributes in many jurisdictions; removing them does not remove proxies. The population contains only approved cardholders (reject-inference problem). Report error rates by group and the trade-offs between parity notions.

Robustness checks

Seed variation of the split; stability of reason codes for the same account across folds; population stability index between train and test score distributions; sensitivity to the mapping of undocumented codes.

Technical metrics

ROC-AUC/Gini, KS, Brier, log loss, calibration slope and intercept, reliability curve; group-wise FPR/FNR and watchlist rates with intervals.

Business interpretation

Expected loss = PD x exposure x loss-given-default (LGD hypothetical) for the watchlist versus the rest; watchlist size at capacity; defaults captured per 100 watchlisted accounts.

Error slices

Age bands and sex (audit slices).
Accounts with no recent delinquency but high utilisation.
Undocumented code groups in EDUCATION and repayment status.
Credit-limit bands.

Implementation steps

Load via ucimlrepo or xlrd; run the EDA rules; document code mappings.
Build features and fold-safe binning; fit the PAY_0 scorecard baseline.
Fit both candidates; calibrate; verify monotonicity with partial dependence.
Evaluate on the test set; produce reason codes (scorecard points or SHAP for the boosted model).
Run the disparity audit and robustness checks.
Write a model-risk style report: purpose, data, performance, calibration, fairness, limitations, monitoring.

Reproduction

Learner-built. One command from raw file to report; pinned environment (add xlrd or use ucimlrepo); split seed and code mappings in config; raw file checksum recorded.

Limitations

One cohort in 2005; no out-of-time validation and an economy unlike today's.
Only approved cardholders; PDs do not apply to applicants.
Several codes are undocumented; mappings are assumptions.

Production and monitoring

Monthly scoring with backtesting of predicted versus realised default rate by score band, PSI drift alerts, documented overrides, reason codes stored with each watchlist action, periodic fairness review, and recalibration triggers when calibration slope leaves the agreed band.

Interview questions

Why impose monotone constraints, and what do you lose?
If you remove SEX from the inputs, is the model fair? What would you check?
The boosted model has higher AUC but worse calibration than the scorecard. Which do you deploy for a watchlist?

Postmortem and alternatives

A multi-month default-horizon or survival model would match lending decisions better than next-month default. In a regulated setting the scorecard often wins on governance cost even when a boosted model is slightly more accurate.

Rubric4 criteria

Criterion	Meets	Exceeds
Data handling	Explicit mappings for undocumented codes and fold-safe binning.	Shows sensitivity of results to the mapping choices.
Probability quality	Calibrated PDs with slope/intercept and reliability curve on the test set.	Compares scorecard and boosted calibration by score band and explains the deployment choice.
Explainability and constraints	Monotonicity verified; reason codes per account.	Shows reason-code stability across folds.
Fairness audit	Group error rates with intervals; proxies discussed.	Discusses competing parity definitions and recommends one for this decision.

References6

Default of Credit Card Clients
Dataset identity and variable definitions.
The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients
Dataset paper; PD estimation versus classification.
Probability calibration (scikit-learn User Guide 1.16)
Calibration methods and diagnostics.
A Unified Approach to Interpreting Model Predictions
SHAP values for per-account reasons from the boosted model.
Interpretable Machine Learning: A Guide for Making Black Box Models Explainable (3rd edition)
Partial dependence and SHAP caveats.
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
LightGBM, a candidate implementation for constrained boosting.

At a glance