S09 · defense simulation

Defend a loan-default model to the risk committee

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

You built a gradient-boosted model that estimates the probability of default within 12 months for personal-loan applicants at a retail lender. All data and results here are synthetic, generated for this exercise, and money is in hypothetical units. In the simulation, 300,000 applications in time order; the model was trained on the approved loans of the first 70% (136,395 loans, 4,873 defaults, 3.6%) and tested out of time on the last 30% (90,000 applications, 58,605 approved). Your written report says:

Model: gradient boosting, AUC 0.741 on approved out-of-time loans, against 0.717 for the current logistic scorecard.
Decision rule: decline an applicant when the predicted default probability exceeds 7.7%.
Calibration: predicted 3.6% against actual 3.7% default among approved test loans.

The committee: the chief risk officer, the model-validation lead and the head of lending (all simulated). Answer in writing; in a live review you would say the same things aloud.

Stage 1 of 4

The validation lead: "A scorecard is auditable line by line. Why accept a boosted model for an AUC gain of 0.024?"

Your results (synthetic, approved out-of-time loans):

	Scorecard	Gradient boosting
AUC, approved test loans	0.717	0.741
AUC difference by quarter of the test period		+0.020, +0.025, +0.021, +0.028
Profit per 1,000 approved loans at the 7.7% rule	85,337	87,965

How do you answer?

The gain holds in each of four test quarters and is worth about 2,600 per 1,000 approved loans (+3.1%). I would keep the scorecard as the documented challenger and let the committee weigh the explanation cost against that gain.Concede the point: an AUC difference of 0.024 is too small to matter in credit, so the scorecard should stay and the boosted model should be shelved for now and reconsidered later.Boosted trees are the standard for tabular credit data, so the committee should expect the model to be at least as good as any scorecard and accept it on that basis alone.