Model validation
Run the scorecard over a lender’s past loans and compare its recommendations with what happened and with what the officers decided. This is the first job with every design partner.
These numbers test the tool, not the scorecard
The synthetic book’s outcomes come from a hidden risk model we wrote. Part of it uses information the scorecard sees, part is noise it cannot see. Officers approve on their own noisy judgement and only approved loans have outcomes, as in a real book. Nothing here says how the scorecard performs on real borrowers.
Gini
0.34
95% CI 0.29–0.39
AUC
0.671
Chance a good loan out-scores a bad one
KS
0.27
Largest gap between good and bad score distributions
Loans with outcome
2,296
633 bad (27.6%) of 4,000 applications
- Declined applicants have no outcome, so these results only describe loans that were made (selection bias). Treat refer/decline bad rates with care.
- 1704 applications have no outcome (declined, or not yet matured) and are left out of the accuracy measures.
Bad rate by band
Should rise from A to E. Bands with no outcomes are empty.
| Band | Applications | With outcome | Bad | Bad rate |
|---|---|---|---|---|
| A | 119 | 114 | 10 | 8.8% |
| B | 776 | 678 | 100 | 14.7% |
| C | 1,332 | 898 | 265 | 29.5% |
| D | 1,256 | 519 | 213 | 41.0% |
| E | 517 | 87 | 45 | 51.7% |
Bad rate by score decile
Loans with an outcome, lowest scores first. A good scorecard slopes down.
Labels show the lowest score in each decile.
Scorecard against officers
How many past applications fall in each cell, and the bad rate where an outcome is known. The key cell: loans officers made that the scorecard would have declined.
| Scorecard says | Officer approved · bad rate | Officer declined | No officer decision |
|---|---|---|---|
| approve | 1,320 · 19.2% | 262 | 0 |
| refer | 267 · 36.0% | 228 | 0 |
| decline | 709 · 40.1% | 1,214 | 0 |
Officers approved 709 loans the scorecard would decline; their bad rate was 40.1%, against 19.2% for loans both approved.