Description

We revisit the problem of sample selection bias in application scoring. Unlike most prior research, our focus is not on how sampling bias impacts the development of credit scorecards. Instead, we study how using a biased sample (i.e., containing only accepted credit applicants) impacts the evaluation of scorecards. We demonstrate that standard practices for scorecard evaluation underestimate the true bad rate substantially. More importantly, we propose a Bayesian evaluation framework to mitigate the adverse effect of the sampling bias on scorecard evaluation. Our framework is agnostic of the employed measure of scorecard performance and supports a range of standard indicators including, but not limited to, the AUC, KS, the Brier score as well as any measure derived from the confusion matrix like precision, recall, or F-score. 

Extensive empirical experiments using synthetic and real-world microlending data provide strong evidence that our Bayesian framework predicts the true performance of a scorecard in production much better than standard practices and benchmarks taken from the literature. 

Last, we propose an approach to shed light on the business value of better practices for scorecard performance prediction. Considering a setting in which an institution aims to optimize its acceptance policy and seeks to determine the profit-maximizing acceptance rate, we find that Bayesian evaluation outperforms alternative approaches with notable margin across a wide range of LGDs. For example, assuming an average loan amount of $375 and an average LGD of 50% (based on our microlending data), we show that the incremental profit of Bayesian evaluation over the baseline of using only data from accepted clients is roughly $30 per loan. This significant increase in profit stems from the fact that Bayesian evaluation recommends a better (i.e., more profitable) acceptance rate.