?
Second-order accuracy metrics for scoring models and their practical use
The paper proposes new second-order accuracy metrics for scoring/rating models, which show the target preference of the model - it is better to diagnose "good" objects or better to diagnose "bad" ones for a constant generally accepted predictive power determined by the first-order metric - the Gini index. Two metrics proposed, they have both an integral representation and a numerical one. The numerical representation of metrics is of two types, the first of which is based on binary events to evaluate the model, the second on the default probability calibration given by the model. Comparison of the results of calculating the metrics allows you to validate the calibration settings of the scoring/rating model and reveals its distortions. The article provides examples of calculating second-order accuracy metrics for ratings of several rating agencies, as well as for the well-known approach to calibration based on van der Burg's ROC curves.