1
Credit Scoring Models and their Quality
Jan Koláček, Martin Řezáč
Department of Mathematics and Statistics
Faculty of Science
Masaryk University
Brno, Czech Republic
www.muni.cz
AS2011 Ribno, September 25th
− 28th
, 2011
CONTENTS 1 - 1
Contents
• Introduction
• Quality indexes
• Proposed index
• Simulation study
• Example
• References
AS2011 Ribno, September 25th
− 28th
, 2011
INTRODUCTION 2 - 1
Credit Scoring Model
Construction of the test
• G0 group of n0 bad clients
• G1 group of n1 good clients
• S – the score for each client (one-dimensional absolutely
continuous random variable)
• D = 0, 1 random variable denotes bad or good client
• c – given cutoff point, c ∈ R
• The client is classified as G1 if S ≥ c and G0 otherwise for given
cutoff point c
AS2011 Ribno, September 25th
− 28th
, 2011
INTRODUCTION 2 - 2
Measure of Diagnostic Accuracy
• T = 1 positive test result
• T = 0 negative test result
Test results: Confusion matrix
Positive test, T = 1 Negative test, T = 0 Total
G1 (D = 1) True positive (TP) False negative (FN) TP + FN
G0 (D = 0) False positive (FP) True negative (TN) FP + TN
Total TP + FP FN + TN n = n0 + n1
AS2011 Ribno, September 25th
− 28th
, 2011
INTRODUCTION 2 - 3
The sensitivity (Se) of the test is its ability to detect good client when
he is good. Se = P(T = 1|D = 1) is a probability P that the test
result is positive (T = 1), given that the client is good (D = 1).
The specificity (Sp) of the test is its ability to exclude the solidity of
client when it is absent. Sp = P(T = 0|D = 0) is a probability P that
the test result is negative (T = 0), given that the client is bad
(D = 0).
Extreme models
Ideal model: Se = Sp = 1
Random model: Se = Sp = 1/2.
AS2011 Ribno, September 25th
− 28th
, 2011
INTRODUCTION 2 - 4
Notation
Assume the realization s ∈ R of random value S (score) is available for
each client.
Let F0, F1 denote cumulative distribution functions of score of bad
and good clients, i.e.
F0(a) = P(S ≤ a | D = 0),
F1(a) = P(S ≤ a | D = 1), a ∈ R.
Assumption: F0, F1 and their corresponding densities f0, f1 are
continuous on R.
AS2011 Ribno, September 25th
− 28th
, 2011
INTRODUCTION 2 - 5
Practice
Empirical estimators of distribution functions
F0(a) = 1
n0
n
i=1
I(si ≤ a ∧ D = 0)
F1(a) = 1
n1
n
i=1
I(si ≤ a ∧ D = 1), a ∈ [L, H],
where
I(A) . . . the indicator of event A
si . . . the score of i-th client
n0, n1 . . . number of bad and good clients, n = n0 + n1
L . . . the minimum value of given score
H . . . the maximum value of given score
AS2011 Ribno, September 25th
− 28th
, 2011
QUALITY INDEXES 3 - 1
Lorenz curve
The curve is given parametrically by
x = F0(a)
y = F1(a), a ∈ R.
Notation: x = F0(a), R(x) = F1(F−1
0 (x))
we can write the Lorenz curve as R(x), x ∈ [0, 1].
AS2011 Ribno, September 25th
− 28th
, 2011
QUALITY INDEXES 3 - 2
0 0.2 0.4 0.6 0.8 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
F0
F
1
A
B
Actual model
Ideal model
Random model
Lorenz curve, Gini index
AS2011 Ribno, September 25th
− 28th
, 2011
QUALITY INDEXES 3 - 3
Gini index
Definition
Gini =
A
A + B
= 2A,
where
A . . . area between the diagonal and Lorenz curve for actual model
A + B . . . area between the diagonal and Lorenz curve for ideal model
Properties
Gini ∈ [0, 1]
random model ⇒ Gini = 0
ideal model ⇒ Gini = 1
AS2011 Ribno, September 25th
− 28th
, 2011
QUALITY INDEXES 3 - 4
Kolmogorov-Smirnov statistics
Definition
KS = max
a∈R
|F0(a) − F1(a)| .
Remark In context with notation R(x) for the Lorenz curve we can
express K-S statistics as
KS = max
x∈[0,1]
|x − R(x)| .
AS2011 Ribno, September 25th
− 28th
, 2011
QUALITY INDEXES 3 - 5
1 2 3 4 5 6 7 8 9 10 11
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
KS
F0
F1
K-S statistics
AS2011 Ribno, September 25th
− 28th
, 2011
QUALITY INDEXES 3 - 6
The Lift, QLift
Definition
Lift(a) =
P(D = 0 | S ≤ a)
P(D = 0)
=
P(S ≤ a | D = 0)
P(S ≤ a)
=
F0(a)
FALL(a)
,
where
FALL(a) = P(S ≤ a) = P(S ≤ a ∧ D = 0) + P(S ≤ a ∧ D = 1).
If we denote pB = P(D = 0), we can write
Lift(a) =
F0(a)
pBF0(a) + (1 − pB)F1(a)
, a ∈ R.
Remark The transformation q = FALL(a) leads to QLift
QLift(q) =
1
q
F0(F−1
ALL(q)), q ∈ (0, 1],
AS2011 Ribno, September 25th
− 28th
, 2011
QUALITY INDEXES 3 - 7
The QLift
AS2011 Ribno, September 25th
− 28th
, 2011
QUALITY INDEXES 3 - 8
Lift Ratio
As analogy to Gini index, we can choose a similar approach to derive
the Lift Ratio (LR) index for Lift
LR =
1
0
QLift(q) dq − 1
1
0
QLiftideal(q) dq − 1
=
A
A + B
,
where QLiftideal(q) represents the value of QLift(q) for the case of
ideal model.
For more detailed description of LR index, see Řezáč and Koláček [3].
AS2011 Ribno, September 25th
− 28th
, 2011
PROPOSED INDEX 4 - 1
Proposed index
Let a ∈ R be a cut-off point. Let us consider the classical contingency
table of given discrimination problem
Σ
P (S>a|D=1)P (D=1) P (S≤a|D=1)P (D=1) n1·
P (S>a|D=0)P (D=0) P (S≤a|D=0)P (D=0) n2·
Σ n·1 n·2 1
AS2011 Ribno, September 25th
− 28th
, 2011
PROPOSED INDEX 4 - 2
We can express the probabilities in the table by cumulative distribution
functions F0, F1. The table takes the form
Σ
(1−F1(a))(1−pB) F1(a)(1−pB) n1·
(1−F0(a))pB F0(a)pB n2·
Σ n·1 n·2 1
Pearson’s Chi-square test of independence for contingency table:
χ2
(a) = (n11n22−n12n21)2
n·1n·2n1·n2·
= (F0(a)−F1(a))2
(F0(a)−F1(a))2+ 1
pB
F1(a)(1−F1(a))+ 1
1−pB
F0(a)(1−F0(a))
AS2011 Ribno, September 25th
− 28th
, 2011
PROPOSED INDEX 4 - 3
The value χ2
(a) describes the power of dependence of both groups
(good and bad clients) for given score value a.
Definition The proposed index KR
KR = max
a∈R
χ2
(a).
Properties of χ2
(a):
• χ2
(a) ∈ [0, 1], ∀a ∈ R
• χ2
(a) → 0 for a → ±∞
• For ideal model ⇒ ∃a ∈ R such that χ2
(a) = 1
• For random model χ2
(a) = 0, ∀a ∈ R
The KR index is a type of “generalization” of KS index. However, it
takes some advantages. Moreover, it reflects the proportion of bad
clients, so it gives more information about actual model then KS
index.
AS2011 Ribno, September 25th
− 28th
, 2011
PROPOSED INDEX 4 - 4
−4 −2 0 2 4 6 8 10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
a
Pearson’svalue
KR
KR
ideal
Actual model
Ideal model
Random model
Proposed Index
AS2011 Ribno, September 25th
− 28th
, 2011
SIMULATION STUDY 5 - 1
Simulation Study
Parameters of simulation
• distribution of bad clients N(µ0, σ2
)
• distribution of good clients N(µ1, σ2
)
• µ0 < µ1
Let us define Mean Difference D (Mahalanobis distance)
D =
µ1 − µ0
σ
.
It describes the difference between score of groups of bad and good
clients. It takes values from 0 to ∞. In our simulation study, we have
calculated all quality indexes for each value of D.
Four cases of models:
pB = 0.05, 0.1, 0.2, 0.4.
AS2011 Ribno, September 25th
− 28th
, 2011
SIMULATION STUDY 5 - 2
0 1 2 3 4 5 6 7 8
0
0.2
0.4
0.6
0.8
1
pB
= 0.05
D
index
KS
KR
GINI
LiftRatio
0 1 2 3 4 5 6 7 8
0
0.2
0.4
0.6
0.8
1
pB
= 0.1
D
index
KS
KR
GINI
LiftRatio
0 1 2 3 4 5 6 7 8
0
0.2
0.4
0.6
0.8
1
pB
= 0.2
D
index
KS
KR
GINI
LiftRatio
0 1 2 3 4 5 6 7 8
0
0.2
0.4
0.6
0.8
1
pB
= 0.4
D
index
KS
KR
GINI
LiftRatio
Dependence on D for all indexes.
AS2011 Ribno, September 25th
− 28th
, 2011
EXAMPLE 6 - 1
Real data
Consumer loans data
• The use of some (not specified) scoring function for predicting the
likelihood of repayment of a client.
• We are interested in determining which clients are able to repay
their loans.
• A test set: 2327 clients – 2030 have repaid their loans (group G1)
and 297 had problems with payments or did not pay (group G0).
Thus pB
.
= 0.13.
• We use mentioned indexes to assess the discrimination power of
given scoring function.
AS2011 Ribno, September 25th
− 28th
, 2011
EXAMPLE 6 - 2
The empirical estimate of Lorenz curve, Gini = 0.803
0 0.2 0.4 0.6 0.8 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Lorenz curve
Lorenz curve
AS2011 Ribno, September 25th
− 28th
, 2011
EXAMPLE 6 - 3
The empirical estimates of F0, F1, KS = 0.757
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
KS
F
0 F
1
K−S statistics
K-S statistics
AS2011 Ribno, September 25th
− 28th
, 2011
EXAMPLE 6 - 4
The empirical estimate of QLift, LR = 0.615
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
1
2
3
4
5
6
7
8
Qlift, LR index
Actual model
Ideal model
QLift and Lift Ratio
AS2011 Ribno, September 25th
− 28th
, 2011
EXAMPLE 6 - 5
The empirical estimate of χ2
, KR = 0.300
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
KR
KR index
KR index
AS2011 Ribno, September 25th
− 28th
, 2011
EXAMPLE 6 - 6
Summary of measures
Gini K–S LR KR
Index for the data 0.803 0.757 0.615 0.300
Conclusions
• all described indexes are widely used in practice
• we developed a new approach to measure power of scoring models
• the proposed index is more conservative
AS2011 Ribno, September 25th
− 28th
, 2011
REFERENCES 7 - 1
References
[1] Anderson, R. The Credit Scoring Toolkit: Theory and Practice for
Retail Credit Risk Management and Decision Automation. Oxford
University Press, Oxford, 2007.
[2] Hand, D.J., Henley, W.E. Statistical Classification Methods in
Consumer Credit Scoring: a review. Journal of the Royal
Statistical Society, Series A. 160 (3), 523-541, 1997.
[3] Řezáč, M., Koláček, J. On Aspects of Quality Indexes for Scoring
Models. 19th International Conference on Computational
Statistics, Paris France, August 22-27, 2010 Keynote, Invited and
Contributed Papers 1, 1517-1524, 2010.
[4] Siddiqi, N. Credit Risk Scorecards: developing and implementing
intelligent credit scoring. Wiley, New Jersey, 2006.
AS2011 Ribno, September 25th
− 28th
, 2011
REFERENCES 7 - 2
[5] Thomas, L.C. A survey of credit and behavioural scoring:
forecasting financial risk of lending to consumers. International
Journal of Forecasting 16 (2), 149-172, 2000.
[6] Thomas, L.C. Consumer Credit Models: Pricing, Profit, and
Portfolio. Oxford University Press, Oxford, 2009.
[7] Thomas, L.C., Edelman, D.B., Crook, J.N. Credit Scoring and Its
Applications. SIAM Monographs on Mathematical Modeling and
Computation, Philadelphia, 2002.
[8] Xu, K. How has the literature on Gini’s index evolved in past 80
years?. economics.dal.ca/RePEc/dal/wparch/howgini.pdf, 2003.
Accessed on 1 December 2009.
AS2011 Ribno, September 25th
− 28th
, 2011