roc & auc, lift
DESCRIPTION
ROC & AUC, LIFT. ד"ר אבי רוזנפלד. Introduction to ROC curves. ROC = R eceiver O perating C haracteristic Started in electronic signal detection theory (1940s - 1950s) Has become very popular in biomedical applications, particularly radiology and imaging גם בשימוש בכריית מידע. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/1.jpg)
ROC & AUC, LIFT
רוזנפלד" אבי ר ד
![Page 2: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/2.jpg)
Introduction to ROC curves
• ROC = Receiver Operating Characteristic
• Started in electronic signal detection theory (1940s - 1950s)
• Has become very popular in biomedical applications, particularly radiology and imaging
• מידע בכריית בשימוש גם
![Page 3: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/3.jpg)
False Positives / Negatives
P N
P 20 10
N 30 90
Predicted
Actu
al
Confusion matrix 1
P N
P 10 20
N 15 105
Predicted
Actu
al
Confusion matrix 2
FN
FP
Precision (P) = 20 / 50 = 0.4Recall (P) = 20 / 30 = 0.666F-measure=2*.4*.666/1.0666=.5
![Page 4: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/4.jpg)
4
Different Cost Measures• The confusion matrix (easily generalize to multi-class)
• Machine Learning methods usually minimize FP+FN • TPR (True Positive Rate): TP / (TP + FN) = Recall• FPR (False Positive Rate): FP / (TN + FP) = Precision
Predicted class
Yes No
Actual class
Yes TP: True positive
FN: False negative
No FP: False positive
TN: True negative
![Page 5: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/5.jpg)
Specific Example
Test Result
People with disease
People without disease
![Page 6: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/6.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
Threshold
![Page 7: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/7.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
True Positives
Some definitions ...
![Page 8: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/8.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
False Positives
![Page 9: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/9.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
True negatives
![Page 10: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/10.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
False negatives
![Page 11: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/11.jpg)
Test Result
without the diseasewith the disease
‘‘-’’
‘‘+’’
Moving the Threshold: left
![Page 12: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/12.jpg)
Tru
e P
osi
tive R
ate
(R
eca
ll)
0%
100%
False Positive Rate (1-specificity)
0%
100%
ROC curve
![Page 13: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/13.jpg)
ה שינוי של הגרף THRESHOLDההשפעה על
![Page 14: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/14.jpg)
Figure 5.2 A sample ROC curve.
![Page 15: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/15.jpg)
של שונים גרפים ROCסוגים
![Page 16: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/16.jpg)
Area under ROC curve (AUC)
כללי • מדד
לגרך • מתחת ROCהשטח
•0.50 , רנדומאלי מחירה .1.0הוא מושלם הוא
![Page 17: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/17.jpg)
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
AUC = 50%
AUC = 90% AUC =
65%
AUC = 100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
AUC for ROC curves
![Page 18: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/18.jpg)
Lift Charts
• X axis is sample size: (TP+FP) / N• Y axis is TP
40% of responses for 10% of costLift factor = 4
80% of responses for 40% of costLift factor = 2Model
Random
![Page 19: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/19.jpg)
Lift factor
0
0.5
1
1.5
2
2.5
3
3.5
4
4.55 15 25 35 45 55 65 75 85 95
Lift
Sample Size
Lift
Val
ue
![Page 20: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/20.jpg)
המדדים בין הקשר
![Page 21: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/21.jpg)
התרגיל ...לקראת
![Page 22: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/22.jpg)
ואז מודל על ימני לחצןCost / Benefit Analysis for Wood
![Page 23: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/23.jpg)
ה את לראות וגם הסף את לשנות אפשרCONFUSION MATRIX
![Page 24: ROC & AUC, LIFT](https://reader035.vdocuments.net/reader035/viewer/2022081417/56813d1f550346895da6dc97/html5/thumbnails/24.jpg)
ה את גם לראות וגם Liftאפשרמחיר השפעת