EVALUATION METRICSFOR CLICK PREDICTION

Evgeniy Zhurin, RuTarget

Binary Classification Error Measurement1) AUC2) RIG 3) LogLoss4) Precision/Recall5) F16) PE, MSE, MAE

AUC1) ignores the predicted probability values2) usually we are interested in parts of

roc curve3) considers Type 1 error and Type 2

error weights equivalently4) dependent on the underlying distribution

of data

RIG

1) bad to compare two model performances with different distributions

2) can be used to compare the relative performance of multiple models trained and tested on the same data

3) is not informative, because score also depends on the data distribution

OR WRITE A SIMULATOR

Thanks!

J. Yi, Y. Chen, J. Li, S. Sett, and T. W. Yan. Predictive model performance: Offline and online evaluations. In KDD, pages 1294–1302,

2013.http://chbrown.github.io/kdd-2013-usb/kdd/p1294.pdf