receiving operating caracterisitcs
TRANSCRIPT
-
7/28/2019 Receiving Operating Caracterisitcs
1/9
Receiver operating characteristic 1
Receiver operating characteristic
ROC curve of three epitope predictors
In signal detection theory, a receiver operating characteristic (ROC),
or simply ROC curve, is a graphical plot of the sensitivity, or true
positive rate, vs. false positive rate (1 specificity or 1 true negative
rate), for a binary classifier system as its discrimination threshold is
varied. The ROC can also be represented equivalently by plotting the
fraction of true positives out of the positives (TPR = true positive rate)
vs. the fraction of false positives out of the negatives (FPR = false
positive rate). Also known as a Relative Operating Characteristic
curve, because it is a comparison of two operating characteristics (TPR
& FPR) as the criterion changes.[1]
ROC analysis provides tools to select possibly optimal models and to
discard suboptimal ones independently from (and prior to specifying)
the cost context or the class distribution. ROC analysis is related in adirect and natural way to cost/benefit analysis of diagnostic decision making. The ROC curve was first developed by
electrical engineers and radar engineers during World War II for detecting enemy objects in battle fields, also known
as the signal detection theory, and was soon introduced in psychology to account for perceptual detection of stimuli.
ROC analysis since then has been used in medicine, radiology, and other areas for many decades, and it has been
introduced relatively recently in other areas like machine learning and data mining.
http://en.wikipedia.org/w/index.php?title=Data_mininghttp://en.wikipedia.org/w/index.php?title=Machine_learninghttp://en.wikipedia.org/w/index.php?title=Radiologyhttp://en.wikipedia.org/w/index.php?title=Medicinehttp://en.wikipedia.org/w/index.php?title=Psychologyhttp://en.wikipedia.org/w/index.php?title=Decision_makinghttp://en.wikipedia.org/w/index.php?title=False_positivehttp://en.wikipedia.org/w/index.php?title=True_positivehttp://en.wikipedia.org/w/index.php?title=Binary_classifierhttp://en.wikipedia.org/w/index.php?title=Specificity_%28tests%29http://en.wikipedia.org/w/index.php?title=Sensitivity_%28tests%29http://en.wikipedia.org/w/index.php?title=Graph_of_a_functionhttp://en.wikipedia.org/w/index.php?title=Signal_detection_theoryhttp://en.wikipedia.org/w/index.php?title=File%3ARoccurves.pnghttp://en.wikipedia.org/w/index.php?title=Epitope -
7/28/2019 Receiving Operating Caracterisitcs
2/9
Receiver operating characteristic 2
Basic concept
Terminology and derivations
from a confusion matrix
true positive (TP)eqv. with hit
true negative (TN)
eqv. with correct rejection
false positive (FP)
eqv. with false alarm, Type I error
false negative (FN)
eqv. with miss, Type II error
sensitivity or true positive rate (TPR)
eqv. with hit rate, recall
false positive rate (FPR)
eqv. with fall-out
accuracy (ACC)
specificity (SPC) or True Negative Rate
positive predictive value (PPV)
eqv. with precision
negative predictive value (NPV)
false discovery rate (FDR)
Matthews correlation coefficient (MCC)
F1 score
Source: Fawcett (2006).
A classification model (classifier or diagnosis) is a mapping of instances into a certain class/group. The classifier or
diagnosis result can be in a real value (continuous output) in which the classifier boundary between classes must be
determined by a threshold value, for instance to determine whether a person has hypertension based on blood
pressure measure, or it can be in a discrete class label indicating one of the classes.
Let us consider a two-class prediction problem (binary classification), in which the outcomes are labeled either as
positive (p) or negative (n) class. There are four possible outcomes from a binary classifier. If the outcome from a
prediction isp and the actual value is also p, then it is called a true positive (TP); however if the actual value is n
then it is said to be a false positive (FP). Conversely, a true negative (TN) has occurred when both the prediction
outcome and the actual value are n, and false negative (FN) is when the prediction outcome is n while the actual
value isp.
http://en.wikipedia.org/w/index.php?title=Binary_classificationhttp://en.wikipedia.org/w/index.php?title=Countable_sethttp://en.wikipedia.org/w/index.php?title=Blood_pressurehttp://en.wikipedia.org/w/index.php?title=Blood_pressurehttp://en.wikipedia.org/w/index.php?title=Hypertensionhttp://en.wikipedia.org/w/index.php?title=Real_numberhttp://en.wikipedia.org/w/index.php?title=Mapping_%28mathematics%29http://en.wikipedia.org/w/index.php?title=Medical_diagnosishttp://en.wikipedia.org/w/index.php?title=Classifier_%28mathematics%29http://en.wikipedia.org/w/index.php?title=F1_scorehttp://en.wikipedia.org/w/index.php?title=Matthews_correlation_coefficienthttp://en.wikipedia.org/w/index.php?title=False_discovery_ratehttp://en.wikipedia.org/w/index.php?title=Negative_predictive_valuehttp://en.wikipedia.org/w/index.php?title=Information_retrieval%23Precisionhttp://en.wikipedia.org/w/index.php?title=Positive_predictive_valuehttp://en.wikipedia.org/w/index.php?title=Specificity_%28tests%29http://en.wikipedia.org/w/index.php?title=Accuracyhttp://en.wikipedia.org/w/index.php?title=Information_retrieval%23Fall-Outhttp://en.wikipedia.org/w/index.php?title=Information_retrieval%23Recallhttp://en.wikipedia.org/w/index.php?title=Hit_ratehttp://en.wikipedia.org/w/index.php?title=Sensitivity_%28test%29http://en.wikipedia.org/w/index.php?title=Type_II_errorhttp://en.wikipedia.org/w/index.php?title=Type_I_errorhttp://en.wikipedia.org/w/index.php?title=False_alarm -
7/28/2019 Receiving Operating Caracterisitcs
3/9
Receiver operating characteristic 3
To get an appropriate example in a real-world problem, consider a diagnostic test that seeks to determine whether a
person has a certain disease. A false positive in this case occurs when the person tests positive, but actually does not
have the disease. A false negative, on the other hand, occurs when the person tests negative, suggesting they are
healthy, when they actually do have the disease.
Let us define an experiment from P positive instances and N negative instances. The four outcomes can be
formulated in a 22 contingency table or confusion matrix, as follows:
actual value
p n total
prediction
outcome
p'True
Positive
False
Positive
P'
n'False
Negative
True
Negative
N'
total P N
ROC space
The ROC space and plots of the four prediction examples.
The contingency table can derive several evaluation
"metrics" (see infobox). To draw an ROC curve, only
the true positive rate (TPR) and false positive rate
(FPR) are needed. TPR determines a classifier or a
diagnostic test performance on classifying positive
instances correctly among all positive samples
available during the test. FPR, on the other hand,
defines how many incorrect positive results occur
among all negative samples available during the test.
A ROC space is defined by FPR and TPR as x and y
axes respectively, which depicts relative trade-offs
between true positive (benefits) and false positive
(costs). Since TPR is equivalent with sensitivity and
FPR is equal to 1 specificity, the ROC graph is
sometimes called the sensitivity vs (1 specificity)
plot. Each prediction result or one instance of a
confusion matrix represents one point in the ROC
space.
The best possible prediction method would yield a point in the upper left corner or coordinate (0,1) of the ROC
space, representing 100% sensitivity (no false negatives) and 100% specificity (no false positives). The (0,1) point is
also called a perfect classification. A completely random guess would give a point along a diagonal line (the
so-called line of no-discrimination) from the left bottom to the top right corners. An intuitive example of random
guessing is a decision by flipping coins (head or tail).
The diagonal divides the ROC space. Points above the diagonal represent good classification results, points below
the line poor results. Note that the output of a poor predictor could simply be inverted to obtain points above the line.
Let us look into four prediction results from 100 positive and 100 negative instances:
http://en.wikipedia.org/w/index.php?title=Coin_flippinghttp://en.wikipedia.org/w/index.php?title=Randomnesshttp://en.wikipedia.org/w/index.php?title=Specificity_%28tests%29http://en.wikipedia.org/w/index.php?title=Sensitivity_%28tests%29http://en.wikipedia.org/w/index.php?title=Specificity_%28tests%29http://en.wikipedia.org/w/index.php?title=Sensitivity_%28tests%29http://en.wikipedia.org/w/index.php?title=File%3AROC_space-2.pnghttp://en.wikipedia.org/w/index.php?title=Confusion_matrixhttp://en.wikipedia.org/w/index.php?title=Contingency_table -
7/28/2019 Receiving Operating Caracterisitcs
4/9
Receiver operating characteristic 4
A B C C'
TP=63 FP=28 91
FN=37 TN=72 109
100 100 200
TP=77 FP=77 154
FN=23 TN=23 46
100 100 200
TP=24 FP=88 112
FN=76 TN=12 88
100 100 200
TP=76 FP=12 88
FN=24 TN=88 112
100 100 200
TPR = 0.63 TPR = 0.77 TPR = 0.24 TPR = 0.76
FPR = 0.28 FPR = 0.77 FPR = 0.88 FPR = 0.12
ACC = 0.68 ACC = 0.50 ACC = 0.18 ACC = 0.82
Plots of the four results above in the ROC space are given in the figure. The result of method A clearly shows the
best predictive power among A, B, and C. The result ofB lies on the random guess line (the diagonal line), and it
can be seen in the table that the accuracy ofB is 50%. However, when C is mirrored across the center point (0.5,0.5),
the resulting method C' is even better than A. This mirrored method simply reverses the predictions of whatever
method or test produced the C contingency table. Although the original C method has negative predictive power,
simply reversing its decisions leads to a new predictive method C' which has positive predictive power. When the C
method predicts p or n, the C' method would predict n or p, respectively. In this manner, the C' test would perform
the best. The closer a result from a contingency table is to the upper left corner, the better it predicts, but the distance
from the random guess line in either direction is the best indicator of how much predictive power a method has. If
the result is below the line (i.e. the method is worse than a random guess), all of the method's predictions must be
reversed in order to utilize its power, thereby moving the result above the random guess line.
Curves in ROC space
Oftentimes, objects are classified based on a
continuous random variable. For example,
imagine that the protein level in diseased
people and healthy people are normally
distributed with means of 2 g/dL and 1 g/dL
respectively. A medical test might measure
the level of a certain protein in a blood
sample and classify any number above a
certain threshold as indicating disease. The
experimenter can adjust the threshold (black
vertical line in figure), which will in turn
change the false positive rate. Increasing the
threshold would result in fewer false
positives, corresponding to a leftward
movement on the curve. The actual shape of
the curve is determined by how much overlap the two distributions have.
http://en.wikipedia.org/w/index.php?title=Normal_distributionhttp://en.wikipedia.org/w/index.php?title=Normal_distributionhttp://en.wikipedia.org/w/index.php?title=Continuous_probability_distributionhttp://en.wikipedia.org/w/index.php?title=File%3AReceiver_Operating_Characteristic.png -
7/28/2019 Receiving Operating Caracterisitcs
5/9
Receiver operating characteristic 5
Further interpretations
Sometimes, the ROC is used to generate a summary statistic. Common versions are:
the intercept of the ROC curve with the line at 90 degrees to the no-discrimination line (also called Youden's J
statistic)
the area between the ROC curve and the no-discrimination line
the area under the ROC curve, or "AUC" ("Area Under Curve"), or A' (pronounced "a-prime") [2]
d' (pronounced "d-prime"), the distance between the mean of the distribution of activity in the system under
noise-alone conditions and its distribution under signal-alone conditions, divided by their standard deviation,
under the assumption that both these distributions are normal with the same standard deviation. Under these
assumptions, it can be proved that the shape of the ROC depends only on d'.
However, any attempt to summarize the ROC curve into a single number loses information about the pattern of
tradeoffs of the particular discriminator algorithm.
Detection Error Tradeoff graph
An alternative to the ROC curve is the Detection Error Tradeoff (DET) graph, which plots the False Negative Rate
(missed detections) vs. the False Positive Rate (false alarms), often on logarithmic scales.
Z-transformation
If a z-transformation is applied to the ROC curve, the curve will be transformed into a straight line. This
z-transformation is based on a normal distribution with a mean of zero and a standard deviation of one. In strength
theory, one must assume that the zROC is not only linear, but has a slope of 1.0. The normal distributions of target
and lures is the factor causing the zROC to be linear.
The linearity of the zROC curve depends on the standard deviations of the target and lure strength distributions. If
the standard deviations are equal, the slope will be 1.0. If the standard deviation of the target strength distribution islarger than the standard deviation of the lure strength distribution, then the slope will be smaller than 1.0. In most
studies, it has been found that the zROC curve slopes constantly fall below 1, usually between 0.5 and 0.9.[3]
Many
experiments yielded a zROC slope of 0.8. A slope of 0.8 implies that the variability of the target strength distribution
is 25% larger than the variability of the lure strength distribution.[4]
Another variable used is d'. d' is a measure of sensitivity for yes-no recognition that can easily be expressed in terms
of z-values. d' measures sensitivity, in that it measures the degree of overlap between target and lure distributions. It
is calculated as the mean of the target distribution minus the mean of the lure distribution, expressed in standard
deviation units. For a given hit rate and false alarm rate, d' can be calculated with the following equation: d'=z(hit
rate)- z(false alarm rate). Although d' is a commonly used parameter, it must be recognized that it is only relevant
when strictly adhering to the very strong assumptions of strength theory made above.[5]
The z-transformation of an ROC curve is always linear, as assumed, except in special situations. The Yonelinas
Familiarity-Recollection model is a two-dimensional account of recognition memory. Instead of the subject simply
answering yes or no to a specific input, the subject gives the input a feeling of familiarity, which operates like the
original ROC curve. What changes, though, is a parameter for Recollection (R). Recollection is assumed to be
all-or-none, and it trumps familiarity. If there were no recollection component, zROC would have a predicted slope
of 1. However, when adding the recollection component, the zROC curve will be concave up, with a decreased
slope. This difference in shape and slope result from an added element of variability due to some items being
recollected. Patients with anterograde amnesia are unable to recollect, so their Yonelinas zROC curve would have a
slope close to 1.0.[6]
http://en.wikipedia.org/w/index.php?title=Detection_Error_Tradeoffhttp://en.wikipedia.org/w/index.php?title=D%27http://en.wikipedia.org/w/index.php?title=Normal_distributionhttp://en.wikipedia.org/w/index.php?title=Standard_deviationhttp://en.wikipedia.org/w/index.php?title=D%27http://en.wikipedia.org/w/index.php?title=Youden%27s_J_statistichttp://en.wikipedia.org/w/index.php?title=Youden%27s_J_statistic -
7/28/2019 Receiving Operating Caracterisitcs
6/9
Receiver operating characteristic 6
Area Under Curve
The Area Under Curve (AUC) is equal to the probability that a classifier will rank a randomly chosen positive
instance higher than a randomly chosen negative one.[7]
It can be shown that the area under the ROC curve is closely
related to the MannWhitney U,[8]
which tests whether positives are ranked higher than negatives. It is also
equivalent to the Wilcoxon test of ranks.[8]
The AUC is related to the Gini coefficient ( ) by the formula
, where:
[9]
In this way, it is possible to calculate the AUC by using an average of a number of trapezoidal approximations.
The machine learning community most often uses the ROC AUC statistic for model comparison.[10]
However, this
practice has recently been questioned based upon new machine learning research that shows that the AUC is quite
noisy as a classification measure[11]
and has some other significant problems in model comparison.[12]
[13]
A reliable
and valid AUC estimate can be interpreted as the probability that the classifier will assign a higher score to a
randomly chosen positive example than to a randomly chosen negative example. However, the critical research
suggests frequent failures in obtaining reliable and valid AUC estimates. Thus, the practical value of the AUC
measure has thus been called into question, raising the possibility that the AUC may actually introduce more
uncertainty into machine learning classification accuracy comparisons than resolution.
Other measures
In engineering, the area between the ROC curve and the no-discrimination line is often preferred, due to its useful
mathematical properties as a non-parametric statistic. This area is often simply known as the discrimination. In
psychophysics, d' is the most commonly used measure.
The illustration at the top right of the page shows the use of ROC graphs for the discrimination between the quality
of different algorithms for predicting epitopes. The graph shows that if one detects at least 60% of the epitopes in a
virus protein, at least 30% of the output is falsely marked as epitopes.
Sometimes it can be more useful to look at a specific region of the ROC Curve rather than at the whole curve. It is
possible to compute partial AUC.[14]
For example, one could focus on the region of the curve with low false positive
rate, which is often of prime interest for population screening tests.[15]
Another common approach for classification
problems in which P
-
7/28/2019 Receiving Operating Caracterisitcs
7/9
Receiver operating characteristic 7
References
[1] Signal detection theory and ROC analysis in psychology and diagnostics : collected papers; Swets, 1996
[2] J. Fogarty, R. Baker, S. Hudson (2005). "Case studies in the use of ROC curve analysis for sensor-based estimates in human computer
interaction" (http://portal. acm.org/citation.cfm?id=1089530).ACM International Conference Proceeding Series, Proceedings of Graphics
Interface 2005. Waterloo, Ontario, Canada: Canadian Human-Computer Communications Society. .
[3] Glanzer, M.; Kim, K., Hilford, A., & Adams, J.K. (1999). "Slope of the receiver-operating characteristic in recognition memory".Journal of
Experimental Psychology: Learning, Memory, and Cognition25 (2): 500
513.[4] Ratcliff, R.; McCoon, G., & Tindal, M. (1994). "Empirical generality of data from recognition memory ROC functions and implications for
GMMs".Journal of Experimental Psychology: Learning, Memory, and Cognition20: 763785.
[5] Zhang, J.; Mueller, S. T. (2005). "A note on ROC analysis and non-parametric estimate of sensitivity".Psychometrika70 (203-212).
[6] Yonelinas, A. P.; Kroll, N. E. A., Dobbins, I. G., Lazzara, M., & Knight, R. T. (1998). "Recolection and familiarity deficits in amnesia:
Convergence of remember-know, process dissociation, and receiver operating characteristic data". Neuropsychology12: 323339.
[7] Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27, 861874.
[8] Mason, S. J.; Graham, N. E. (2002). "Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves:
Statistical significance and interpretation" (http://reia.inmet.gov. br/documentos/cursoI_INMET_IRI/Climate_Information_Course/
References/Mason+ Graham_2002. pdf). Quarterly Journal of the Royal Meteorological Society (128): 21452166. .
[9] Hand, D.J., & Till, R.J. (2001). A simple generalization of the area under the ROC curve to multiple class classification problems. Machine
Learning, 45, 171186.
[10] Hanley, JA; BJ McNeil (1983-09-01). "A method of comparing the areas under receiver operating characteristic curves derived from the
same cases" (http://radiology. rsnajnls.org/cgi/content/abstract/148/3/839).Radiology148 (3): 839843. PMID 6878708. . Retrieved
2008-12-03.
[11] Hanczar, B., Hua, J., Sima, C., Weinstein, J., Bittner, M. and Dougherty, E.R. (2010). Small-sample precision of ROC-related estimates.
Bioinformatics 26 (6): 822830.
[12] Lobo, J. M., Jimnez-Valverde, A. and Real, R. (2008), AUC: a misleading measure of the performance of predictive distribution models.
Global Ecology and Biogeography, 17: 145151.
[13] Hand, D.J. (2009). Measuring classifier performance: A coherent alternative to the area under the ROC curve. Machine Learning, 77:
103123.
[14] McClish, Donna Katzman (1989-08-01). "Analyzing a Portion of the ROC Curve" (http://mdm.sagepub. com/cgi/content/abstract/9/3/
190).Med Decis Making9 (3): 190195. doi:10.1177/0272989X8900900307. PMID 2668680. . Retrieved 2008-09-29.
[15] Dodd, Lori E.; Margaret S. Pepe (2003). "Partial AUC Estimation and Regression" (http://www.blackwell-synergy. com/doi/abs/10.
1111/1541-0420. 00071).Biometrics59 (3): 614623. doi:10.1111/1541-0420.00071. PMID 14601762. . Retrieved 2007-12-18.
[16] http://www.soe.ucsc.edu/~karplus/papers/better-than-chance-sep-07. pdf
[17] D.M. Green and J.M. Swets (1966). Signal detection theory and psychophysics. New York: John Wiley and Sons Inc.. ISBN 0-471-32420-5.
[18] M.H. Zweig and G. Campbell (1993). "Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine".
Clinical chemistry39 (8): 561577. PMID 8472349.
[19] M.S. Pepe (2003). The statistical evaluation of medical tests for classification and prediction. New York: Oxford.
[20] N.A. Obuchowski (2003). "Receiver operating characteristic curves and their use in radiology".Radiology229 (1): 38.
doi:10.1148/radiol.2291010898. PMID 14519861.
[21] Spackman, K. A. (1989). "Signal detection theory: Valuable tools for evaluating inductive learning".Proceedings of the Sixth International
Workshop on Machine Learning. San Mateo, CA: Morgan Kaufmann. pp. 160163.
General references
X. H. Zhou, N. A. Obuchowski, and D. M. McClish (2002). Statistical Methods in Diagnostic Medicine. NewYork, USA: Wiley & Sons. ISBN 9780471347729.
Further reading
Fawcett, Tom (2004)ROC Graphs: Notes and Practical Considerations for Researchers (http://home.comcast.
net/~tom.fawcett/public_html/papers/ROC101.pdf); Machine Learning, 2004
Zou, K.H., O'Malley, A.J., Mauri, L. (2007). Receiver-operating characteristic analysis for evaluating diagnostic
tests and predictive models. Circulation, 6;115(5):6547.
Lasko, T.A., J.G. Bhagwat, K.H. Zou and Ohno-Machado, L. (2005). The use of receiver operating characteristic
curves in biomedical informatics.Journal of Biomedical Informatics, 38(5):404415.
Balakrishnan, N., (1991)Handbook of the Logistic Distribution, Marcel Dekker, Inc., ISBN 978-0-8247-8587-1.
http://home.comcast.net/~tom.fawcett/public_html/papers/ROC101.pdfhttp://home.comcast.net/~tom.fawcett/public_html/papers/ROC101.pdfhttp://en.wikipedia.org/w/index.php?title=Morgan_Kaufmannhttp://www.soe.ucsc.edu/~karplus/papers/better-than-chance-sep-07.pdfhttp://www.blackwell-synergy.com/doi/abs/10.1111/1541-0420.00071http://www.blackwell-synergy.com/doi/abs/10.1111/1541-0420.00071http://mdm.sagepub.com/cgi/content/abstract/9/3/190http://mdm.sagepub.com/cgi/content/abstract/9/3/190http://radiology.rsnajnls.org/cgi/content/abstract/148/3/839http://reia.inmet.gov.br/documentos/cursoI_INMET_IRI/Climate_Information_Course/References/Mason+Graham_2002.pdfhttp://reia.inmet.gov.br/documentos/cursoI_INMET_IRI/Climate_Information_Course/References/Mason+Graham_2002.pdfhttp://portal.acm.org/citation.cfm?id=1089530 -
7/28/2019 Receiving Operating Caracterisitcs
8/9
Receiver operating characteristic 8
Gonen M., (2007)Analyzing Receiver Operating Characteristic Curves Using SAS, SAS Press, ISBN
978-1-59994-298-1.
Green, W.H., (2003)Econometric Analysis, fifth edition, Prentice Hall, ISBN 0-13-066189-9.
Heagerty, P.J., Lumley, T., Pepe, M. S. (2000) Time-dependent ROC Curves for Censored Survival Data and a
Diagnostic MarkerBiometrics, 56:337344
Hosmer, D.W. and Lemeshow, S., (2000)Applied Logistic Regression, 2nd ed., New York; Chichester, Wiley,
ISBN 0-471-35632-8.
Brown, C.D., and Davis, H.T. (2006) Receiver operating characteristic curves and related decision measures: a
tutorial, Chemometrics and Intelligent Laboratory Systems, 80:2438
Mason, S.J. and Graham, N.E. (2002) Areas beneath the relative operating characteristics (ROC) and relative
operating levels (ROL) curves: Statistical significance and interpretation. Q.J.R. Meteorol. Soc., 128:21452166.
Pepe, M.S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford. ISBN
0198565828.
Carsten, S. Wesseling, S., Schink, T., and Jung, K. (2003) Comparison of Eight Computer Programs for
Receiver-Operating Characteristic Analysis. Clinical Chemistry, 49:433439
Swets, J.A. (1995). Signal detection theory and ROC analysis in psychology and diagnostics: Collected papers.Lawrence Erlbaum Associates.
Swets, J.A., Dawes, R., and Monahan, J. (2000) Better Decisions through Science. Scientific American, October,
pages 8287.
External links
Kelly H. Zou's bibliography of ROC literature and articles (http://www.spl.harvard.edu/archive/spl-pre2007/
pages/ppl/zou/roc.html)
A more thorough treatment of ROC curves and signal detection theory (http://www-psych.stanford.edu/~lera/
psych115s/notes/signal/)
Tom Fawcett's ROC Convex Hull: tutorial, program and papers (http://home.comcast.net/~tom.fawcett/
public_html/ROCCH/index.html)
Peter Flach's tutorial on ROC analysis in machine learning (http://www.cs.bris.ac. uk/~flach/ICML04tutorial/
index.html)
The magnificent ROC (http://www.anaesthetist.com/mnm/stats/roc/)An explanation and interactive
demonstration of the connection of ROCs to archetypal bi-normal test result plots
Web-based calculator for ROC Curves (http://www.rad.jhmi. edu/jeng/javarad/roc/)by John Eng (http://
www.rad.jhmi.edu/jeng/javarad/)
Convex Hull, cost trade off, etc (http://www.cs.ucl.ac.uk/staff/W.Langdon/roc/)
http://www.cs.ucl.ac.uk/staff/W.Langdon/roc/http://www.rad.jhmi.edu/jeng/javarad/http://www.rad.jhmi.edu/jeng/javarad/http://www.rad.jhmi.edu/jeng/javarad/roc/http://www.anaesthetist.com/mnm/stats/roc/http://www.cs.bris.ac.uk/~flach/ICML04tutorial/index.htmlhttp://www.cs.bris.ac.uk/~flach/ICML04tutorial/index.htmlhttp://home.comcast.net/~tom.fawcett/public_html/ROCCH/index.htmlhttp://home.comcast.net/~tom.fawcett/public_html/ROCCH/index.htmlhttp://www-psych.stanford.edu/~lera/psych115s/notes/signal/http://www-psych.stanford.edu/~lera/psych115s/notes/signal/http://www.spl.harvard.edu/archive/spl-pre2007/pages/ppl/zou/roc.htmlhttp://www.spl.harvard.edu/archive/spl-pre2007/pages/ppl/zou/roc.htmlhttp://en.wikipedia.org/w/index.php?title=Scientific_Americanhttp://en.wikipedia.org/w/index.php?title=Oxford_University_Presshttp://en.wikipedia.org/w/index.php?title=John_Wiley_%26_Sonshttp://en.wikipedia.org/w/index.php?title=Prentice_Hall -
7/28/2019 Receiving Operating Caracterisitcs
9/9
Article Sources and Contributors 9
Article Sources and ContributorsReceiver operating characteristic Source: http://en.wikipedia.org/w/index.php?oldid=458459565 Contributors: 07 Matthew, AlanUS, Amosfolarin, AndrewHZ, Anoko moonlight, Avenue,
BOR, Bayle Shanks, Biostusa, Bobbadoc, Brethvoice, Britlak, Bwonka, CWenger, Calimo, Cburnett, Chharris, ColinFine, Coolhandscot, Cpandar, Crypticfortune, D6, Datamule, Davidruben,
Debresser, Difference engine, Dlefree-loc-work, Dr.007, Dvandeventer, Dvir-ad, Editor 2k7, Erich gasboy, Fayenatic london, Flammifer, Fmelo, Fschoonj, Giftlite, Hannes Rst, Herr blaschke,
Hike395, IbnWiki, Illegalimmigrant, Indon, InverseHypercube, J04n, Jarekt, Jfaroc, Jickh, JonDePlume, Jpbowen, Jrme, Kai walz, Kevin k, Kku, Kriegman, Llorenzi, Luci Sandor, Luis
Dantas, MarmotteNZ, MartinSpacek, Martynas Patasius, Mathfreq, Melcombe, Michael Hardy, Mietchen, MilFlyboy, Myudelson, Nephron, Noblestats, Nonpareility, Noogz, Ofap, Oleg
Alexandrov, Oliver Jennrich, Ooooooooo, Pgan002, Ph.eyes, Pmjones, Qwfp, Rich Farmbrough, Rleduc42, Rod57, Sagaciousuk, Schutz, Seans Potato Business, Seglea, Sepreece, Shawnmjones,
Sjara, Smit8678, Tayste, Tdunning, The Anome, Tjamison, Turadg, Wackojane, Wfolta, YK Times, Zzztimbo, 157 anonymous edits
Image Sources, Licenses and ContributorsFile:Roccurves.png Source: http://en.wikipedia.org/w/index.php?title=File:Roccurves.png License: Creative Commons Attribution-ShareAlike 3.0 Unported Contributors: Original uploader
was BOR at en.wikipedia
Image:ROC space-2.png Source: http://en.wikipedia.org/w/index.php?title=File:ROC_space-2.png License: Creative Commons Attribution-ShareAlike 3.0 Unported Contributors:
ROC_space.png: Indon derivative work: Kai walz (talk)
File:Receiver Operating Characteristic.png Source: http://en.wikipedia.org/w/index.php?title=File:Receiver_Operating_Characteristic.pngLicense: GNU Free Documentation License
Contributors: kakau. Original uploader was Kku at de.wikipedia
License
Creative Commons Attribution-Share Alike 3.0 Unported//creativecommons.org/licenses/by-sa/3.0/