carrie bennette on behalf of andrew vickers
DESCRIPTION
How do we know whether a marker or model is any good? A discussion of some simple decision analytic methods. Carrie Bennette on behalf of Andrew Vickers Pharmaceutical Outcomes Research and Policy Program (PORPP) University of Washington. Overview of talk. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/1.jpg)
How do we know whether a marker or model is any good?
A discussion of some simple decision analytic methods
Carrie Bennetteon behalf of Andrew Vickers
Pharmaceutical Outcomes Research and Policy Program (PORPP)
University of Washington
![Page 2: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/2.jpg)
Overview of talk
• Marker research in cancer: state of the science
• Traditional statistical methods for evaluating predictions
• Decision analytic approaches
![Page 3: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/3.jpg)
Overview of talk
• Marker research in cancer: state of the science
• Traditional statistical methods for evaluating predictions
• Decision analytic approaches
![Page 4: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/4.jpg)
A combination of common and minor variations in five regions of DNA can help predict a man’s risk of getting prostate cancer, researchers reported Wednesday. A company formed by researchers at Wake Forest University School of Medicine is expected to make the test available in a few months …. It should cost less than $300. This is, some medical experts say, a first taste of what is expected to be a revolution in medical prognostication
![Page 5: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/5.jpg)
SNP panel
• Predictive accuracy of SNP panel (as calculated by AV): 0.57
• Predictive accuracy of single PSA in middle age: 0.75
• Doesn’t add to standard predictors (Nam et al.)
![Page 6: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/6.jpg)
Systematic review of molecular markers in cancer
• 129 papers published in 2005 and 2006 eligible for analysis
• More markers than papers
• 97% included inference statistics
• 36% included marker in a multivariable model
• 11% measured predictive accuracy
• 0 used decision analytic techniques
![Page 7: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/7.jpg)
Overview of talk
• Marker research in cancer: state of the science
• Traditional statistical methods for evaluating predictions
• Decision analytic approaches
![Page 8: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/8.jpg)
Example: Binary test for cancer on biopsy
• Patients with high PSA are referred to biopsy
• But most patients with high PSA don’t have prostate cancer
• Could a second marker help?
• Study of biopsy cohort: 26% had cancer
– Assess presence of two markers
![Page 9: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/9.jpg)
Traditional biostatistical metrics
Sensitivity Specificity PPV NPV LR+ LR- AUC (Youden)
Brier(mean squared error)
Test A 91% 40% 35% 92% 1.52 0.23 0.65 0.47
Test B 51% 78% 45% 82% 2.32 0.63 0.64 0.29
![Page 10: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/10.jpg)
Which test is best?
• Sensitivity / specificity insufficient to determine which test should be used:
– “Depends on whether sensitivity or specificity is more important”
![Page 11: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/11.jpg)
Conclusion about traditional metrics
• Traditional biostatistical techniques for evaluating models, markers and tests do not incorporate clinical consequences
• Accordingly, they cannot inform clinical practice
![Page 12: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/12.jpg)
Overview of talk
• Marker research in cancer: state of the science
• Traditional statistical methods for evaluating predictions
• Decision analytic approaches
![Page 13: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/13.jpg)
Threshold probability
• Predicted probability of disease is p=
• Define a threshold probability of disease as pt
• Patient accepts treatment if p= ≥ pt
• pt describes how patients values relative harm of false positive and false negative
![Page 14: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/14.jpg)
Decision theory
“I would biopsy a man if his risk of prostate cancer was 20% or more, that is, I would conduct no more than 5 biopsies to find one cancer. I consider the harms associated with delaying the diagnosis of prostate cancer to be four times worse than the harms, risks and inconvenience of biopsy.”
![Page 15: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/15.jpg)
Treat: Sens. Spec. Prev. Net benefit
Test A 91% 40% 26%91% × 26% -
(1 – 40%) × (1 – 26%) × (0.2 ÷ 0.8) = 0.1256
Test B 51% 78% 26%51% × 26% -
(1 – 78%) × (1 – 26%) × (0.2 ÷ 0.8) = 0.0919
Everyone 100% 0% 26%100% × 26% -
(1 – 0%) × (1 – 26%) × (0.2 ÷ 0.8) = 0.075
No-one 0% 100% 26%0% × 26% -
(1 – 100%) × (1 – 26%) × (0.2 ÷ 0.8) = 0
Worked example at pt of 20%
![Page 16: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/16.jpg)
Net benefit has simple clinical interpretation
• Net benefit of 0.126 at pt of 20%
• Using the model is the equivalent of a strategy that led to 126 patients per 1000 with cancer being biopsied with no unnecessary biopsies
![Page 17: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/17.jpg)
Net benefit has simple clinical interpretation
• Difference between model and treat all at pt of 20%.
– 0.051
• Divide by weighting 0.051/ 0.25 = 0.204
– 204 fewer false positives per 1000 patients for equal number of true positives
– E.g. 204 fewer patients undergoing biopsy without missing any cancers
![Page 18: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/18.jpg)
Decision curve analysis
4. Vary pt over an appropriate range
Vickers & Elkin Med Decis Making 2006;26:565–574
1. Select a pt 2. Positive test defined as 3. Calculate “Clinical Net Benefit” as:
tppˆ
![Page 19: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/19.jpg)
Decision analysis
All markers
PSA
Free, Total PSA
Biopsy all
Biopsy none
Vickers JCO 2009
![Page 20: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/20.jpg)
![Page 21: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/21.jpg)
Gallina vs. Partin
AUC 0.81 AUC 0.78
P=0.02
![Page 22: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/22.jpg)
Decision curve analysis
![Page 23: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/23.jpg)
![Page 24: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/24.jpg)
![Page 25: Carrie Bennette on behalf of Andrew Vickers](https://reader036.vdocuments.net/reader036/viewer/2022081519/5681315b550346895d97d10f/html5/thumbnails/25.jpg)
Conclusion
• Huge number of markers proposed
• Evidence base is very weak for most
• Traditional biostatistical methods do not assess clinical value of a marker
• Simple decision analytic methods can distinguish potentially useful from useless models and markers