ebmrc how to use an article about a diagnostic test akbar soltani. md, endocrinologist tehran...
TRANSCRIPT
EBMRC
How to Use an Article About a Diagnostic Test
Akbar Soltani. MD, EndocrinologistTehran University of Medical Sciences (TUMS)
Endocrine and Metabolism Research Center (EMRC)Evidence-Based Medicine Research Center (EBMRC)
Shariati Hospitalwww.soltaniebm.com
www.ebm.irwww.avincennact.ir
EBMRC
OBJECTIVES•Objectives of testing•Diagnostic research•Critical appraisal•Summary
EBMRC
• 2/3 legal claims against GPs in UK
• 40,000-80,000 US hospital deaths from misdiagnosis per year
• Adverse events, negligence cases, serious disability more likely to be related to misdiagnosis than drug errors
• Diagnosis uses <5% of hospital costs, but influences 60% of decision making
EBMRC
Objectives of testing • Increasing certainty of the
presence or absence of disease • This requires sufficient
discriminative power. • 2×2 table relating test outcome
to a reference standard.
EBMRC
Objectives of testing
• Supporting clinical management
• Determining localization, and shape of arterial lesions is necessary for treatment decisions
EBMRC
Objectives of testing
• Assessing prognosis As the starting point for clinical follow up and informing patients.
• Monitoring clinical course When a disease is untreated, or during or after treatment.
• Measuring fitness For example, for sporting activity or for employment.
EBMRC
OBJECTIVES•Objectives of testing•Diagnostic research•Critical appraisal•Summary
EBMRC
Evidence Based Approach
How to Use an Article About a Diagnostic Test
EBMRC
Appraising diagnostic tests: 3 easy steps
1. Are the results valid?
2. What are the results?
3. How Can the Results be Applied to Patient Care?
EBMRC
Some definitions
True positive
A
False positive
B
False negative
C
True negative
D
Disease Present Absent
Positive
Negative
TestResult
Sensitivity = A / (A+C)
Specificity = D / (B+D)
EBMRC
Tip…..
• Sensitivity is useful to me– ‘The new rapid chlamydia test was positive in 47 out of 56 women with
chlamydia (sensitivity =83.9%)’
• Specificity seems a bit confusing!– ‘The new rapid chlamydia test was negative in 600 of the 607 women
who did not have chlamydia (specificity = 98.8%)’
• So…the false positive rate is sometimes easier– False positive rate = 1 – specificity– So a specificity of 98.8% means that the new rapid test is wrong (or
falsely positive) in 1.2% of women
EBMRC
Some definitions
True positive
A
False positive
B
False negative
C
True negative
D
Disease Present Absent
Positive
Negative
TestResult
PPV = A / (A+B)
NPV = D / (C+D)
Sen*prevalence Sen*prev+(1-Spec) *(1-prev)
Spec*(1-prev)(1-Sen)*prev+ Spec*(1-prev)
EBMRC
Series of patientsSeries of patients
Index testIndex test
Reference (“gold”) standardReference (“gold”) standard
Compare the results of the Compare the results of the index test with the reference index test with the reference
standard, blindedstandard, blinded
Basic structure of diagnostic studiesBasic structure of diagnostic studies
EBMRC
Appraising diagnostic tests: 3 easy steps
1. Are the results valid?
2. What are the results?
3. How Can the Results be Applied to Patient Care?
EBMRC
• What were the key selection (inclusion & exclusion) criteria?
• Were they replicable?
• List important selection criteria; e.g. age group, gender, risk profile, medical history.
• There should be sufficient information in the paper to allow the reader to theoretically select a similar population
EBMRC
• Did selection lead to an appropriate spectrum of participants (like those assessed in practice)
• Participants with the range of common presentations of the target disorder and with commonly confused diagnosis
EBMRC
1. Was an appropriate spectrum of patients included? Spectrum bias
• You want to find out how good chest X rays are for diagnosing pneumonia in the Emergency Department
• Best = all patients presenting with difficulty breathing get a chest X-ray
• Spectrum bias = only those patients in whom you really suspect pneumonia get a chest X ray
EBMRC
Spectrum bias
• Studies from referral centers
• Patients with negative results are less likely to referred
EBMRC
2) Did the results of the test being evaluated influence the decision to perform the reference standard? 2. Were all patients
subjected to the gold standard? Verification (work-up) bias
• What was the gold standard of diagnosis?
• The validity of the study requires that there is an accepted, valid and replicable reference standard of diagnosis.
EBMRC
gold standard• 1.Laboratory tests (Infectious & endocrine diseases)• 2.Imaging (DVT, PTE )• 3.Biopsy
(Cancer , vasculitis)• 4.Autopsy (neurologic diseases)• 5.long-term follow-up (SLE , MS)
EBMRC
2) Did the results of the test being evaluated influence the decision to perform the reference standard? 2. Were all patients
subjected to the gold standard? Verification (work-up) bias
• You want to find out how good is exercise ECG (“treadmill test”) for identifying patients with angina
• The gold standard is angiography
• Best = all patients get angiography
• Verification (work-up bias) = only patients who have a positive exercise ECG get angiography
EBMRC
• You want to find out how good is exercise ECG (“treadmill test”) for identifying patients with angina
• All patients get the gold standard (angiography)• Observer bias = the Cardiologist who does the
angiography knows what the exercise ECG showed (not blinded)
• Another example: The pulmonary nodule on CT, and comparison to CXR
3. Was there an independent, blind or objective comparison with the gold standard? Observer bias
EBMRC
4) Were the methods for performing the test described in sufficient detail to permit replication?
• This description should cover all issues that are important in the preparation of the patient (diet, drugs to be avoided, precautions after the test), the performance of the test (technique, possibility of pain), and the analysis and interpretation of its results.
EBMRC
Series of patientsSeries of patients
Index testIndex test
Reference (“gold”) standardReference (“gold”) standard
Compare the results of the Compare the results of the index test with the reference index test with the reference
standard, blindedstandard, blinded
Basic structure of diagnostic studiesBasic structure of diagnostic studies
EBMRC
Read this abstract
• Scan in UTI abstract
EBMRC
• Scan in UTI abstract
Index test
Series of patients
Gold standard
Accuracy
EBMRC
Series of patients
Index test
Reference (“gold”) standard
Compare the results of the index test with the reference
standard
Women presenting with history of recurrent UTIs
Self diagnosis based on symptoms
Positive urine culture
172 episodes:
Positive urine culture in 144 (84%)
EBMRC
Do reports meet the standards?
• Between 1978 and 1993 the authors found 112 articles, predominantly in radiological tests and immunoassays.
• Few of the standards were met consistently - ranging from 46% avoiding workup bias down to 9% reporting accuracy in subgroups.
EBMRC
EBMRC
EBMRC
EBMRC
Appraising diagnostic tests: 3 easy steps
1. Are the results valid?
2. What are the results?
3. How Can the Results be Applied to Patient Care?
EBMRC
What are the Results?Biostatistics Review
• Sensitivity : Of all the people with a particular disease, the proportion who will test positive for it (PID)
• Specificity : Of all the people without the disease, the proportion who will test negative for it (NIH)
A gold standard diagnosis is already known, (presumably without error). The above terms are describing the properties of a particular TEST.
EBMRC
What are the Results?Biostatistics Review
Example: Suppose the Prevalence of a particular disease in a population is 10%. (of 1000 people, 100 have the disease)
Sensitivity: 90/100 = 90%
Specificity: 810/900 = 90%Dz (+) Dz (-)
Test(+) 90 90Test(-) 10 810
100 900
EBMRC
What are the Results?Biostatistics Review
• Positive Predictive Value : Of all the people who tested positive for a disease, the proportion that actually has it
• Negative Predictive Value : Of all the people who tested negative for a disease, the proportion that actually does not have it
In these patients, what you know are their test results, from which you are trying to determine whether they actually have the disease.
EBMRC
What are the Results?Biostatistics Review
Same example: Prevalence of a particular disease in the population is 10%.
PPV: 90/180 = 50%
NPV: 810/820 = 99%
Dz (+) Dz (-)
Test(+) 90 90 180Test(-) 10 810 820 Sens: 540/600 = 90%
Spec: 360/400 = 90%
EBMRC
What are the Results?Note: The PPV and NPV of a test are DEPENDENT on
prevalence.
Suppose the Disease Prevalence is 60%
Sens: 540/600 = 90%
Spec: 360/400 = 90%
PPV: 540/580 = 93%
NPV: 360/420 = 86%
Dz (+) Dz (-)
Test(+) 540 40 580Test(-) 60 360 420
600 400
EBMRC
What are the Results?The use of PPV and NPV when describing a diagnostic test
has several drawbacks:
1. They are dependent on the prevalence of disease2. The prevalence of disease in the general population is
not the same as that of the patients you see in clinic/ER.3. Not all tests have results that can be categorized as “+”
or “-”.
For these reasons, PPV and NPV may soon be considered “Old School”.
EBMRC
In My Experience!
Likelihood of disease and the PPVLikelihood of disease and the PPVof diagnostic tests in specialty settingof diagnostic tests in specialty setting
A test with 98% sen &spA test with 98% sen &spPrimary care setting p=2% ppv=50%Primary care setting p=2% ppv=50%Specialty setting p=30% ppv=95%Specialty setting p=30% ppv=95%
EBMRC
A disease with a prevalence of 4% must be diagnosed.
It has a sensitivity of 50% and a specificity of 90%.
If the patient tests positive, what is the chance they have the disease?
Try itTry it
EBMRC
Prevalence of 4%, Sensitivity of 50%, Specificity of 90%
4
96
2
9.6
100
11.6 people test positive……….
of whom 2 have the disease
So, chance of disease is 2/11.6 about 17%
Disease +ve
Disease -ve
Testing +ve
Sensitivity = 50%
False positive rate = 10%
EBMRC
Prevalence of 4%, Sensitivity of 50%, Specificity of 90%
• Doctors with an average of 14 yrs experience
….answers ranged from 1% to 99%
….half of them estimating the probability as 50%
Gigerenzer G BMJ 2003;327:741-744
EBMRC
What are the Results?
Likelihood Ratios
Simply stated, the Likelihood Ratio is how much more likely someone is to get a positive test result if they have the disease, compared to someone who doesn’t.
In fact, the LR is how much more likely someone is to get any particular test result (positive, negative, “intermediate probability, etc) if they have disease, compared to someone who doesn’t.
EBMRC
What are the Results?From our previous example:The LR is (90/100) / (90/900) = 9.0
This is the LR for a (+) result. Someone with disease is 9 times as likely to test positive than someone without it.
Note also that :LR = SENS / 1 - SPEC
WOWO
Dz (+) Dz (-)
Test(+) 90 90Test(-) 10 810
100 900
EBMRC
What are the Results?The LR for a negative test is:
(10/100) / (810/900) = 0.11
Someone with disease is 0.11 times as likely (1/9) to test negative than someone without it.
Dz (+) Dz (-)
Test(+) 90 90Test(-) 10 810
100 900
EBMRC
Likelihood RatiosLR = 18High
Mr. A Pre-Test Prob.
15%
Post-Test Prob. 20%
Post-Test ProbabilityPre-Test
Probability
Mrs. BPre-Test Prob.
40%
Post-Test Prob. 90%
LR = 1.2V/Q ScanResults
EBMRC
Application: Using Post-test Probability
Diseaseruled IN
Disease ruled OUT
Disease not
ruled in or out
Above this point,treat
Below this point,no further testing
Determined by:Complications of untreated disease
Risks of therapyComplications of tests
Cost
EBMRC
What do likelihood ratios mean?
LR>10 = strong positive test result
LR<0.1 = strong negative test result
LR=1
No diagnostic value
EBMRC
What do likelihood ratios mean?
McGee: Evidence based Physical Diagnosis (Saunders Elsevier)
EBMRC
? Appendicitis:
McBurney tenderness LR+ = 3.4
Pre test 5%
%
%
Bayesian reasoning
Fagan nomogram
Post test 20%
EBMRC
Appraising diagnostic tests: 3 easy steps
1. Are the results valid?
2. What are the results?
3. How Can the Results be Applied to Patient Care?
EBMRC
How Can the Results be Applied to Patient Care?
Will the results change my management strategy?
• Always ask yourself the above before ordering a test• What is your treatment threshold?• What is your threshold for NOT treating?
If you find that most patients are having test results with a LR near 1.0, the test is not very useful.
EBMRC
How Can the Results be Applied to Patient Care?
Will patients be better off as a result of the test?
Do the benefits of performing the test outweigh the associated risk?
If the answer to this is unclear, should a randomized controlled trial be done to determine if the test actually gives any benefit?
EBMRC
Summary •Objectives of testing•Diagnostic research•Critical appraisal
EBMRC
•THANK YOUTHANK YOU